{"componentChunkName":"component---src-templates-post-jsx","path":"/en/post-mortem-cpu-spike-100-production-drupal","result":{"data":{"markdownRemark":{"html":"

It's 9:47 AM on a Tuesday morning. A New Relic notification comes in: CPU at 94% on the production servers. Thirty seconds later, a second alert: average response time at 8 seconds. Then a Slack message from the client: \"The site is down, our teams can't work anymore.\"

\n

This post-mortem is an illustrative scenario, built from patterns regularly encountered on Drupal platforms in production. The timeline, the commands, the decisions, and above all the mistakes are representative of this type of incident.

\n

The goal is not to show that everything always goes smoothly. The goal is to document how such an incident unfolds, why, and what we put in place so it doesn't happen again.

\n
\n

Platform Context

\n\n
\n

Incident Timeline

\n

09:47 — First New Relic Alert

\n

CPU at 94% on the application container. The alert is configured on an 80% threshold sustained for more than 2 minutes. The trigger means the situation had been going on for a while even before the notification.

\n

First action: check the New Relic dashboard.

\n

Immediate finding: the number of in-flight web transactions had tripled compared to normal. Throughput had gone from 40 req/s to around 140 req/s within 15 minutes.

\n

09:51 — Access Log Analysis

\n

SSH into the production environment:

\n
ssh deploy@prod-web-01
\n

Reading the access logs in real time:

\n
tail -f /var/log/nginx/access.log | grep -v \"varnishd\"
\n

Immediate observation: an abnormal volume of requests on URLs with varied query strings. Example pattern:

\n
185.220.x.x - - [31/May/2026:09:48:12 +0000] "GET /?page=1&sort=asc&filter=new HTTP/1.1" 200 48291\n185.220.x.x - - [31/May/2026:09:48:12 +0000] "GET /?page=2&sort=desc&filter=old HTTP/1.1" 200 48301\n185.220.x.x - - [31/May/2026:09:48:13 +0000] "GET /?page=1&sort=asc&filter=featured HTTP/1.1" 200 48288
\n

Several hundred requests per minute from a range of IPs within the same /24 block. The query strings differ on every request, which prevents any cache hit.

\n

09:54 — Confirmation: Bot Storm + Systematic Cache Miss

\n

Extracting the most active IPs:

\n
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
\n

Result: the top 5 IPs accounted for 3,200 requests over the last 10 minutes. Clearly non-human.

\n

Checking on the Cloudflare side: the requests do arrive through Cloudflare (CF-Ray headers present), but Cloudflare forwards them all. No WAF rule blocks this pattern.

\n

Checking the Varnish cache:

\n
varnishstat -1 -f MAIN.cache_hit,MAIN.cache_miss
\n

Abnormally high cache miss ratio: 78% misses instead of the usual 15%. Explanation: the unknown query strings push every request out of the cache. Varnish does not normalize the parameters — it treats ?page=1&sort=asc and ?page=1&sort=desc as two different URLs.

\n

Every miss travels all the way to Drupal. Drupal generates the full page. The CPU blows up.

\n

10:02 — First Emergency Measure: IP Blocking at the Cloudflare Level

\n

Creating an emergency Cloudflare WAF rule:

\n
(ip.src in {185.220.0.0/24}) => Block
\n

Immediate effect: requests from those IPs are blocked. CPU drops back to 60%. Not normal yet, but the pressure eases.

\n

Problem: the bots use multiple IP ranges. Within a few minutes, new IPs take over from other /24 blocks.

\n

10:09 — Second Measure: Challenge on Suspicious User Agents

\n

Rather than chasing IPs one by one, we switch to behavior-based targeting. Cloudflare rule added:

\n
(http.user_agent contains "python-requests") or \n(http.user_agent contains "curl") or \n(http.user_agent eq "") \n=> Managed Challenge (CAPTCHA)
\n

Result: request volume drops by 60%. Simple bots (Python scripts with no JS handling) no longer get through.

\n

10:14 — Third Measure: Query String Normalization in Varnish

\n

The real underlying problem remains untouched: Varnish does not normalize query strings. Even with the bots blocked, a misconfigured Google crawl or a partner who adds UTM parameters to every request can reproduce the same effect.

\n

Change in the Varnish VCL configuration:

\n
sub vcl_recv {\n    # Normalize the query strings\n    set req.url = regsuball(req.url, "([?&])(utm_[^&]*)", "");\n    set req.url = regsuball(req.url, "([?&])(sort|filter|page)=[^&]*", "");\n    \n    # Remove the trailing ? if all params have been stripped\n    set req.url = regsub(req.url, "\\?$", "");\n}
\n
\n

Note: this change required a redeployment and was tested in staging before going to production. You don't modify Varnish live without a safety net.

\n
\n

10:31 — Back to Normal

\n

CPU drops back to 22%. Average response time back under 400ms. The client is informed. The site is stable.

\n

Total incident duration: 44 minutes.

\n
\n

Root Cause Analysis

\n

The incident has three causes, not just one. That's always the case in real production incidents.

\n

Cause 1 — No Upstream Rate Limiting

\n

Cloudflare was configured in standard proxy mode, with no rate limiting rule enabled. Any IP could send as many requests as it wanted with no friction.

\n

Cloudflare rate limiting is available on all paid plans. It wasn't enabled because \"it had never caused a problem.\" A classic mistake.

\n

Cause 2 — Varnish Not Configured to Normalize Query Strings

\n

By default, Varnish treats every unique combination of query strings as a distinct URL. Without normalization, a bot that varies its parameters on every request bypasses the cache 100% of the time, even if the rendered content is identical.

\n

This configuration should have been in place from the start. It wasn't.

\n

Cause 3 — No Application-Level Circuit Breaker

\n

When Drupal receives more requests than it can handle, it keeps accepting them until the CPU saturates. There is no native mechanism in Drupal that says: \"I'm saturated, I'll return a temporary 503 instead of drowning further.\"

\n

This behavior creates a spiral: the more loaded the CPU is, the longer each request takes to process, the more the queue of pending requests grows, the higher the CPU climbs.

\n
\n

What We Should Have Had in Place

\n

Cloudflare Rate Limiting

\n

A simple rule to put in place on every project:

\n
Path: /*\nThreshold: 100 requests per IP per minute\nAction: Managed Challenge\nDuration: 10 minutes
\n

For sensitive endpoints (/admin, /user, /api): a much lower threshold, 10 req/min, with a direct Block action.

\n

Query String Normalization in Varnish from the Start

\n

The VCL configuration must be reviewed on every project to:

\n\n

Alerting on the Varnish Cache Miss Ratio

\n

New Relic or CloudWatch can monitor varnishstat. A cache miss ratio that exceeds 40% over a 5-minute window should trigger an alert, not wait until CPU is at 94%.

\n

A Watchlist of Bots

\n

Maintain a list of user agents known for aggressive scraping and create a Cloudflare rule that systematically sends them to a Managed Challenge. This list builds up incident after incident — better to start early.

\n
\n

Measures Put in Place After the Incident

\n

1. Rate Limiting Enabled on All Production Environments

\n

A global rule at 100 req/min/IP, with specific rules at 10 req/min on:

\n\n

2. Varnish VCL Revised with Query String Normalization

\n

An allowlist of parameters permitted in the cache key. Any parameter not listed is stripped before the request reaches the cache.

\n

3. New Relic Throughput Alert

\n

Triggers if the number of transactions/minute exceeds 3× the average of the last 7 days for more than 3 minutes.

\n

4. Incident Response Runbook

\n

A document in Confluence that spells out exactly what to do in case of a CPU spike:

\n
    \n
  1. Open New Relic → Transaction overview → identify the most expensive URLs
    2. \n
  2. Open the access logs → identify abnormal IPs
    3. \n
  3. Block the IP ranges in Cloudflare → check the effect on CPU
    4. \n
  4. If insufficient → temporarily enable Cloudflare \"Under Attack\" mode
    5. \n
  5. If Drupal is unreachable → switch to maintenance mode + notify the client
    6. \n
  6. Post-incident: analyze the logs, identify the root cause, document
    7. \n
\n

5. Quarterly Load Testing

\n

A k6 or Gatling test on the staging environment to verify the stack holds up at 3× nominal traffic. This test would have revealed the query string normalization weakness well before the incident.

\n
\n

Lessons Learned

\n

Most production incidents are not caused by a single problem. Here, the bot was the trigger, but the real problem was the absence of rate limiting and the misconfigured cache. Without the bot, the platform was stable. But it was fragile.

\n

Alerting on CPU alone is insufficient. At 94% CPU, it's too late. You need to alert upstream: cache miss ratio, abnormal throughput, rising P95 response time. CPU is a consequence, not a cause.

\n

Varnish configuration is not \"set and forget.\" It must be reviewed on every project, with particular attention to URL normalization. A base VCL template should be part of the starter kit for any Drupal project using Varnish.

\n

Document during the incident, not just after. While resolving it, we took notes in a dedicated Slack thread. This post-mortem is built from those notes. Without them, the timeline would have been impossible to reconstruct precisely.

\n

Conclusion

\n

A 100% CPU spike in production is stressful. It always will be. But the difference between a team that handles the incident in 44 minutes with a solid prevention plan, and a team that spends 4 hours hunting for the cause in the dark, is preparation.

\n

Rate limiting, query string normalization, multi-level alerting, and a documented runbook: these are four simple elements that cost little to put in place and can prevent a lot of damage — technical, commercial, and relational.

\n

Enterprise Drupal platforms generally don't fall because of a bug in core. They fall because one element of their stack wasn't configured to withstand an unforeseen scenario. This post-mortem is a concrete example.

","excerpt":"It's 9:47 AM on a Tuesday morning. A New Relic notification comes in: CPU at 94% on the production servers. Thirty seconds later, a second alert: average…","frontmatter":{"date":"2026-05-31","metaDate":"2026-05-31","title":"Post-mortem: CPU Spike to 100% in Drupal Production — Analysis and Resolution","tags":["Drupal","Production","Performance","Post-mortem","New Relic","Cloudflare","Bot Traffic","Incident","Drupal 11"],"path":"/post-mortem-cpu-spike-100-production-drupal","cover":{"childImageSharp":{"fluid":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAFgABAQEAAAAAAAAAAAAAAAAAAQAF/8QAFgEBAQEAAAAAAAAAAAAAAAAAAAIE/9oADAMBAAIQAxAAAAHIR1yVH//EABQQAQAAAAAAAAAAAAAAAAAAACD/2gAIAQEAAQUCX//EABQRAQAAAAAAAAAAAAAAAAAAABD/2gAIAQMBAT8BP//EABQRAQAAAAAAAAAAAAAAAAAAABD/2gAIAQIBAT8BP//EABQQAQAAAAAAAAAAAAAAAAAAACD/2gAIAQEABj8CX//EABkQAAMAAwAAAAAAAAAAAAAAAAABEBEhMf/aAAgBAQABPyFR90KZP//aAAwDAQACAAMAAAAQwA//xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAEDAQE/ED//xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAECAQE/ED//xAAcEAADAAEFAAAAAAAAAAAAAAAAAREhMUFhcfD/2gAIAQEAAT8Qjkx5CpNqXUEGoYbU/9k=","aspectRatio":1.5001937233630376,"src":"/static/e94c64bca92ef8d77addbf763315523d/88110/cover.jpg","srcSet":"/static/e94c64bca92ef8d77addbf763315523d/0b320/cover.jpg 480w,\n/static/e94c64bca92ef8d77addbf763315523d/60b32/cover.jpg 960w,\n/static/e94c64bca92ef8d77addbf763315523d/88110/cover.jpg 1920w,\n/static/e94c64bca92ef8d77addbf763315523d/40175/cover.jpg 2880w,\n/static/e94c64bca92ef8d77addbf763315523d/e58c2/cover.jpg 3840w,\n/static/e94c64bca92ef8d77addbf763315523d/e5b5f/cover.jpg 3872w","srcWebp":"/static/e94c64bca92ef8d77addbf763315523d/d1a9d/cover.webp","srcSetWebp":"/static/e94c64bca92ef8d77addbf763315523d/bc3bf/cover.webp 480w,\n/static/e94c64bca92ef8d77addbf763315523d/39337/cover.webp 960w,\n/static/e94c64bca92ef8d77addbf763315523d/d1a9d/cover.webp 1920w,\n/static/e94c64bca92ef8d77addbf763315523d/fcbe1/cover.webp 2880w,\n/static/e94c64bca92ef8d77addbf763315523d/c136d/cover.webp 3840w,\n/static/e94c64bca92ef8d77addbf763315523d/228a9/cover.webp 3872w","sizes":"(max-width: 1920px) 100vw, 1920px"},"resize":{"src":"/static/e94c64bca92ef8d77addbf763315523d/c4f3a/cover.jpg"}}}}}},"pageContext":{"isCreatedByStatefulCreatePages":false,"pathSlug":"/post-mortem-cpu-spike-100-production-drupal","locale":"en","prev":{"fields":{"locale":"en"},"frontmatter":{"path":"/drupal-ai-content-transformation","title":"Why AI Will Transform the Editorial Experience in Drupal","tags":["Drupal 11","IA","Contribution","AI"]}},"next":null}}}