HTTP 429 Is Not a Traffic Problem: How a Rate Limit Silently Broke Our Search Traffic

HTTP 429 means "too many requests". Most teams treat it as a polite way to slow down scrapers. That is a mistake. Google treats a 429 the same way it treats a 5xx server error, and it will drop pages from Search Console reporting under "Server error (5xx)" without telling you the real status code.

We learned this the expensive way. This is the full diagnosis, with the numbers, so you can run the same check on your own site in under an hour.

The symptom that pointed the wrong way

Search Console reported 4,212 pages under "Server error (5xx)" for webrecast.com. The natural conclusion was that our origin was falling over. Every manual check said otherwise. Pages loaded in under 300 ms. Uptime monitors were green. Application logs had no 5xx entries at all.

If your error count is huge but your origin is healthy, stop debugging the origin. Start debugging what sits between Google and your origin. In most stacks that is a CDN, a reverse proxy, or a rate limiter.

Why 429 lands in the 5xx bucket

Google's crawler classifies any response it cannot fetch content from as a server-side problem. That includes 429, 503, and connection resets. From Google's perspective the distinction does not matter: the page did not come back, so the page gets recorded as a server error.

This matters because a rate limiter default is often tighter than you think. Traefik's rateLimit middleware defaults to something you configure yourself, but a common starting point is an average of 10 requests per second with a burst of 20. A single user never hits that. Googlebot does, because it crawls in bursts of parallel connections.

Step 1: Reproduce the error with concurrent requests

A single curl will not show you the problem. We checked individual URLs for weeks and always got 200. The failure only appears under parallel load, which is exactly how crawlers behave.

Run a burst test against 20 to 50 of your own URLs with concurrent requests. Record every status code. We used a small script that fetched 319 sitemap URLs with 20 parallel connections.

The result: 154 responses were 200. 165 were 429. Roughly half of everything Google tried to crawl was being rejected, and no single-request test had ever shown it.

If your burst returns clean 200s, your problem is elsewhere and this guide ends here for you. If you see 429s, continue to step 2.

Step 2: Find which layer is limiting

A typical request path has at least three places a limit can live:

1. The CDN edge, for example Cloudflare rate limiting rules

2. The ingress proxy, for example Traefik or nginx rate limit directives

3. The application itself

Check them in that order. Read the response headers on a 429 response. If you see Cloudflare headers with a challenge or block reference, the rule lives in the Cloudflare dashboard. If the 429 comes back with your proxy's signature, the limit is at the ingress.

Our headers pointed at the ingress. The Traefik configuration had a rate limit middleware set to an average of 10 requests per second with a burst of 20, attached to the site's router.

Step 3: Check who shares the bucket

This is the part most teams miss. Rate limiters count per client IP by default. But if your traffic arrives through a tunnel, a proxy, or a CDN that forwards requests, every visitor can appear as the same source IP.

That was our exact situation. All traffic reached the cluster through a cloudflared tunnel, so the proxy saw one client: the tunnel. Every visitor and every crawler shared a single 10 request per second bucket. One moderately active user could starve Googlebot, and Googlebot could starve users.

If your architecture has a tunnel or reverse proxy in front of the ingress, either raise the shared limit high enough that it cannot matter, or configure the limiter to use a real client IP header such as X-Forwarded-For.

Step 4: Fix with numbers, not guesses

Set the limit from measured capacity, not intuition. Two rules:

• If the bucket is shared, size it for your origin's total capacity. We raised ours to an average of 200 requests per second with a burst of 500, matching limits already proven on another service on the same hardware.
• If the bucket is per client, a limit of 20 to 50 requests per second per IP is enough to stop abuse without touching legitimate crawlers.

Do not remove the limiter entirely. You want it as protection against real abuse, just not at a level a search engine trips over.

Step 5: Verify with the same burst test

Re-run the exact test from step 1. Do not trust the change until you see it under load. Ours went from 165 failures out of 319 requests to 319 out of 319 returning 200.

Then resubmit your sitemap in Search Console. The crawl queue does not refresh on its own schedule after a fix, and you want a fresh crawl to overwrite the error history attached to your pages.

Checklist

Run this against your own site this week:

• Burst test 50 URLs with parallel requests and count non-200s
• Read the response headers on one failed request to identify the limiting layer
• Confirm whether your limiter counts real client IPs or one shared proxy IP
• Set the limit from measured origin capacity
• Re-run the burst test, then resubmit the sitemap
• Recheck Search Console after seven days

What we would do differently

We should have burst-tested the site before the error count grew. The 429s had been running for weeks, and because the pages reported as "5xx" were still indexed, nothing visibly broke. The damage was silent: Google deprioritized crawling a site that kept refusing it.

If you run a site behind a CDN or tunnel, the one number worth knowing today is what a parallel crawl of your own sitemap returns. It is the closest simulation of how Google actually sees you.

---

Have you traced a Search Console error to something unexpected? Leave a note about what it turned out to be. If you want to check status codes against a batch of URLs, our HTTP status code reference covers every code Google reports on, including the ones that hide behind a 5xx label.

Tool link: https://webrecast.com/en/http-status/429