429 Too Many Requests in Your Site Crawl: Is Googlebot Affected?

By Idea to Traffic Editorial Team · Published October 23, 2026 · Last updated October 23, 2026

A wave of 429 errors in a Screaming Frog or Semrush crawl is alarming the first time you see it. The code means 429 Too Many Requests: the server, or something sitting in front of it, decided your crawler was asking for pages too quickly and told it to back off. The real question is not whether your audit tool got throttled. It is whether Googlebot is getting the same answer, because Google reads a 429 very differently from an SEO tool.

What a 429 means to Google

Google's own documentation on HTTP status codes says its crawlers treat 429 as a signal that the server is overloaded, and it is counted as a server error, in the same bucket as a 500 or 503. That has three practical consequences:

• Crawling slows for the whole hostname. When Google sees a significant number of 429, 500 or 503 responses, it lowers the crawl rate across the entire host, including the URLs that are still returning normal pages.
• Short bursts are fine. Google documents returning 429 or 503 on purpose as an emergency brake for a couple of hours or one to two days. Crawling speeds back up on its own once the errors stop.
• Long runs cost you pages. Google warns that if Googlebot sees these codes on the same URL for multiple days, that URL may be dropped from the index. Already indexed pages are kept for a while, but not forever.

Google updated its "Reduce the Google crawl rate" page on October 6, 2026 to add examples of the Retry-After header. You can send it with a 503 or 429 as a number of seconds, such as Retry-After: 120, or as a UTC date. The mechanism itself did not change, but it is a useful reminder that a 429 is a message Google listens to.

So a 429 shown to Screaming Frog is an inconvenience. A 429 shown to Googlebot for a week is an indexing problem.

Why so many sites return 429s now

The pattern people are reporting in late 2026 has a simple cause: hosts and security layers are much stricter than they were two years ago, mostly because of the flood of AI crawlers and scrapers. Common sources of a 429 on a small site:

• Managed hosting limits. Shopify, Netlify, many WordPress hosts and cheap shared plans rate limit by IP address. A desktop crawler running 5 to 10 threads from one home IP trips these limits quickly.
• A CDN or WAF rule. Cloudflare rate limiting rules, Wordfence, Imunify360 and similar tools return 429 when a single client exceeds a request threshold.
• Bot management. Security features that score clients as bots may throttle or challenge anything that does not look like a browser. Well known search engine crawlers are usually on an allow list. Your audit tool usually is not.
• An origin that is actually struggling. Sometimes the server really is overloaded, often a database-backed site on a small plan.

The first three usually affect only your audit tool. The fourth affects everyone, including Google and real visitors.

Check whether Googlebot is seeing 429s

Do not guess. Search Console shows you exactly what Google received.

1. Open Search Console, go to Settings, then Crawl stats and click Open report.
2. Look at the By response table. Healthy sites are dominated by OK (200), plus some 301 and 304 responses. If server errors or other error responses are a visible slice and growing, Google is being throttled too. Click the row to see sample URLs and dates.
3. Check Host status on the same report. A warning on server connectivity in the last 90 days means Google had trouble reaching you.
4. Run URL Inspection on a page your crawler flagged, then click Test live URL. If the live test fetches the page fine, Googlebot is not blocked right now.

A tip that saves time: do not test by spoofing the Googlebot user agent with curl. Cloudflare and many hosts verify Googlebot by IP address, so a fake Googlebot from your laptop can get blocked even when the real one is waved through. That test tells you nothing.

If you use Cloudflare, also open Security, then Events (labelled Security Analytics on newer dashboards), and filter by the user agent or by your own IP. Each event names the rule or feature that acted on the request, which tells you exactly what is returning the 429.

Reproduce the limit in one minute

To see where the threshold sits, fire a quick burst at one URL from a terminal and count the status codes:

for i in $(seq 1 60); do
  curl -s -o /dev/null -w "%{http_code}\n" https://example.com/
done | sort | uniq -c

If you get 60 lines of 200, a burst of about 60 sequential requests is fine. If 429s appear partway through, your host limits a single IP somewhere below that. Add -D - to one request to print the headers and look for Retry-After or a server header that names the layer responding, such as cloudflare, nginx or a host's own proxy. Replace example.com with your own domain, and only run this against sites you own.

Fix the audit crawler, not the site

When Search Console looks clean, the 429s are a crawler problem. Slow it down and identify it:

• Screaming Frog: go to Configuration, then Speed. Drop Max Threads to 1 or 2 and tick Limit URL/s, starting at 1 or 2 URLs per second. A 2,000 page site still finishes in under 20 minutes at 2 URLs per second.
• Semrush Site Audit: open the project's Site Audit settings and set the crawl delay to the slower option (one URL every two seconds) or to respect robots.txt.
• Allow list your tool if you control the firewall. In Cloudflare, a custom rule that matches your own IP address with the action Skip can bypass rate limiting for you alone. Keep the rule tight: your IP, not a user agent string, because anyone can fake a user agent.
• Crawl outside peak hours on shared hosting, when the server has spare capacity.

When Google is being throttled too

If Crawl stats shows Googlebot hitting errors, treat it as urgent, because the clock on multi-day errors is running.

• Find the rule. Use your CDN or security plugin logs to find which rule fired on Googlebot requests. Verified search engine crawlers should be exempt. If they are not, add an exception based on verified bot status or Google's published IP ranges, not on user agent.
• Raise or remove an over tight rate limit. A per-IP limit of 10 requests per minute might stop scrapers, but it will also stop Google during a busy crawl.
• Fix real overload. If the origin is genuinely struggling, add full page caching or move static assets to a CDN. A static site on Cloudflare Pages serves cached files and rarely produces server errors at all.
• Check AI crawler blocking separately. Blocking AI bots in robots.txt or with a bot management toggle is fine. Just confirm the rule does not also catch Googlebot or Bingbot.

Once the errors stop, Google ramps the crawl rate back up gradually on its own. There is no button to request a faster crawl, so give it several days and watch the Crawl stats chart flatten out. If you want a quick outside view of the basics while you wait, the free site checker runs 30 structural checks on any URL, including status codes and robots.txt.

Quick decision guide

• Only your audit tool sees 429s, Search Console is clean: slow the tool down, ignore the rest.
• Search Console shows errors for a few hours during a traffic spike: watch it, no action needed.
• Search Console shows errors for more than one or two days: find and fix the rule or the overload today.
• Pages are dropping out of the index alongside those errors: fix the errors first, then use URL Inspection to request indexing for your most important pages.

Where this question came from

• r/SEO: Increase in 429 errors during crawls?
• Google: How HTTP status codes affect Google's crawlers
• Search Engine Roundtable: Google Search Crawl Rate Doc Adds Retry-After HTTP & More

Questions about 429 errors and crawling