--- title: "Crawl budget" description: "Crawl budget: how Google decides how many pages to crawl, what wastes it, and why most WordPress sites don't need to worry about it." language: "en" locale: "en-US" type: "definition" canonical: "https://www.wp-masquerade.com/glossary/crawl-budget/" published: "2026-10-28T08:00:00+00:00" modified: "2026-10-28T19:59:43+00:00" translations: en-US: "https://www.wp-masquerade.com/glossary/crawl-budget/index.md" fr-FR: "https://www.wp-masquerade.com/fr/definitions/budget-de-crawl/index.md" es-ES: "https://www.wp-masquerade.com/es/glosario/crawl-budget/index.md" de-DE: "https://www.wp-masquerade.com/de/glossar/crawl-budget/index.md" it-IT: "https://www.wp-masquerade.com/it/glossario/crawl-budget/index.md" pt-BR: "https://www.wp-masquerade.com/pt/glossario/crawl-budget/index.md" nl-NL: "https://www.wp-masquerade.com/nl/woordenlijst/crawlbudget/index.md" --- # Crawl budget Crawl budget is the set of URLs a search engine like Google can and wants to crawl on a website. It depends on how much load the server can handle without slowing down and on how much interest the engine has in the pages; according to Google, it mainly concerns very large sites. Googlebot doesn't have unlimited time for each site. It has to avoid overloading the server and choose which URLs are worth visiting. Crawl budget is the name for that trade-off. The concept gets a lot of attention, but Google explicitly reserves it for large sites: for a WordPress blog or business site with a few hundred pages, it is almost never the real problem. ## How Google sets the budget Google's guide to managing crawl budget combines two elements: - **The crawl capacity limit**: the number of simultaneous connections and the pace Googlebot allows itself without straining the server. It goes up when the site responds quickly and goes down when it slows down or returns server errors. - **Crawl demand**: how interested Google is in your URLs, which depends on the size of the site, how often it is updated, how popular the pages are and their quality. Crawl budget is the set of URLs that Google can and wants to crawl. The guide is mainly aimed at sites with more than a million pages that change roughly every week, sites with more than 10,000 pages that change every day, and sites where many URLs are stuck as “Discovered - currently not indexed” in Search Console. These thresholds are only indicative. ## What wastes crawling - Duplicate URLs, such as sorted or filtered versions of the same listing: this is [duplicate content](https://www.wp-masquerade.com/glossary/duplicate-content/) at scale. - Worthless pages left open to crawlers. - “Soft 404s”, empty or error pages served with a 200 status code, which keep getting crawled. - Long [redirect](https://www.wp-masquerade.com/glossary/301-redirect/) chains. - Deleted pages that still return 200 instead of a [404 error](https://www.wp-masquerade.com/glossary/404-error/) or a 410 status. To keep URLs out of crawling for good, Google recommends the [robots.txt](https://www.wp-masquerade.com/glossary/robots-txt/) file. On the other hand, it advises against using `noindex` for this purpose: the crawler still has to download the page to read the rule, so the time is spent anyway. *Video: “Crawl Budget and the Crawl Stats report - Google Search Console Training” (Google Search Central)* ## For WordPress Most WordPress sites don't need to worry about it. The topic becomes real on a large WooCommerce store, whose filters and sort options can generate thousands of URL combinations, or on a media site with a huge number of archives (tags, dates, pagination). The Crawl Stats report, under Settings in Search Console, shows the number of requests, the average response time and the breakdown of HTTP status codes. Two situations deserve special attention. The first is a hack: thousands of injected spam pages divert crawling away from your real content, as described in our article on [SEO spam after a hack](https://www.wp-masquerade.com/hacked-wordpress-site-seo-spam-deindexing/). The second is server load: scanner bots hammering your site don't use up Google's budget, but they can slow down shared hosting, and a slow server reduces crawl capacity. Keeping those bots out, for example with the banning feature of the [WP Masquerade Pro bot traps](https://www.wp-masquerade.com/features/bot-traps/), lightens the load; it replaces neither properly sized hosting nor caching. ## Related terms - [Indexing](https://www.wp-masquerade.com/glossary/indexing/) - [Robots.txt](https://www.wp-masquerade.com/glossary/robots-txt/) - [Duplicate content](https://www.wp-masquerade.com/glossary/duplicate-content/) - [404 error](https://www.wp-masquerade.com/glossary/404-error/) - [XML sitemap](https://www.wp-masquerade.com/glossary/xml-sitemap/) ## Sources - [Google Search Central : gérer le budget d’exploration des grands sites](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) - [Google Search Central : codes d’état HTTP, erreurs réseau et DNS](https://developers.google.com/search/docs/crawling-indexing/http-network-errors)