Definition

The crawl budget is the number of pages a search engine is willing to crawl on a site over a given period. It depends on two combined factors:

• The crawl rate limit: how much the server can handle without degraded performance
• The crawl demand: how interested the engine is in the site (popularity, freshness, authority)

When it becomes an issue

For small sites (under 10,000 URLs), crawl budget is rarely a problem. It becomes critical for:

• E-commerce sites with tens of thousands of product pages
• Sites with many filters and URL parameters
• Publishers releasing high content volumes
• Multilingual sites with hreflang across many regions

How to optimize it

• Maintain a clean XML sitemap
• Block useless sections via robots.txt
• Use the canonical tag properly
• Remove low-value pages and fix 404 / 5xx errors
• Improve server speed (core web vitals)