Quick answer
Crawl budget is the number of pages Google will crawl on your site within a given time. For most small business websites with under a few thousand pages it is not a limiting factor, but wasted crawling on duplicates, filters and broken URLs can still delay indexing, so it is worth keeping the site tidy.
Last updated: November 2026
Crawl budget is one of those SEO phrases that sounds important and often gets used to sell unnecessary work. Google itself has said that most sites do not need to worry about it. Yet there are situations where crawling problems do hold back small sites, and they are usually easy to fix.
This guide explains what crawl budget is in plain English, who it matters for, the signs that Google is wasting time on the wrong URLs, and a practical clean-up list for WordPress and e-commerce sites.
You do not need log files or specialist tools to get started. Search Console and a basic crawl of your own site will tell you most of what you need to know.
In this guide
- What crawl budget actually means
- Who needs to worry about it
- Signs that crawling is a problem
- Where to look in Search Console
- Fix 1: Remove or block low-value URLs
- Fix 2: Clean up redirects and broken links
- Fix 3: Tidy your XML sitemap
- Fix 4: Handle duplicates with canonicals
- Fix 5: Improve internal linking and site speed
- A short example
- What not to do
- FAQs
What crawl budget actually means
Google does not have unlimited time for every website. For each site it balances two things: the crawl capacity limit, which is how much Googlebot can fetch without overloading your server, and crawl demand, which is how much Google wants to crawl based on popularity and freshness.
Together these produce the number of URLs Googlebot is willing to request. If a large share of those requests go to useless pages, such as duplicate parameter URLs, calendar pages or endless filters, your important pages may be crawled less often or discovered later.
Who needs to worry about it
Google has said crawl budget is mainly relevant to very large sites with hundreds of thousands of URLs, sites with frequently changing content such as news, and sites that auto-generate huge numbers of pages. A typical brochure site, local service business or small blog rarely hits limits.
E-commerce stores with faceted navigation, sites with many tag and archive pages, and sites that have been hacked or have generated thousands of spam URLs are exceptions. For them, crawl waste can be real and slow down indexing of new products and pages.
Signs that crawling is a problem
- New or updated pages take weeks to appear in Google.
- Search Console shows large numbers of URLs as “Discovered, currently not indexed” or “Crawled, currently not indexed”.
- Google crawls many parameter, filter or internal search URLs.
- Server logs or crawl stats show Googlebot spending most time on low-value URLs.
- Your server is slow or returns errors during crawls.
Where to look in Search Console
Open Settings, then Crawl stats, to see how many requests Google makes each day, average response time and the breakdown by response code and file type. A sudden spike in 404s, redirects or server errors is a red flag. The Pages report shows why URLs are not indexed, which often reveals duplicates, soft 404s and redirected pages.
Compare the number of URLs Google knows about with the number you actually want indexed. A big gap suggests wasted crawl, while a close match suggests you are fine.
Fix 1: Remove or block low-value URLs
Identify URLs that add no search value: internal search results, tag archives with a handful of posts, session or tracking parameters, and duplicate sort and filter combinations. Use a noindex tag for pages that should exist but not be indexed, and robots.txt rules for areas that should not be crawled at all.
Be careful with robots.txt. A mistake can block important content, and blocking a URL prevents Google from seeing a noindex tag on it. For WordPress, decide whether tag and author archives are useful, and noindex them if they are thin.
Fix 2: Clean up redirects and broken links
Long redirect chains and redirect loops waste crawl and slow users. Update internal links to point straight to final URLs, and fix links that lead to 404 pages. A site crawl using a tool such as Screaming Frog or Sitebulb will list redirects, broken links and orphan pages in minutes.
Keep a redirect map when you restructure pages, and use single-step 301 redirects. Our technical SEO service covers redirect audits, broken link repairs and migrations.
Fix 3: Tidy your XML sitemap
Your sitemap should list only canonical, indexable, 200-status URLs that you want in search. Remove redirected, noindex and 404 pages, and keep lastmod dates accurate. Submit it in Search Console and reference it in robots.txt.
A clean sitemap is a strong hint to Google about what matters. A messy one full of junk teaches Google to distrust it.
Fix 4: Handle duplicates with canonicals
Where similar content exists on several URLs, such as product variants, print versions or UTM-tagged links, use canonical tags to point to the preferred version, and be consistent about www versus non-www, HTTP versus HTTPS and trailing slashes.
Canonicals are hints rather than commands, so make sure internal links, sitemaps and redirects all point to the same preferred URL. Mixed signals cause Google to pick its own choice, often not yours.
Fix 5: Improve internal linking and site speed
Pages that are deep in the site or only reachable through a few links are crawled less. Link important pages from your homepage, navigation and related content, and keep click depth shallow. Fast servers also allow Google to crawl more, so improve response times with good hosting and caching.
Fixing orphan pages is especially valuable. Add links from relevant pages, or remove pages that no longer serve a purpose.
A short example
Consider a Birmingham online shop with 400 products that generates over 20,000 URLs because every colour, size and sort option creates a new parameter URL. Google spends most of its time on those variations, while new products wait weeks to be indexed.
After adding canonicals to the clean product URLs, blocking internal search and sort parameters in robots.txt, and tidying the sitemap to list only products and categories, new products start appearing in Google within days and Search Console shows far fewer duplicate URLs.
What not to do
- Do not block CSS or JavaScript files that Google needs to render pages.
- Do not noindex pages you want to rank.
- Do not use crawl budget as an excuse to avoid fixing slow hosting or poor content.
- Do not obsess over crawl stats if your site is small and pages index quickly.
A simple monthly routine keeps problems from building up: check the Pages and Crawl stats reports in Search Console, run a quick site crawl, look for new error spikes after plugin or theme updates, and review any pages you published or removed. It takes under an hour and catches most crawling problems early, long before they affect traffic.
Technical SEO Services
Crawling, indexing, speed and schema fixed properly.
Frequently asked questions
Is crawl budget a ranking factor?
No. It affects how often and how deeply Google crawls your site, not how pages rank. But pages that are not crawled and indexed cannot rank.
How can I increase my crawl budget?
Improve server speed, remove low-value and duplicate URLs, fix errors and earn more links and engagement. Google crawls more when a site is fast, healthy and popular.
Should I block pages in robots.txt to save crawl budget?
Only for genuinely useless URLs such as internal search or filters. For pages that should stay out of results but be crawlable, use noindex instead.
Does a small WordPress site need to think about crawl budget?
Usually not, but keeping archives, parameters and redirects tidy still helps pages get indexed faster.
