Quick answer
A robots.txt file tells crawlers which parts of your site they may access. The most damaging mistakes are blocking the whole site with Disallow: /, blocking CSS and JavaScript, confusing blocking with noindex, and leaving a staging rule live after launch. Test any change in Search Console before and after publishing.
Last updated: November 2026
A robots.txt file is a tiny text file with outsized power. One misplaced line can stop Google crawling your whole website, and it happens more often than most owners realise, usually after a redesign or when a developer forgets to remove a setting.
Robots.txt is also misunderstood. It controls crawling, not indexing, and it is not a security tool. Treating it as either can cause problems.
This guide lists the mistakes we see most often on UK business sites, explains how to check yours in minutes, and gives a sensible WordPress template you can adapt.
In this guide
- What robots.txt does and does not do
- Mistake 1: Blocking the whole site
- Mistake 2: Blocking CSS, JavaScript or images
- Mistake 3: Using robots.txt to hide sensitive content
- Mistake 4: Blocking pages you want noindexed
- Mistake 5: Forgetting the sitemap and using the wrong syntax
- Mistake 6: Blocking AI crawlers by accident
- A safe WordPress robots.txt template
- How to test your robots.txt
- Real-world scenarios we see
- What to do after a mistake
- Staging sites and launches
- FAQs
What robots.txt does and does not do
Robots.txt sits at the root of your domain, for example yourdomain.co.uk/robots.txt. It tells well-behaved crawlers which paths they are allowed to request. Google, Bing and most legitimate crawlers obey it, though malicious bots ignore it.
It does not remove pages from Google. A blocked URL can still appear in results if other sites link to it, usually without a description. To keep a page out of search results you need a noindex tag, and Google must be able to crawl the page to see that tag.
Mistake 1: Blocking the whole site
The line “Disallow: /” under “User-agent: *” blocks every crawler from every page. It is common on staging sites, which is correct there, but disastrous if it is copied to the live site. WordPress also has a setting, “Discourage search engines from indexing this site”, that can add similar blocks.
Check your live file now. If you see a full block and your site is live, remove it and request recrawling. Rankings can recover, but the longer it stays, the more visibility you lose.
Mistake 2: Blocking CSS, JavaScript or images
Google needs to render pages like a browser to understand them. Blocking /wp-content/ or /wp-includes/ wholesale can prevent access to styles and scripts, leading to poor rendering and mobile usability problems.
A safe default for WordPress blocks only the admin area, while allowing the AJAX endpoint that many themes and plugins use. Avoid copying long block lists from old tutorials.
Mistake 3: Using robots.txt to hide sensitive content
Listing private paths in robots.txt tells everyone, including bad actors, exactly where to look. Do not rely on it for security. Protect sensitive areas with authentication and proper permissions, and use noindex or password protection for pages that should not appear in search.
Mistake 4: Blocking pages you want noindexed
If you add noindex to a page and also block it in robots.txt, Google cannot crawl it to see the noindex instruction, so the page may stay indexed. Choose one approach deliberately: allow crawling and use noindex for pages that should drop out of results, and use robots.txt only to save crawl on genuinely useless sections.
Mistake 5: Forgetting the sitemap and using the wrong syntax
- Add a Sitemap line pointing to your XML sitemap so crawlers can find it.
- Use correct paths, with a leading slash and case-sensitive names.
- Put each directive on its own line under the right user-agent group.
- Do not use unsupported directives such as noindex inside robots.txt, which Google ignores.
- Remember that rules for specific user-agents override the general group.
Mistake 6: Blocking AI crawlers by accident
Many security plugins and hosts now add rules for AI crawlers such as GPTBot or CCBot. That may be what you want, but check it was a decision, not a default. If your aim is visibility in AI search, blocking the crawlers of those tools can reduce your chances of being cited.
Decide your policy deliberately and document it. Our guide to generative engine optimisation explains how crawler access fits into AI visibility.
A safe WordPress robots.txt template
A sensible starting point is: User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php, and a Sitemap line pointing to your sitemap index. For WooCommerce sites you may also block cart, checkout and account pages, and parameters that create duplicate URLs.
Keep it short. Every additional rule is a possible mistake. If you are not certain what a line does, leave it out and test.
How to test your robots.txt
- View yourdomain.co.uk/robots.txt in a browser and read every line.
- Use the robots.txt report in Google Search Console to see the version Google fetched and any errors.
- Use the URL Inspection tool to confirm key pages are crawlable.
- Crawl your site with a tool that respects robots.txt and see whether important sections are skipped.
- Re-check after redesigns, migrations and plugin changes.
Real-world scenarios we see
A common case is a Manchester trades business whose new website was built by a freelancer on a staging copy. Two weeks after launch, enquiries fall sharply. The cause: the live robots.txt still contains a full block, so Google stopped crawling new pages and dropped several from results. Removing the block and resubmitting the sitemap restored visibility within a few weeks.
Another is an online shop whose developer blocked all parameter URLs, including the ones used by product filters that customers actually link to. Rankings for filtered category pages vanished because Google could not crawl them. The fix was to allow the filters that had search demand and block only the genuinely useless combinations.
A third is a law firm blog whose security plugin blocked all AI crawlers by default. The firm wanted to be cited by AI search, but its content was invisible to those tools until the setting was reviewed. None of these needed advanced skills to diagnose, just someone reading the file line by line and checking it against the site’s goals.
The pattern is clear: robots.txt problems are usually the result of copying defaults, not malicious errors. A five-minute review every time you change the site, and a clear owner for the file, prevents almost all of them.
What to do after a mistake
If you discover that important pages were blocked, fix the file, then use URL Inspection to request indexing for key pages and resubmit your sitemap. Monitor the Pages report over the following weeks as Google recrawls.
Keep a record of what changed and when, and set a rule that nobody edits robots.txt without testing. Our technical SEO service and website SEO audit both include a robots.txt and indexing review.
Staging sites and launches
The most expensive robots.txt mistakes happen at launch. A site built on a staging domain with a full block is copied to the live domain and the block goes with it. Add a launch checklist step to remove the block, check the WordPress search visibility setting, and test the live robots.txt and a few key URLs within minutes of going live.
Also protect your staging site properly, using password protection, so it is not indexed or crawled. Blocking alone is not enough, because staging URLs can leak into results through links.
Technical SEO Services
Crawling, indexing, speed and schema fixed properly.
Frequently asked questions
Does robots.txt remove a page from Google?
No. It stops crawling, but a blocked URL can still appear in results if other pages link to it. Use noindex to keep a page out of search.
Where should my robots.txt file be located?
At the root of the domain, for example yourdomain.co.uk/robots.txt. Each subdomain needs its own file.
Can robots.txt improve my rankings?
Not directly. Used well, it can help crawlers focus on important pages, but it is mainly a tool for avoiding wasted crawl and protecting low-value areas.
How quickly does Google notice robots.txt changes?
Google typically refreshes the file within a day, but recrawling and reindexing affected pages can take longer.
