13 November 2026

Quick answer A robots.txt file tells crawlers which parts of your site they may access. The most damaging mistakes are blocking the whole site with Disallow: /, blocking CSS and JavaScript, confusing blocking with noindex, and leaving a staging rule live after launch. Test any change in Search Console before and after publishing. Last updated: November 2026 A robots.txt file is a tiny text file with outsized power. One misplaced line can stop Google crawling your whole website, and it happens more often than most owners realise, usually after a redesign or when a developer forgets to remove a setting. Robots.txt is also misunderstood. It controls crawling, not indexing, and it is not a security tool. Treating it as either can cause problems. This guide lists the mistakes we see most often on UK business sites, explains how to check yours in minutes, and gives a sensible WordPress template you can adapt. In this guide What robots.txt does and does not do Mistake 1: Blocking the whole site Mistake 2: Blocking CSS, JavaScript or images Mistake 3: Using robots.txt to hide sensitive content Mistake 4: Blocking pages you want noindexed Mistake 5: Forgetting the sitemap and using the wrong syntax Mistake 6: Blocking AI crawlers by accident A safe WordPress robots.txt template How to test your robots.txt Real-world scenarios we see What to do after a mistake Staging sites and launches FAQs What robots.txt does and does not do Robots.txt sits at the root of your domain, for example yourdomain.co.uk/robots.txt. It tells well-behaved crawlers which paths they are allowed to request. Google, Bing and most legitimate crawlers obey it, though malicious bots ignore it. It does not remove pages from Google. A blocked URL can still appear in results if other sites link to it, usually without a description. To keep a page out of search results you need a noindex tag, and Google must be able to crawl the page to see that tag. Mistake 1: Blocking the whole site The line “Disallow: /” under “User-agent: *” blocks every crawler from every page. It is common on staging sites, which is correct there, but disastrous if it is copied to the live site. WordPress also has a setting, “Discourage search engines from indexing this site”, that can add similar blocks. Check your live file now. If you see a full block and your site is live, remove it and request recrawling. Rankings can recover, but the longer it stays, the more visibility you lose. Mistake 2: Blocking CSS, JavaScript or images Google needs to render pages like a browser to understand them. Blocking /wp-content/ or /wp-includes/ wholesale can prevent access to styles and scripts, leading to poor rendering and mobile usability problems. A safe default for WordPress blocks only the admin area, while allowing the AJAX endpoint that many themes and plugins use. Avoid copying long block lists from old tutorials. Mistake 3: Using robots.txt to hide sensitive content Listing private paths in robots.txt tells everyone, including bad actors, exactly where to look. Do not rely on it for security. Protect sensitive areas with authentication and proper permissions, and use noindex or password protection for pages that should not appear in search. Mistake 4: Blocking pages you want noindexed If you add noindex to a page and also block it in robots.txt, Google cannot crawl it to see the noindex instruction, so the page may stay indexed. Choose one approach deliberately: allow crawling and use noindex for pages that should drop out of results, and use robots.txt only to save crawl on genuinely useless sections. Mistake 5: Forgetting the sitemap and using the wrong syntax Add a Sitemap line pointing to your XML sitemap so crawlers can find it. Use correct paths, with a leading slash and case-sensitive names. Put each directive on its own line under the right user-agent group. Do not use unsupported directives such as noindex inside robots.txt, which Google ignores. Remember that rules for specific user-agents override the general group. Mistake 6: Blocking AI crawlers by accident Many security plugins and hosts now add rules for AI crawlers such as GPTBot or CCBot. That may be what you want, but check it was a decision, not a default. If your aim is visibility in AI search, blocking the crawlers of those tools can reduce your chances of being cited. Decide your policy deliberately and document it. Our guide to generative engine optimisation explains how crawler access fits into AI visibility. A safe WordPress robots.txt template A sensible starting point is: User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php, and a Sitemap line pointing to your sitemap index. For WooCommerce sites you may also block cart, checkout and account pages, and parameters that create duplicate URLs. Keep it short. Every additional rule is a possible mistake. If you are not certain what a line does, leave it out and test. How to test your robots.txt View yourdomain.co.uk/robots.txt in a browser and read every line. Use the robots.txt report in Google Search Console to see the version Google fetched and any errors. Use the URL Inspection tool to confirm key pages are crawlable. Crawl your site with a tool that respects robots.txt and see whether important sections are skipped. Re-check after redesigns, migrations and plugin changes. Real-world scenarios we see A common case is a Manchester trades business whose new website was built by a freelancer on a staging copy. Two weeks after launch, enquiries fall sharply. The cause: the live robots.txt still contains a full block, so Google stopped crawling new pages and dropped several from results. Removing the block and resubmitting the sitemap restored visibility within a few weeks. Another is an online shop whose developer blocked all parameter URLs, including the ones used by product filters that customers actually link to. Rankings for filtered category pages vanished because Google could not crawl them. The fix was to allow the filters that had search demand and block only the genuinely useless combinations. A third is a