AI Search
Should your website block AI crawlers?
How to decide which AI crawlers should reach your website, what each choice changes and what proof to ask a web expert to show.
If you’re asking whether to block AI crawlers on your website, the answer is more involved than it may seem because different bots can use the same page for different jobs. One may collect public content for model training, another may help an AI search tool find sources, and a third may fetch a page because a person asked a question. Blocking all three can therefore produce very different results.
Start with the use you want to allow or restrict, not the letters ‘AI’ in a bot’s name. Let’s say you want a public service page to be found but paid research, licensed work or an unfinished preview may need tighter rules. A specialist can turn that choice into the right site settings and show whether they work.
AI crawlers don’t all do the same job
A crawler is software that requests pages automatically, but the reason for each request is what matters when you decide whether to allow it. The controls published by AI companies cover at least three different uses: model training, finding sources for AI search and retrieving a page after a user asks for it.
Model training
Training controls cover content that may help develop future models. OpenAI’s publisher and developer guidance explains that GPTBot can be excluded from pages a publisher doesn’t want used for possible training, while Anthropic’s web-crawling guidance describes ClaudeBot as a crawler that collects web content which could help train its models.
Google’s version needs extra care because Google-Extended is a control token, not a bot that appears in server logs. Google’s documentation for Google-Extended explains that it controls whether content already crawled by Google may help train future Gemini models and ground some other Gemini tools, without affecting inclusion or ranking in Google Search.
The decision is whether you want that company to use a particular set of public pages for model training, which may depend on licences, promises to customers and the value of the work. It isn’t the same choice as whether you want those pages to appear in search.
AI search
Search crawlers help an AI tool find and save sources for its answers. OpenAI’s guidance advises allowing OAI-SearchBot when a publisher wants its content to remain eligible for ChatGPT summaries and snippets, while Anthropic’s guidance explains that blocking Claude-SearchBot may reduce a site’s visibility in user search results.
Google Search works differently because its AI features guidance treats AI Overviews and AI Mode as part of Search. Googlebot and the usual Search preview rules therefore govern access and how pages appear, so blocking Google-Extended doesn’t opt a page out of Google’s AI search features.
Blocking a search crawler may reduce the chance of that service finding, summing up or linking to your pages, whereas allowing it only makes those pages eligible and doesn’t promise a source link, visit or enquiry.
Fetching a page when a user asks
Some AI services use a separate bot when a person asks an AI assistant to open or use a particular page. For example, Anthropic identifies Claude-User as its user-requested bot and warns that blocking it can reduce reach in user-led web search. The searchDecoded crawler study also analysed ChatGPT-User and Perplexity-User separately for the same reason and found that both retrieve pages in response to a user’s request rather than as part of a planned model-training crawl.
This can matter when a customer asks an AI assistant to compare a service, read a help page or use facts from your site. You may still block that access, but it should be a clear choice rather than a side effect of one blanket AI rule.
The clearest published policies separate AI uses
searchDecoded examined the robots rules of 100 prominent UK consumer websites across six sectors in August 2026. Only 13 named at least one of the eight AI crawler or content-use controls in the main study, while 72 let those controls inherit the site’s general robots rules. The dominant published approach was therefore not a considered ‘allow AI’ or ‘block AI’ choice, but the result of rules that were already in place for automated access more broadly.
The sites that did name AI controls show why a more deliberate policy can be useful. Five of the 13 named Applebot-Extended, ClaudeBot, GPTBot and Google-Extended together, which brought several training and content-use decisions into view rather than leaving them to a general rule. More importantly for this article, two sites allowed the relevant OpenAI and Anthropic search crawlers while blocking the matching training crawlers across the whole site.
That split is a practical example of the mixed policy this article recommends. A business can remain available to an AI search service while setting a different rule for model training, provided the controls are mapped and tested correctly. The study recorded published instructions rather than testing whether every bot obeyed them, but its findings show that a blanket decision isn’t the only workable option.
Four questions to answer before changing anything
You don’t need to name every user-agent or edit robots.txt yourself, but you do need to give a specialist a clear brief. Start with four questions:
- Which content are you deciding about? Split public sales pages from paid, licensed, private, copied, unfinished or dated work. You rarely need a whole-site choice when the concern applies to one area.
- Which use are you happy with? Decide in turn about model training, AI search and a fetch started by a user. ‘AI access’ is too broad to guide a site change.
- What result could access help or harm? Benefits may include source links and good visits, while costs may include unwanted use, server load, stale answers or a clash with a licence.
- What proof will show whether the rule worked? Ask for the exact change, the pages it affects, a test result, server logs where they exist and a rollback plan.
These questions keep the owner’s role in the right place because you set the sales and content limits, while a search expert, developer or web host finds the right control, tests the change and watches for side effects.
The AI crawler access briefing template turns those decisions into a record you can send to the person making the change. It keeps each type of access separate, captures the affected content and sets out the evidence, rollback and monitoring you expect.
Build an AI crawler access brief Record what you want to allow or restrict, then give a specialist a clear evidence and rollback checklist.When blocking an AI crawler can make sense
Blocking can make sense when a bot’s job clashes with a clear rule or causes a measured problem. Examples include:
- keeping paid or licensed content out of a company’s model-training crawl;
- keeping previews, test sites or copied parameter pages out of an automatic fetch;
- restricting a crawler that creates heavy server load without a clear gain;
- enforcing a content or contract rule that the right legal or sales adviser has checked;
- stopping an unknown or rule-breaking bot after server logs show what it asks for.
The last case is a site protection problem rather than just a robots.txt task because anyone can copy a name in a user-agent string. Before blocking traffic at the server or firewall, the person responsible should check how the company proves that a bot is its own and make sure the rule won’t catch customers, accessibility tools or normal search crawlers.
When allowing access can make sense
Access may help when your site exists to make services, products, places or skills easy to find. Allowing a search crawler can keep a page eligible to support an AI answer, while a user-request bot can help an AI assistant fetch current facts when someone asks for them.
That value still needs proof. OpenAI says referral links from ChatGPT include a utm_source=chatgpt.com parameter, so analytics may show visits from a source link. searchDecoded’s AI citation measurement guide explains why a mention, source link and sale are three forms of proof. Google’s AI search features have their own eligibility rules, they aren’t model-training controls.
For many public sites, a mixed rule is a sound starting point: keep helpful sales pages open to search and user-led fetching, then make a separate choice about model training. It won’t suit every website, but it stops one goal being lost while you deal with another.
A robots.txt rule is a request rather than a lock
The Robots Exclusion Protocol is the standard behind robots.txt, and its specification explains that the rules ask crawlers to honour your wishes rather than acting as access control or a lock. The file is public, so don’t list a secret path there in the hope of hiding it.
Major firms publish rules for their own bots, but a less honest crawler may ignore them. Cloudflare’s managed robots guidance makes the same distinction: robots.txt states your wishes, whereas a firm block needs another site control. Other web hosts may offer a similar tool.
A firm block brings its own risks because a broad firewall rule can stop the wrong traffic, while a bot that hides its name may evade a simple user-agent check. This is why an old ‘block all AI bots’ list is unsafe: bot names, jobs and proof methods can change, so the final rule needs current guidance and a test on your site.
What a web expert should show you
Sound advice should be easy to review before it goes live. Ask the person making the change to provide:
- the goal the rule supports;
- the exact bot control and its stated role;
- the pages or sections affected, including any exceptions;
- the current rule and proposed change as a clear before-and-after comparison;
- a test showing how the final file is read;
- server or protection logs when bot traffic or rule-following is part of the reason;
- the expected effect on training, search reach or a user-led fetch;
- a rollback and monitoring plan.
They should also explain what the proof can’t show. For example, a fall in bot requests doesn’t prove that old content has been removed from a model, and an AI visit doesn’t prove that one bot setting caused it. If no visits appear, the reason may be low demand, missing tracking or the absence of a source link rather than a broken rule.
If someone recommends a single rule without first asking what you want to protect or keep visible, they have chosen a technical answer before understanding the business decision. That can block useful access, leave the wrong content exposed or create a change nobody can justify later.
Choose the use before you choose the control
Instead of judging AI crawlers as one group, decide what you want each crawler to do with each type of content and where you need to set a boundary.
Write that rule in everyday words first by naming the content, the use you want to allow or restrict and the proof you expect later. A specialist can then map it to current bot controls, test the result and watch for side effects.
You may end with some bots allowed, some blocked and some paths treated in their own way. Although that is less dramatic than a sitewide switch, it gives you a choice you can explain and a change you can check.
