Owner-to-specialist template
Build an AI crawler access brief
Record what different AI crawlers may do with your content, then give the person making the change a clear scope, evidence checklist and rollback requirement.
Start with the business choice
Decide by purpose rather than bot names
You don't need to know every user-agent before using this template. Define which content is in scope and what you want each type of access to achieve, then ask a web specialist to map those choices to current controls and test the result.
Scope and reason
Record the boundaries before the settings
Separate public sales pages from paid, licensed, private or unfinished content. A precise scope can avoid a whole-site block when the concern applies to only one area.
Model training
May companies use the pages in scope to help train future models?
Treat this as a content-use decision that can differ from your search visibility decision.
AI search
May AI search services find and use the pages as sources for answers?
Allowing access can support eligibility, but it cannot guarantee a source link, visit or enquiry.
User-requested fetching
May an assistant fetch a page after a person asks it to use your website?
This request happens in response to a user and should not be bundled automatically with model training.
Unknown or unverified bots
How should the site treat automated traffic that cannot be verified or causes a measured problem?
A server or firewall response needs current verification evidence so it does not block customers or legitimate crawlers.
Specialist hand-off
Add the controls and exceptions already known
Leave this section open when you don't know the technical answer. The specialist should identify the current bot names, applicable rules and any stronger server controls rather than asking you to guess.
Evidence before launch
Ask for a change you can review and reverse
Mark each point only when the specialist has supplied the evidence. These checks don't score the policy; they make the proposed change easier to understand, approve and undo if it catches the wrong traffic.
Generated hand-off
Copy the brief for your web specialist
The brief is generated from your entries and stays in this browser until you reset it.