Free tool
Set your AI crawler policy in seconds.
Choose which of the 12 AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot and more) may read your site, and get append-ready robots.txt rules. Three honest presets: full visibility, block training but stay citable, or block everything.
Per-crawler control
Your robots.txt rules
Append these rules to your existing robots.txt (do not replace it), deploy, then verify with the crawler checker.
Pick a stance deliberately, not by accident
Most sites have never made an explicit decision about AI crawlers: they inherit whatever their CMS, CDN or a copied robots.txt did, and many block everything without knowing it. The stakes are asymmetric for a business: an accidental block removes you from the answers buyers now read, while allowing access costs nothing. Check what your current file actually does with the free crawler checker before and after you deploy new rules.
robots.txt is one of three files in your AI-access stack. It controls who may read. An llms.txt suggests what to read first. And your actual pages decide what AI can quote. None of them changes what third-party sources say about you, which is where most AI answers come from; you can see that picture for your own brand in about a minute with the free visibility check below.
AI robots.txt: quick answers
Should I block AI crawlers in robots.txt?
It depends on what your site is for. If you sell something, blocking AI crawlers makes you invisible at the exact moment buyers ask assistants for recommendations, so most businesses should allow them. Publishers who monetise content sometimes block training crawlers like GPTBot and CCBot while leaving answer-time crawlers open so they stay citable. The generator has a preset for each stance.
What is the difference between training crawlers and answer crawlers?
Training crawlers (GPTBot, CCBot, Google-Extended, Applebot-Extended, ClaudeBot) gather pages to train future models. Answer crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-User) build the search indexes behind AI answers or fetch your pages live while answering a user. Blocking the first group limits training exposure; blocking the second removes you from AI answers today.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended only controls Gemini training. AI Overviews are built from Google's normal search index via Googlebot, so blocking Google-Extended leaves them untouched, and blocking Googlebot would remove you from Google Search entirely.
Why does the generator warn me about Bingbot?
Bingbot is Microsoft's main search crawler. Its index feeds Copilot and parts of ChatGPT search, but it is also how you appear in ordinary Bing results. Blocking it removes you from both, which is almost never what a business wants, so the block-all preset deliberately leaves it allowed.
Where do these rules go?
Append them to the robots.txt at your web root (yourdomain.com/robots.txt), keeping any rules you already have. Crawlers read the file top to bottom and use the most specific user-agent group that matches them. After deploying, run our free crawler checker to confirm each bot sees the rule you intended.
Crawler access is step one. What does AI say about you?
Run a free AI visibility check and see whether assistants name your brand, in under a minute.