Free tool
Is your site blocking AI crawlers?
Enter your domain to see whether your robots.txt blocks GPTBot, ClaudeBot, PerplexityBot and 9 other AI crawlers, and whether you publish an llms.txt. Sites that block these crawlers cannot be read by the AI assistants buyers now ask for recommendations, and many block them by accident.
Why crawler access decides your AI visibility
Every AI answer about your category starts with what the engine can read. Crawlers like OAI-SearchBot and PerplexityBot build the indexes those engines search, and live fetchers like ChatGPT-User and Claude-User open your pages mid-answer to pull pricing, features and facts. A robots.txt rule that blocks them removes you from that entire pipeline: the assistant simply builds its answer from competitors and third-party pages instead. Our research found AI assistants already misdescribe most brands from memory (27 of 30 in our July 2026 study), and a blocked crawler takes away the engine's ability to correct itself from your own site.
Accidental blocking is common. Some CDN and security products ship bot-protection presets that deny every non-Google crawler. Some robots.txt files carry a blanket Disallow: / from a staging setup that was never removed. If the table above shows red for crawlers you never chose to block, your CDN settings or robots.txt history is the first place to look. The fix is usually a two-line change: GEO starts with being readable.
Access is necessary but not sufficient. Being crawlable gets you into the pipeline; being recommended depends on what the sources the engines trust say about you. That second half is what the free AI visibility checker measures.
AI crawler questions, answered
How do I check if my site blocks GPTBot or ClaudeBot?
Enter your domain above. The tool fetches your robots.txt and evaluates the access rules for 12 AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, exactly the way the crawlers themselves read the file. You see an allowed, partial or blocked verdict per crawler.
Should I block AI crawlers or allow them?
It depends on what you sell. If you want buyers to find and choose you through AI assistants, blocking their crawlers makes you invisible at the exact moment a buyer asks for recommendations. Publishers who monetise content sometimes choose to block trainers like GPTBot while allowing search-focused crawlers. Many businesses block everything by accident through CDN defaults and never know.
What is the difference between GPTBot and ChatGPT-User?
GPTBot crawls pages in bulk to train OpenAI's models. ChatGPT-User fetches a specific page live when ChatGPT needs it to answer a user's question. Blocking GPTBot limits training exposure; blocking ChatGPT-User stops ChatGPT reading your site in real time, which directly hurts your presence in answers.
Does blocking Google-Extended remove me from Google AI Overviews?
No. Google-Extended only controls whether Google may use your pages to train Gemini models. AI Overviews are built from Google's normal search index via Googlebot, so blocking Google-Extended does not remove you from AI Overviews, and blocking Googlebot would remove you from Google Search entirely.
What is llms.txt and does this tool check it?
llms.txt is a proposed markdown file at yourdomain.com/llms.txt that gives AI systems a curated map of your most important pages. The tool checks whether you have one. Adoption by engines is still unconfirmed, but the file costs minutes to create and cannot hurt.