What is an AI crawler?
An AI crawler is an automated bot run by an AI company to fetch web pages. The term covers three different jobs: collecting training data for models, building the search indexes behind AI answers, and fetching pages live when a user asks an assistant a question.
AI crawler is an umbrella term rather than the name of a single bot. OpenAI alone operates GPTBot for training data, OAI-SearchBot for its search index and ChatGPT-User for live page fetches, and Anthropic, Google, Perplexity and Meta each run their own equivalents. Treating them all as one thing is where most robots.txt mistakes start, because the sensible decision differs depending on what each crawler is for.
The three categories do different jobs. Training crawlers such as GPTBot and ClaudeBot collect pages for the datasets used to train future models. Google-Extended belongs in this category too, although it is a control token read by Google's existing crawlers rather than a separate bot. Search-index crawlers such as OAI-SearchBot and PerplexityBot build the retrieval indexes that AI search products consult when composing answers. User-fetchers such as ChatGPT-User and Claude-User request a page in real time because someone has just asked a question the assistant wants to answer from the live web. A site that blocks this third group cannot have its pages read during those live lookups.
Each of these bots identifies itself with a user agent token, and robots.txt is the standard mechanism for allowing or refusing them individually. Compliance is voluntary. The major operators publish their tokens and say they honour the file, though compliance has been disputed for some, so robots.txt is best treated as a clear signal rather than a hard barrier. The trade-offs also differ by category. Blocking training crawlers keeps newly crawled content out of future training datasets at little immediate cost to visibility. Blocking search-index crawlers and user-fetchers takes your pages out of the material AI engines can retrieve, which leaves them describing your brand from memory. Discoverable's July 2026 study of 30 brands measured what that looks like: 27 of 30 (90 percent) were misdescribed when engines answered from memory, compared with 13 of 30 (43 percent) when live web search was available.
In practice the useful question is not whether to block AI crawlers but which ones. Many publishers refuse training bots while leaving retrieval and user-fetching bots open, which protects their content from training without giving up presence in AI answers. It is worth checking what your robots.txt actually does, since blanket rules added during the first wave of AI blocking in 2023 often still catch retrieval bots that did not exist then. Discoverable's free AI crawler checker tests a domain's robots.txt against 12 AI crawlers and checks for an llms.txt file, so you can see exactly which categories you are letting in.