The New York Times sues Brave, a search supplier to Claude

The Times's complaint says Brave built its search index by copying "over 30 billion webpages" and that Brave "has never disclosed the name of its 'bot'", so it crawled them "without identifying itself or providing website owners with any mechanism to block its crawler." It says Brave then sold that content through its Search API to AI companies, naming Anthropic, Perplexity, Mistral, xAI and Cohere, and it also targets Brave's own AI answers, Ask Brave, and a paywall blocker in the Brave browser. The Times asks for damages and an order permanently stopping the conduct it alleges.

The Daily News, the Chicago Tribune and the Orlando Sentinel filed a separate complaint in the same court on the same day, naming the same five AI companies. Brave Software is the only defendant in both cases, and nothing has been decided. Both complaints are allegations, and Brave has the chance to answer them in court.

Anthropic's list of sub-processors names Brave Search for "Web Search" across all products. It also names TurboPuffer for web search, in all products except Claude for Government. Anthropic does not publish how Claude's search combines the two, or how either works alongside its own crawlers.

Anthropic runs three crawlers of its own, which it describes on its help page for site owners: ClaudeBot collects content that could contribute to training, Claude-User fetches pages when a user's question calls for it, and Claude-SearchBot improves search results. Anthropic warns that blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results."

Brave's page about its crawler says it "does not advertise a differentiated user agent", so there is no Brave line to write in robots.txt. Instead, "if a domain or page is not crawlable by Googlebot, then Brave Search's bot will not crawl it either." The same page says "robots.txt is not used to prevent a page from being indexed". To take a page out of Brave's results, it points to the noindex directive, which works only once Brave fetches the page again.

So a site that names Claude-SearchBot in robots.txt and lets Googlebot in has told Anthropic's crawler to stay away but given Brave's crawler no instruction beyond the one it reads for Googlebot.

We read robots.txt for the 100 sites Google cited in the AI Overviews from our October reading of 50 UK brand searches, and for the 50 brands' own sites. 82 of the cited sites served a file we could read. Googlebot could reach the homepage on 81 of them; Reddit's file shut out every crawler we tested.

Of those 81, 15 block at least one of Anthropic's crawlers from the homepage, and all 15 block ClaudeBot, the training crawler. Seven also block Claude-SearchBot: bbc.co.uk, bbc.com, facebook.com, theguardian.com, thesun.co.uk and two local news sites. Those seven took 28 of the 314 citations in the three October readings. Five of them block Claude-User as well.

The brands themselves are open. Of the 38 brand sites whose file we could read, only Asda blocks an Anthropic crawler, and that is ClaudeBot. None blocks Claude-SearchBot or Claude-User, and all 38 let Googlebot in, so nothing in their files keeps Brave's crawler out.

If you want Claude to find you, check that Googlebot can crawl your key pages, since that is the rule Brave's crawler follows, and that robots.txt does not block Claude-SearchBot or Claude-User. Then search your brand name and two or three of your buyers' questions on Brave Search to see which of your pages it returns. A page missing there may also be missing from what Claude's search can find.

If you want to stay out, a robots.txt line will not do it for Brave. Its crawler has no name to block, and its page points to the noindex directive to remove a page from its results, a blunt tool for a page you want found elsewhere. Watch the two cases too: if a court limits what Brave may index or sell, the sources Claude's search can reach may change.

To see what AI assistants say about your category today, the free AI visibility check asks buyer questions about your business on Claude, ChatGPT and Google AI Overviews, with no login, and shows who each answer names and the pages it cites.

On 10 November 2026 at 06:10 UTC we fetched robots.txt over https for 117 sites: the 100 that Google cited in AI Overviews when we searched 50 UK brand names on Google UK on 6 October (three readings, 314 citations), plus the 17 brand sites not already among them. A file counted as readable when it returned HTTP 200 as plain text. For each crawler we used the group that names it, or the catch-all group when none does, applied the longest matching rule with ties going to Allow, and tested the homepage path only.

94 of the 117 files were readable. The other 23, including 18 cited sites and 12 brand sites (some are both), returned a refusal, an error or a web page instead, and are left out of every count. Some sites serve a different robots.txt to different visitors, and a rule for a deeper path can differ from the homepage, so these counts describe what our server was sent for the homepage on that morning.