Bing splits AI training from AI answers

Bing now treats two uses of your content separately. Training is whether Microsoft may use it to build its AI models. Grounding is whether it may be used to write web-grounded answers in Bing, Copilot and Microsoft 365. Bing's post says these preferences "do not change eligibility for traditional Bing search results".

Training can be set for a whole site or a path in robots.txt, with a shared token called `__usage_ai-training`, or for a single page with a meta tag such as `<meta name="Bingbot" content="ai-training:disallow">` or the matching X-Robots-Tag header. Grounding works only page by page, through the meta tag or the header. Bing's post says "There is no site-wide robots.txt grounding preference."

Grounding has three settings. Allow is the default. Disallow keeps the page out of grounding altogether. DisallowExtended limits it to Microsoft's own products, such as Copilot and Microsoft 365, and keeps it out of the grounding services Microsoft supplies to other companies, for which Bing names Web IQ and the grounding API in Foundry. Microsoft has described Web IQ as "the same infrastructure powering Copilot, ChatGPT" and others, so a page set to DisallowExtended may also drop out of answers outside Microsoft. Bing is blunt about the cost: "If you select DisallowExtended or Disallow for grounding, you will see reduced brand visibility in AI experiences."

The new preferences replace NoArchive and NoCache, the tags Bing introduced in 2023 for the same purpose. From 15 December 2026, sites using those tags on that date can record a temporary grounding preference in Bing Webmaster Tools. From 1 February 2027, Bing honours the new preferences as it crawls pages. After 1 March 2027, NoArchive and NoCache are no longer read as a training or grounding preference, and pages that carried them default to Allow for grounding, unless a temporary preference was recorded, until Bing recrawls them and finds a new one. Temporary preferences expire on 30 September 2027.

So a site that used NoCache to keep its pages out of Copilot will find them eligible again after 1 March 2027 unless it acts. For most brands the change runs the other way: nothing is needed to stay in AI answers, because Allow is the default for both.

In a Search Central post on 30 October, John Mueller said Google plans to support a shared AI training control in robots.txt, based on the IETF draft draft-kelley-aipref-training-preference-signal, and aims to do so in 2027. The draft is not a finished standard. Google's crawler documentation gives the user agent as `__usage_ai-training` and the timing as early 2027.

If a site uses the shared token to opt out of training and does not mention Google-Extended, Google plans to treat it as if Google-Extended were disallowed. That matters because Google-Extended covers more than training: the same documentation says it also controls grounding in Gemini Apps, and Google's post says sites that "specifically allow Google-Extended are eligible to appear in responses" there. The documentation also says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." For AI Overviews and AI Mode, Google's page on AI features points to the usual Search controls instead: nosnippet, data-nosnippet, max-snippet and noindex.

We took the 78 AI Overviews from our Google UK brand search reading of 6 October, which cited 100 different domains 314 times, and on 4 November read each domain's robots.txt. We could read 89. Thirteen of them name Google-Extended and disallow it for the whole site, among them LinkedIn, Facebook, Yahoo Finance and the BBC. Between them they took 55 of the 314 citations and appeared in 37 of the 78 overviews. Fifteen of the 89 block OpenAI's training crawler, GPTBot, the same way. None yet uses the `__usage_ai-training` token, which no provider supports today.

The 50 brands themselves block far less. Of the 43 brand sites whose robots.txt we could read, Asda was the only one to block Google-Extended, and it blocks GPTBot and Anthropic's ClaudeBot too. We also loaded each brand's homepage and looked for NoArchive or NoCache in a robots or Bingbot meta tag or the X-Robots-Tag header. None of the 41 homepages that loaded properly carried either.

This is a small, descriptive check of brand-name searches on one engine, and the robots.txt files were read four weeks after the overviews were recorded. It is consistent with what Google's documentation says: blocking Google-Extended does not stop a site being cited in AI Overviews.

If you want AI assistants to recommend you, leave grounding on Allow, which needs no change, and do not set DisallowExtended or Disallow on the pages you want quoted, such as product, pricing and comparison pages. If you want to keep your content out of model training, you can now do that in Bing without leaving its AI answers. When the shared token arrives at Google, add an explicit `Allow` for Google-Extended if you want to stay eligible for grounding in Gemini Apps.

Search your templates and CDN settings for NoArchive and NoCache. If you find them and still want those pages out of Bing's AI answers, record a temporary preference in Bing Webmaster Tools from 15 December and set the new page-level grounding tag before 1 March 2027.

Our free AI crawler checker reads your robots.txt and shows, crawler by crawler, which AI crawlers it blocks, Google-Extended and Bingbot among them. Our Copilot visibility tracker page explains how Discoverable reads Copilot answers.

The cited sites and the 50 brands come from our 6 October 2026 reading, in which we searched the names of 50 UK brands on Google UK three times each, logged out, and recorded the domains each AI Overview cited; that reading's CSV is free to download from its page. On 4 November 2026, at about 06:07 UK time, we requested https:// plus each domain and /robots.txt once, following redirects, and loaded each brand's homepage once. A file counted as readable when it returned a non-empty text file; access-denied pages, bot checks and errors were counted as unreadable. A site counted as blocking a crawler when its robots.txt named that crawler in a user-agent line and gave it `Disallow: /` with no `Allow: /`. Homepages that returned an error or a bot-check page were left out. We checked each site once, so a site may have changed its file since the overviews were recorded.