Information correct as of 2 November 2026. No company has yet said the new controls are in effect.
A two-line entry in robots.txt will soon tell several AI companies at once that a site does not want its content used to train their models. Microsoft is adding a second, separate choice about whether its AI answers may draw on a page, and for most UK firms that second choice is the one to leave as it is.
On 30 October 2026 Google and Microsoft both announced support for a shared AI training opt-out, written as a robots.txt group headed User-agent: __usage_ai-training. Google’s crawler documentation puts its support in early 2027, and Google plans to read the opt-out as a Google-Extended disallow unless the file names Google-Extended. Microsoft starts reading it on 1 February 2027. It is also letting sites set training preferences page by page and adding a separate AI Grounding preference for its AI answers; together they replace the NoArchive and NoCache tags it has used as AI controls since 2023. OpenAI said the same day that it aims to honour the signal “in the coming months”, and Anthropic, where the draft’s author works, says it has started implementing it.
For a law or accountancy firm worried about its published guidance being reused, that turns one decision into two, with different prices. Blocking grounding outright stops a firm’s tagged pages informing the answers that Copilot, Microsoft 365 and Bing’s AI features give to people looking for advice, and Microsoft warns that this sharply cuts their visibility. Blocking training through the shared entry carries no such warning from Microsoft, but once Google applies the signal, the opted-out pages also leave Gemini’s grounded answers unless a separate Google-Extended group allows them. Most firms should keep AI answers open, and block training only for a clear reason.
Four AI companies plan to read one shared training opt-out
The entry uses what its draft calls a policy token: a name in the User-agent line that stands for a use of content and matches no crawler. Under User-agent: __usage_ai-training, a Disallow line records a preference against AI training for the paths it matches, and an Allow line records a preference in favour. A path that no line matches also counts in favour. Normal robots.txt matching applies, so the longest matching path wins, and the draft’s own example opts a whole site out while leaving its press kit open.
Training here means “using an asset to modify the learned parameters of a generative AI model”, a definition taken from the IETF’s AI Preferences working group. Models used only for jobs such as classification or ranking fall outside it. The mechanism itself comes from an individual Internet-Draft that Kevin Kelley of Anthropic submitted on 30 October. It carries no official status in the IETF’s standards process yet, and it lapses on 3 May 2027 unless a new version replaces it.
A User-agent: * group never counts as the signal, so a robots.txt file without a __usage_ai-training group expresses no preference through the shared signal. The entry is about how content is used and does not block crawling: each crawler still follows the rules written for its own name. It also looks forward only. The draft ties each page to the robots.txt file in force when it was fetched, so an opt-out does not cover copies collected before it was published, and it cannot undo training that has already happened.
Microsoft will honour the entry from 1 February 2027, and Google plans to from early 2027. OpenAI says the shared signal will sit alongside its existing robots.txt controls, so GPTBot rules still apply, and Kelley told the working group that Anthropic’s support across its services will take some time. The draft lets each company treat its own token as more specific than the shared one. Any crawler whose operator has not adopted the token still needs its own group; our guide to which AI crawlers robots.txt can block lists them.
At Google, the opt-out also covers Gemini’s answers
John Mueller’s post on Google Search Central says that when a site opts out through the shared token and does not name Google-Extended, Google plans to treat the file “as if the publisher had disallowed Google-Extended”. Google’s crawler documentation gives the timing: support is planned for early 2027.
That matters because Google-Extended is more than a training control. According to the same documentation, it decides whether content Google crawls may train future Gemini models and whether it may ground answers in Gemini Apps and in Grounding with Google Search on Vertex AI. Once Google applies the signal, pages opted out through the shared entry alone will therefore leave Gemini’s grounded answers as well as its training data. Mueller’s post confirms the other side: sites that explicitly allow Google-Extended stay eligible to appear in Gemini Apps and receive clicks from it.
At Google, then, Gemini training and Gemini’s answers still sit behind one switch. A site can opt out of training elsewhere and stay in Gemini by giving Google-Extended its own group with Allow: /, and that also lets Gemini train on it.
A site’s place in Google Search is unaffected either way. Google-Extended plays no part in inclusion or ranking there, and AI Overviews, AI Mode and Discover’s AI features have their own setting: the Search generative AI control in Search Console, open to every site since 31 August 2026, which does not touch training. Our guide to whether to opt out of Google AI Overviews covers that decision.
Bing gives AI answers a separate switch
Microsoft calls the new preference AI Grounding: using a page as input when an AI answer is generated. Ranking pages in ordinary Bing results does not count. It can be set only on the page, with a meta tag or an X-Robots-Tag HTTP header, and it takes three values. Allow is the default and needs no tag. DisallowExtended keeps a page in Microsoft’s own AI experiences, which include AI Mode, AI Overview, Copilot and Microsoft 365, and out of the grounding it provides to other companies through Web IQ and Microsoft Foundry. Disallow keeps it out of both.
For training, Microsoft follows the shared robots.txt entry and adds a page-level version that overrides robots.txt for that page. The two preferences are independent. Microsoft’s FAQ says the old NoArchive and NoCache tags “treated training and grounding as a single decision”; from 1 February a firm can refuse training and still let Copilot cite its pages.
Microsoft is blunt about what refusing grounding costs. Its announcement warns that choosing Disallow or DisallowExtended “will significantly reduce your content’s visibility in AI-generated experiences”. Neither preference changes eligibility for ordinary Bing search results, Microsoft says, and leaving them open guarantees no inclusion, traffic or payment. Both depend on recrawling, too: a tag takes effect only when Bingbot next fetches the page.
If the table extends beyond the screen, scroll sideways to view all columns.
| Control | Where it is set | What it covers | When it applies |
|---|---|---|---|
__usage_ai-training |
Where it is setrobots.txt, whole site or by path | What it coversGenerative AI training by companies that adopt it; at Google, Gemini grounding as well, unless Google-Extended has its own group | When it appliesMicrosoft from 1 February 2027; Google planned for early 2027; OpenAI aims for the coming months; Anthropic not yet dated |
Bing ai-training |
Where it is setMeta tag or X-Robots-Tag, per page |
What it coversMicrosoft’s training; overrides robots.txt for that page | When it appliesFrom 1 February 2027 |
Bing ai-grounding |
Where it is setMeta tag or X-Robots-Tag, per page only |
What it coversMicrosoft’s AI answers and the grounding it provides to others; disallowextended blocks only the second |
When it appliesFrom 1 February 2027 |
| Google-Extended | Where it is setrobots.txt | What it coversGemini training, plus grounding in Gemini Apps and Vertex AI | When it appliesNow |
| Search generative AI control | Where it is setSearch Console | What it coversAI Overviews, AI Mode and Discover’s AI features | When it appliesNow |
Sites using NoArchive or NoCache have until 1 March 2027
Microsoft’s deadline matters only to sites that use NoArchive or NoCache today and want to keep their restrictions. For everyone else, both new preferences default to Allow, and nothing changes unless the site adds a tag or a training group to robots.txt.
Bing Webmaster Tools opens a temporary grounding preference on 15 December 2026, but only for sites using either tag on that date. Microsoft starts honouring the new tags on 1 February 2027 as it recrawls. A site using either tag that wants to stay out of grounding must publish the new AI Grounding tag on its pages and record the temporary preference before 1 March 2027, because Microsoft may not recrawl every page between 1 February and 1 March. The old tags can stop counting sooner. Microsoft’s announcement says it will disregard them on a site as soon as it finds any new preference there, while its FAQ limits that to the use the new preference covers, so a site that wants to keep its restrictions should move training and grounding to the new preferences at the same time. After 1 March they stop counting for training or grounding anywhere, and a page with no temporary preference on record defaults to Allow for grounding until Bingbot returns and finds its new tag.
The temporary preference lasts until that recrawl or 30 September 2027, whichever comes first. There is no temporary setting for training: a training preference takes effect when Microsoft next crawls the robots.txt file or page that carries it, and IndexNow can prompt a recrawl without guaranteeing a date. NoArchive and NoCache then go back to their original job in Bing search, which was controlling cached copies of pages.
Should law and accountancy firms block AI training?
Most do not need to, and almost none should block AI answers. A firm’s guides and partner profiles exist to be found by people with a problem. Grounding is how those pages turn up, with a link, when the same person asks Copilot or Microsoft 365 instead of a search box, and Microsoft’s own warning spells out what refusing it costs.
Training is a weaker lever. It shapes what future models know, without the links and citations that grounding brings, and under the draft an opt-out only reaches content collected after it is published.
The strongest case for a training opt-out belongs to firms whose published know-how is itself something they sell, such as a technical library or paid training material. For them the shared entry is the right tool, applied only to the folders that hold that material. They still have to decide what Google gets: with the shared entry alone, Gemini will stop drawing on those pages for its answers once Google applies the signal, and with an explicit Google-Extended allow, Gemini can train on them as well.
None of these settings protects anything confidential. They are preferences that the participating companies have said they will honour, so material that must not be reused belongs behind a login.
What should UK marketing teams do next?
Only the first step has a hard deadline. The rest can be settled in one meeting with whoever signs off the website.
- Look for NoArchive and NoCache. Check the live HTML and response headers of a few key pages and PDFs for
noarchiveornocache. A plugin or an old template may have added one without anyone deciding to restrict AI use. If the restriction was deliberate, record the temporary grounding preference in Bing Webmaster Tools once it opens on 15 December 2026, so that it is in place before Microsoft starts reading new tags on 1 February, and publish the new training and grounding preferences together before 1 March 2027. - Leave AI answers open unless a page must stay out. No tag is needed to allow grounding. For a page that must stay out of Microsoft’s AI answers, add the meta tag below, or use
ai-grounding:disallowextendedto stay in Copilot and Microsoft 365 while leaving third-party grounding.
PDFs cannot carry meta tags, so send the same value as an HTTP header:<meta name="Bingbot" content="ai-grounding:disallow">X-Robots-Tag: Bingbot: ai-grounding:disallow - Limit any training opt-out to the material that needs it. A firm protecting a know-how library might publish this group, which covers every URL that starts with
/knowledge-library/:
The group also records a preference in favour of training for every other path on the site.User-agent: __usage_ai-training Disallow: /knowledge-library/ - Repeat the rule for Google-Extended. Google offers no way to refuse Gemini training on a page while keeping it in Gemini’s grounded answers, so the library either stays out of both or goes into both. Google-Extended already works, so this group applies once Google has read the updated file, without waiting for Google’s support for the shared entry:
If the file already has a Google-Extended group, check how the two combine: robots.txt merges groups for the same name, and the most specific rule wins. Replacing the disallow withUser-agent: Google-Extended Disallow: /knowledge-library/Allow: /keeps the library in Gemini’s answers and lets Gemini train on it. - Keep the company tokens. Groups for GPTBot, ClaudeBot and other named crawlers keep working, and crawlers whose operators have not adopted the shared token still need them.
- Take a baseline before 1 February 2027. Note your citations in the AI Performance report in Bing Webmaster Tools now, so any change after Bingbot recrawls pages with new tags shows up against real numbers.
What to watch next
Google’s crawler documentation is where the gap with Microsoft would close. If Google gives Gemini grounding a control of its own, or narrows what the shared opt-out switches off, the advice for Google changes; until then, early 2027 is the only date Google has given. OpenAI has promised a further update when it implements the signal, and Anthropic has not put a date on its support.
In the UK, the CMA’s conduct requirement of 3 June 2026 gave Google nine months to put its publisher controls in place, including an opt-out from content being used to fine-tune its AI models. Watch whether Google presents its early-2027 support for the shared signal as part of meeting that requirement.
At the IETF, the draft lapses on 3 May 2027 unless a new version appears, and Kelley has offered it as input to the working group’s own work on attaching preferences to content. Microsoft has promised advance notice if the syntax changes as the standard is finalised, and Bing Webmaster Tools should show the temporary grounding preference to eligible sites from 15 December.
The real lesson from Google and Bing’s AI training announcements
The shared token is meant to settle one question, training, in one place. Whether AI answers may quote a firm’s pages is still decided company by company. OpenAI and Anthropic handle it through separate search crawlers, Microsoft is giving it a switch of its own, and Google does the same for AI Overviews and AI Mode while still tying Gemini’s answers to Gemini training. For a firm that sells advice, the quoting side is the one that can bring in new clients, so the default is worth keeping, and any training opt-out deserves a narrower scope than a whole site.
Our guide to robots.txt and AI crawlers has the company tokens the shared entry does not replace, and our guide on making a business easier for AI search to find, trust and mention covers how firms earn those citations in the first place.
Talk to AIWIZ for a check of your robots.txt file and page headers before Microsoft’s 1 March deadline.