Google, Microsoft, OpenAI and Anthropic back one robots.txt AI training opt-out

Until now, a website that did not want its pages used to train AI models had to play a long game of whack a mole. Every AI company has its own crawler name or its own control token, and a site owner had to find each one, write a separate rule for it in robots.txt and keep that list up to date as new bots appeared. On 30 October 2026, the biggest names in AI agreed to make that a lot simpler.

Google, Microsoft and OpenAI each announced on the same day that they will honor one shared signal in robots.txt that expresses a website's preference about generative AI training. The specification behind it was written by Kevin Kelley of Anthropic, who told the Internet Engineering Task Force (IETF) the same afternoon that Anthropic "has already started implementing support for this draft". Four of the most important AI developers now say they will read the same line in the same file.

The three things you need to know

1. The signal: one shared rule instead of a list of bots. Robots.txt is the small text file at the root of a website that tells crawlers what they may fetch. Normally each group in that file starts with the name of a crawler, such as Googlebot or GPTBot. The new mechanism uses what the draft calls a "policy token" instead of a crawler name. A site writes "User-Agent: __usage_ai-training" followed by "Disallow: /", and that group expresses a preference against AI training for the whole site. Paths still work as usual, so a site can block training in general and allow a press kit folder, for example. According to OpenAI, operators previously had to "identify individual crawlers and maintain separate rules for each and every provider." Google says site owners will no longer be "managing separate instructions for every AI developer."

2. The timing: different start dates, none of them today. Nobody reads the token yet. Microsoft gave the only firm date: it will honor the new preferences from 1 February 2027, as it crawls. Google's blog post says it aims to support the control "in 2027", and a note Google added to its crawler documentation says early 2027. OpenAI says it aims "to implement the signal in the coming months" and will share details when it does. Kelley wrote that Anthropic has started implementing support but did not say when it will be live, and that it will take "some time to add support across our services".

3. The limits: a request, not a lock. The token states a preference about how content is used. It does not stop anyone from crawling. The draft says plainly that it "is not a directive for crawling behavior". OpenAI calls its participation a "voluntary effort", and the draft itself contains no enforcement mechanism. There is also an important detail at Google: a site that opts out with the shared token, and does not mention Google-Extended separately, will be treated as if it had disallowed Google-Extended. According to Google's documentation, that control covers grounding in Gemini Apps too, which means those pages would also stop appearing in Gemini answers that cite the web. Rankings in Google Search and Bing are not affected.

How the rule is written

The draft, titled draft-kelley-aipref-training-preference-signal-00, is short. Its central promise is that it changes nothing about robots.txt itself. It "introduces no new directive names and no new grammar", and it reuses the path matching and longest match rules of RFC 9309, the Robots Exclusion Protocol that the IETF published in 2022. Parsers that do not know the new token simply see a group for a user agent that does not exist and carry on as before.

The double underscore at the start of "__usage_ai-training" has a practical reason. Some deployed parsers match user agent names loosely, and the prefix makes accidental matches unlikely. Consumers must match the token exactly, apart from case.

One rule will surprise many site owners. Wildcard groups, the familiar "User-Agent: *", do not count. A site that blocks every bot with a wildcard rule has, under the draft, said nothing about AI training. If no group in the file names the shared token, "then no preference has been expressed". Kelley described this as deliberate: every robots.txt file published today expresses no preference at all until its owner opts in on purpose.

The reverse also holds. A rule for a company specific token such as Google-Extended keeps its current meaning, but it does not count as a shared opt-out for OpenAI or Microsoft. Existing controls such as GPTBot, Google-Extended and Anthropic's ClaudeBot keep working alongside the new signal. OpenAI, Google and Anthropic all said explicitly that the shared token complements their existing controls rather than replacing them.

What counts as AI training

The draft borrows its definition from a separate IETF document, the AI Preferences vocabulary. AI training means "using an asset to modify the learned parameters of a generative AI model". Models that only classify, rank or score content are excluded, even if they could generate text. Retrieving a page at answer time to ground a chatbot response is also outside the definition, because it does not change a model's parameters.

OpenAI's wording is slightly narrower. Its post talks about "foundational generative AI training", and it does not say whether that word is meant to limit the scope. The company says it will publish details of its implementation later.

The draft also says that "private agreements MAY override preferences expressed via this mechanism". A publisher with a licensing deal can publish a blanket opt-out and still let that deal govern its relationship with a specific AI company. And the preference that applies to a page is the one in the robots.txt file at the time the page was retrieved. The draft does not address content that was collected before a site added the token.

Microsoft goes furthest

Microsoft published the most detailed plan. On top of the shared robots.txt token, it adds a page level training preference that can be set in a meta tag or an HTTP header, and a separate AI Grounding preference that controls whether a page may be used to inform generated answers in Bing, Copilot and Microsoft 365. Grounding has three values: Allow, which is the default, Disallow, and a new option called DisallowExtended. That last one keeps a page available to Microsoft's own AI experiences but excludes it from grounding services that Microsoft sells to third parties.

Microsoft also set a calendar. Temporary grounding transition settings open in Bing Webmaster Tools on 15 December 2026. The new preferences take effect on 1 February 2027. After 1 March 2027, the old NoArchive and NoCache tags will no longer be read as preferences for training or grounding, so sites that relied on them need to switch. Microsoft is also unusually candid about the trade off. Restricting grounding reduces brand visibility in AI experiences, and in its words, "Neither choice guarantees inclusion, traffic, or payment."

Google ties training and Gemini together

Google's approach is a translation rule rather than a new system. A shared token opt-out becomes a Google-Extended disallow, unless the site has written its own Google-Extended rule. Because Google-Extended covers both training and grounding in Gemini Apps and Vertex AI, a blanket opt-out at Google is broader than the same line at Microsoft. Google says that "websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." In practice, a site that wants to refuse training everywhere but stay visible in Gemini answers can add an explicit Google-Extended allow group next to the shared opt-out. Google calls this the ability to "opt out across all providers, or to pick and choose."

Google also linked the move to European law. It says the proposal is consistent with the EU Code of Practice for General Purpose AI model providers under the EU AI Act, which recognizes robots.txt as a primary way for publishers to express machine readable preferences.

Not a standard yet

For all the attention, the signal has no official standing. The IETF datatracker page says the draft "is not endorsed by the IETF and has no formal standing in the IETF standards process." It is an individual submission by Kelley, who is listed as the sole author with an Anthropic affiliation, and it expires on 3 May 2027 unless it is revised. Kelley described it as input to the working group's unfinished work on how preferences attach to content. Microsoft said that although the working group continues toward a final standard, the current work "represents significant progress that merits adoption."

It is also not yet universal. Meta, Amazon and Apple took part in the wider discussion or run their own AI crawlers, but none of them is named in the announcements as a company that will read the token. OpenAI and Anthropic both ended their statements by asking other AI providers to join.

Why it matters

For publishers and bloggers, the immediate win is housekeeping. One line in one file is much easier to maintain than a growing list of vendor specific rules, and it is far less likely to break when a new crawler appears. For the AI companies, a shared signal is a way to show regulators and publishers that they respect opt-outs, at a time when the EU's AI rules and copyright fights put pressure on how training data is collected.

The bigger story, though, is the split between training and grounding. Training shapes what future models know. Grounding decides which pages are cited, with links, in the answers people read today. Microsoft treats them as separate choices. Google bundles them for Gemini. A brand that wants to be cited in AI answers but kept out of model training will need different settings at each company, and it will need to check them carefully before adding a blanket opt-out.

And none of this is enforcement. A voluntary preference only works if crawlers respect it. The companies that signed up today are the ones most likely to do so. The real test is everyone else.

Dany's take

This is a genuinely useful step, and I like that it reuses robots.txt instead of inventing yet another file or protocol. Every site owner already knows where that file lives. One clear line for AI training is the kind of boring plumbing the web needs, and getting Google, Microsoft, OpenAI and Anthropic to agree on the same spelling is no small thing.

But read the fine print before you paste it in. If you care about showing up in Gemini answers, the shared opt-out costs you that visibility at Google unless you add a separate Google-Extended rule. Your existing wildcard block does nothing for training. And until early 2027, nobody actually reads the token. My advice for djcroman readers who run a blog: decide what you want for training and what you want for AI citations as two separate questions, and write your robots.txt for both.

My question for you: will you block AI training on your own site, or do you want your content in the models? Tell me what you think.

Sources: OpenAI: Making it easier for websites to express AI training preferences, Google Search Central Blog: Supporting a shared AI training control for websites, Microsoft Bing Blog: Your content, your choice, IETF: draft-kelley-aipref-training-preference-signal-00, PPC Land.