Robots.txt AI training opt-out gets one shared token

OpenAI and Google both said on Oct. 30, 2026 that they will honor one shared robots.txt token, __usage_ai-training, for a website's AI training preference. A site that writes User-Agent: __usage_ai-training and Disallow: / will, once providers implement it, tell every participating provider "no generative AI training" in one place instead of listing each company's crawler. Neither company reads the token yet: OpenAI says "in the coming months," and Google's crawler documentation says early 2027.

The catch is in how Google will read it. Google plans to treat a shared opt-out as if the site had disallowed Google-Extended, unless the file names Google-Extended directly. That token also controls grounding in Gemini Apps, so a blanket opt-out can cost more than training use. If you want to be cited in Gemini Apps answers, decide on Google-Extended explicitly before you add the shared line.

What is the __usage_ai-training token in robots.txt?

It is a "policy token": a name used in a robots.txt User-Agent line that states how content may be used, not which bot may crawl. The format comes from an individual Internet-Draft, Machine-Readable Training Preference Signal, written by Kevin Kelley of Anthropic and posted on Oct. 30. It builds on the IETF AI Preferences (AIPREF) working group's vocabulary.

The draft's own example looks like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That group expresses a preference against AI training for the whole site except /press-kit/, which it marks as open to training. "AI training" has a narrow meaning here: using content "to modify the learned parameters of a generative AI model." Models used only for classification, ranking or scoring are outside it.

The draft adds no new directives or grammar to RFC 9309, the robots.txt standard. Path matching and longest-match precedence work as they do today, and the draft says existing parsers will ignore the group like any user agent they don't recognize.

One point on status. The IETF datatracker labels the document an individual draft that "is not endorsed by the IETF and has no formal standing in the IETF standards process." The token name or details may still change before anything becomes a standard.

OpenAI and Google gave different timelines

Both committed to reading the shared signal, on different timelines, and both kept their own controls.

OpenAI said it "will honor a common signal" for "foundational generative AI training," aims to implement it "in the coming months," and will publish implementation details then. Its existing crawler controls "will continue to apply and be respected."

Google said it aims "to support this new control along with other AI providers in 2027." Its crawler documentation now notes the plan for "early 2027." Google added that the approach is consistent with the EU Code of Practice for general-purpose AI model providers, which treats robots.txt as a primary way for publishers to state machine-readable preferences.

How does the shared token interact with GPTBot and Google-Extended?

The shared token adds to company-specific rules; it does not replace them. Here is what each line does, based on the vendors' current documentation.

Three rules in the draft decide most real files:

• Wildcards don't count. A User-Agent: * group "MUST NOT be interpreted as matching the policy token." A site that blocks every bot with * has still not stated a training preference under this scheme.
• No group means no preference. If the file never names __usage_ai-training, the site has expressed nothing either way.
• Specific rules can win. The draft says specific expressions should take precedence and lets providers treat their own tokens as more specific. Google's version: the shared opt-out maps to Google-Extended only when Google-Extended isn't explicitly specified.

The draft also keeps the two jobs separate. The token "is not a directive for crawling behavior." To stop a bot from fetching pages, you still write a group for that bot.

What it costs to opt out of Gemini grounding

Google-Extended covers two things at once: training future Gemini models and grounding answers in Gemini Apps. Google's announcement says sites that specifically allow Google-Extended "are eligible to appear in responses and receive clicks via grounding in Gemini Apps." Google also states that Google-Extended does not affect inclusion or ranking in Google Search.

BaristaLabs interpretation: once Google implements the shared token, a site-wide Disallow under __usage_ai-training with no Google-Extended group would also remove the site from Gemini Apps grounding. You can't keep grounding and refuse Gemini training through these tokens, because one token covers both. That makes this a business decision for whoever owns site traffic, not a one-line technical fix.

OpenAI splits the same jobs differently. Training sits on GPTBot and ChatGPT search on OAI-SearchBot, and OpenAI says each setting is independent. A training opt-out there leaves ChatGPT search visibility alone, provided OAI-SearchBot stays allowed. An earlier post covers how to measure OpenAI search and bot traffic before you change those rules.

Should you add the token to robots.txt now?

Add it once your training policy is settled, and keep your existing per-company lines. The group costs nothing today, since parsers that don't know the token skip it. The draft also ties preferences to "the robots.txt file in effect when the asset was retrieved," so a file that states your policy early is in place when providers start reading it.

Don't remove GPTBot or Google-Extended rules in exchange. Neither provider reads the shared token yet, and both say their own controls continue.

A site that wants to refuse training but stay visible in ChatGPT search would keep its current groups and add the shared one:

User-Agent: __usage_ai-training
Disallow: /

User-Agent: GPTBot
Disallow: /

User-Agent: OAI-SearchBot
Allow: /

Whether to add User-Agent: Google-Extended with Allow: / depends on the trade-off above: allowing it keeps Gemini Apps grounding and also allows Gemini training.

Who needs to decide this

Anyone who publishes content they would treat differently for training and for discovery: publishers, documentation teams, ecommerce catalogs and businesses whose public pages are their product. The machine visitor terms sheet from July is a good way to set a policy per content area before you write the file. If AI assistants are part of how customers find you, the AI-readable website guide covers what those assistants should be able to read.

Recommended first step

Pull your live robots.txt and answer three questions in writing:

1. Do you have a GPTBot group, and does it match your training policy?
2. Do you have a Google-Extended group, and do you want Gemini Apps grounding?
3. Is any training preference expressed only through User-agent: *? Under the draft, that one does not count.

Then add the __usage_ai-training group with the paths you want to protect, and check again when OpenAI publishes its implementation details. If you want help aligning robots.txt, your site framework's generated rules and your AI search goals, see AI-assisted website development.

Sources and scope

• OpenAI, Making it easier for websites to express AI training preferences, Oct. 30, 2026.
• Google Search Central Blog, Supporting a shared AI training control for websites, Oct. 30, 2026.
• Google, Google's common crawlers (Google-Extended), updated Oct. 30, 2026.
• IETF, draft-kelley-aipref-training-preference-signal-00, Oct. 30, 2026.
• OpenAI, Overview of OpenAI crawlers, read Oct. 30, 2026.
• IETF, RFC 9309: Robots Exclusion Protocol, September 2022.

These sources describe announced plans. They don't say which other AI providers will honor the token, how OpenAI will resolve conflicts between the token and GPTBot, or whether the token will keep its current name once the IETF process continues.