How to Opt Out of AI Training With One robots.txt Rule (WordPress and Next.js)

For years, keeping your site out of AI training meant listing every company's crawler by name: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, and so on. Miss one and you've said nothing to that company. Last week the big four lined up behind one rule that covers all of them. Here's what it is, how to add it to a WordPress or Next.js site, and why I'm not adding it to mine.

Kevin Kelley, who works on standards at Anthropic, published an Internet-Draft called "Machine-Readable Training Preference Signal" on October 30. It's an individual draft, not something the IETF has adopted. The IETF Datatracker lists it as an individual submission, and Kelley told the AI Preferences working group he's offering it as input to their work. It borrows its definition of AI training from the working group's own vocabulary draft: using content to change the learned parameters of a generative AI model.

The same day, each company said where it stands:

• Anthropic: In that working group post, Kelley wrote that "Anthropic has already started implementing support for this draft" and that it'll "take us some time to add support across our services." No live date yet.
• OpenAI: OpenAI says it "will honor a common signal" and aims "to implement the signal in the coming months." It calls this a voluntary effort.
• Google: Google's Search Central blog says it aims to support it "in 2027." Its crawler docs narrow that to "early 2027."
• Microsoft: The Bing blog says "Starting February 1, 2027, Microsoft will honor the new preferences as it crawls."

All four say their existing per-company controls keep working alongside it.

It's an ordinary robots.txt group. The only new thing is the name on the User-Agent line:

User-Agent: __usage_ai-training
Disallow: /

That says "please don't train AI models on anything on this site." A few rules from the draft matter more than they look:

• User-Agent: * doesn't count. The draft says wildcard groups "MUST NOT be interpreted as matching the policy token." If your robots.txt only has a * group, like most sites, you haven't expressed a training preference at all.
• No group means no preference. Not a yes, not a no. Microsoft treats that as Allow, its default.
• Partial rules cut both ways. Paths you Disallow carry a preference against training. Paths you Allow, or simply leave out of a group that exists, carry a preference *in favor*. So Disallow: /courses/ on its own says "train on everything else."
• It's not a crawl block. The draft says the token "is not a directive for crawling behavior." Your rules for named crawlers like GPTBot or Googlebot still decide what gets fetched.
• It's a request. Nothing technically stops a company that doesn't honor it.

This is the part worth slowing down for if you want ChatGPT, Claude, Copilot or Gemini to mention and link your site.

• OpenAI keeps training and search separate. Its crawler docs say a site can allow OAI-SearchBot "to appear in search results while disallowing GPTBot."
• Anthropic splits ClaudeBot (training) from Claude-SearchBot and Claude-User, which handle search and fetching pages for users, per its crawler help article.
• Microsoft treats training and grounding as "independent preferences." Grounding is what feeds Copilot answers and their citations, and you set it per page, not in robots.txt.
• Google is the catch. Google's blog says that when a site uses the shared token to opt out "and Google-Extended is not explicitly specified," Google will treat it "as if the publisher had disallowed Google-Extended." Google-Extended covers both Gemini training *and* grounding in Gemini Apps, per Google's crawler docs. Google's blog adds that "websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

So your options look like this:

1. No training anywhere, and you're OK losing Gemini Apps grounding: add the shared rule.
2. No training at OpenAI, Anthropic and Microsoft, but stay in Gemini Apps (and accept Gemini training): add the shared rule plus User-Agent: Google-Extended with Allow: /.
3. You want AI tools to find and use your content: do nothing, or say so explicitly with Allow: /.

WordPress builds a "virtual" robots.txt on the fly, and you can add to it with the robots_txt filter. One catch first: if there's a real robots.txt file in your site's root folder, your web server serves that file and WordPress never runs the filter. Check with your host or an FTP client.

Option 1: a small must-use plugin. Save this as wp-content/mu-plugins/ai-training-preference.php (create the mu-plugins folder if it isn't there). Must-use plugins load automatically and can't be switched off by accident from the Plugins screen.

<?php
/**
 * Plugin Name: AI training preference
 * Description: Adds the shared __usage_ai-training group to WordPress's virtual robots.txt.
 */
add_filter( 'robots_txt', function ( $output, $public ) {
	// If "Discourage search engines" is on, WordPress already blocks everything. Leave it alone.
	if ( ! $public ) {
		return $output;
	}
	$output .= "\nUser-agent: __usage_ai-training\nDisallow: /\n";
	return $output;
}, 10, 2 );

For option 2 from the list above, add User-agent: Google-Extended and Allow: / the same way.

Option 2: Yoast SEO. Go to Yoast SEO, then Tools, then File editor, and add the group. Yoast creates a real robots.txt file, so after that, edit the file, not the filter. If the File editor menu is missing, file editing is turned off on your install.

Option 3: Rank Math. Turn on Advanced Mode, then go to Rank Math SEO, General Settings, Edit robots.txt. Rank Math edits the virtual file, so you'll need to delete any real robots.txt file first.

Whichever you pick, open yoursite.com/robots.txt afterward and make sure the new group is there.

With the App Router, app/robots.ts generates robots.txt from a MetadataRoute.Robots object. Add the token as its own rule:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      // Your normal crawl rules stay as they are.
      { userAgent: '*', allow: '/', disallow: ['/admin/', '/preview/'] },
      // The shared AI training preference. This group doesn't block crawling.
      { userAgent: '__usage_ai-training', disallow: '/' },
      // Optional: keep Gemini Apps grounding (this also allows Gemini training).
      { userAgent: 'Google-Extended', allow: '/' },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}

That builds this robots.txt:

User-Agent: *
Allow: /
Disallow: /admin/
Disallow: /preview/

User-Agent: __usage_ai-training
Disallow: /

User-Agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml

Drop the Google-Extended rule if you want option 1. If you use a plain app/robots.txt or public/robots.txt file instead, paste the same groups in.

Keep your GPTBot, ClaudeBot and Google-Extended rules for now. Nobody reads the shared rule yet, and all four companies say their own controls still apply. Microsoft also says its old NoArchive and NoCache tags stop counting as training or grounding preferences after March 1, 2027, so if you rely on those, switch before then.

One robots.txt gotcha while you're in there: under RFC 9309 and Google's robots.txt rules, a crawler that finds a group with its own name follows that group and ignores the * group. If you add User-Agent: GPTBot with Disallow: /courses/, GPTBot no longer sees your * rules for /admin/. Repeat anything you still need. The training token group doesn't have this problem, because it isn't a crawler.

Nothing, for now. My robots.txt only has a * group that keeps crawlers out of my admin area and draft previews, so under the draft it expresses no training preference. That's on purpose. I write these posts to be found, read and quoted, and I'd rather ChatGPT, Claude, Copilot and Gemini know who I am and what I do than not. Opting out would cost me Gemini Apps grounding for almost no upside.

If I sold courses, paid research or anything behind a login, I'd make a different call and opt those paths out. I'll revisit when Microsoft starts reading the rule in February.

If you're not sure which option fits your business, or you want it set up without breaking your SEO, let's talk.

Does User-Agent: * with Disallow: / opt me out of AI training?
Not under this draft. Wildcard groups never match the training token. It does block crawlers that follow robots.txt from fetching your site at all, which is a much bigger hammer.

Is __usage_ai-training an official standard?
No. It's an individual Internet-Draft dated October 30, 2026, offered to the IETF AI Preferences working group. The IETF hasn't adopted it.

When will AI companies start honoring it?
Microsoft from February 1, 2027. Google says early 2027. OpenAI says the coming months. Anthropic says it has started implementing support but hasn't announced a date.

Will opting out of AI training hurt my Google Search rankings?
Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." It does control grounding in Gemini Apps.

Does it remove my content from models that already trained on it?
The draft ties the preference to the robots.txt in effect when content is fetched. It doesn't say anything about content collected earlier.

• IETF, draft-kelley-aipref-training-preference-signal-00, "Machine-Readable Training Preference Signal" (October 30, 2026)
• IETF Datatracker, draft-kelley-aipref-training-preference-signal
• Kevin Kelley, "New draft: draft-kelley-aipref-training-preference-signal-00," ai-control mailing list (October 30, 2026)
• IETF, draft-ietf-aipref-vocab-08, "A Vocabulary For Expressing AI Usage Preferences"
• OpenAI, "Making it easier for websites to express AI training preferences" (October 30, 2026)
• OpenAI, Overview of OpenAI Crawlers
• Google Search Central Blog, "Supporting a shared AI training control for websites" (October 30, 2026)
• Google, Google's common crawlers (Google-Extended)
• Google, How Google interprets the robots.txt specification
• Microsoft Bing Blog, "Your content, your choice: clearer preferences for training and grounding" (October 30, 2026)
• Anthropic Privacy Center, Does Anthropic crawl data from the web, and how can site owners block the crawler?
• RFC 9309, Robots Exclusion Protocol
• WordPress Developer Resources, robots_txt hook
• Yoast, How to edit robots.txt through Yoast SEO
• Rank Math, How to edit robots.txt with Rank Math
• Next.js Docs, robots.txt