How to Block AI Training Crawlers in robots.txt: The New __usage_ai-training Directive
Transfon Team
7 min read
On 30 October 2026, Anthropic published an IETF draft proposing a new robots.txt extension: a policy token called __usage_ai-training. Two lines in your robots.txt, and every AI company that honours the draft stops using your content to train generative models. Anthropic honours it now. Microsoft follows on 1 February 2027, Google in early 2027, and OpenAI within months. Meta has expressed support without committing to a date. The directive cannot undo data already collected, does not block AI search or grounding, and has no effect on crawlers that choose to ignore robots.txt.
Why the existing approach is broken
Today, blocking AI training crawlers means maintaining a separate robots.txt entry for every company: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, meta-externalagent for Meta, Bytespider for ByteDance, and so on. New crawlers appear without notice, and the list never stabilises.
The __usage_ai-training token is a cross-vendor standard. One entry covers any AI company that adopts the spec, including future entrants, without requiring publishers to know each crawler's name.
How the directive works
__usage_ai-training is a "policy token" in standard robots.txt syntax. It tells a crawler how downloads may be used, separately from whether the crawler is allowed to visit at all.
A crawler blocked by your existing User-Agent rules still cannot visit. A crawler permitted by your existing rules but covered by __usage_ai-training: Disallow: / may still fetch pages for search indexing or for answering user questions, but it may not use those pages to update the learned weights of a generative AI model.
The IETF draft defines AI training as "using an asset to modify the learned parameters of a generative AI model." Models used exclusively for non-generative tasks are outside the scope.
The robots.txt syntax
No new grammar is needed. The draft uses standard RFC 9309 robots.txt rules: a User-Agent: line followed by Disallow: and Allow: paths.
Block AI training on all pages:
User-Agent: __usage_ai-training
Disallow: /
Block AI training everywhere except a press kit:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
Training rules and crawler rules are independent. Blocking training on / does not stop any crawler from visiting. A full robots.txt combining both looks like this:
# Search engines
User-Agent: *
Disallow:
# Per-company training blocks (keep until each adopts the standard)
User-Agent: GPTBot
Disallow: /
User-Agent: ClaudeBot
Disallow: /
User-Agent: Google-Extended
Disallow: /
# Cross-vendor AI training preference
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
Download the full annotated template with entries for all major AI crawlers and instructions on which to remove as each company adopts the standard.
What the directive does not cover
Already-collected data. If a crawler fetched your pages before you added this entry, those copies may already be in a training dataset. The draft provides no mechanism to demand deletion. The signal only affects crawls that happen after you publish it.
AI search and grounding. When an AI assistant retrieves your page to answer a user question, that is grounding, not training. Microsoft has announced a separate grounding preference that would cover that use case; the current IETF draft covers training only.
Non-generative AI. The spec explicitly excludes models performing exclusively non-generative tasks. A company using your content to train a spam classifier is outside the scope.
Crawlers that ignore it. Compliance is voluntary. Each company decides independently whether to honour the standard and when. Until a crawler's operator confirms support, per-company entries remain the fallback.
Who honours it and when
| Company | Support | Date |
|---|---|---|
| Anthropic | Confirmed | 30 October 2026 |
| Microsoft | Confirmed | 1 February 2027 |
| Confirmed | Early 2027 | |
| OpenAI | Committed | Coming months, no fixed date |
| Meta | Expressed support | No date committed |
Microsoft and OpenAI have also said they will extend the control to third-party tools built on their models.
Microsoft separately plans a grounding preference that would prevent AI tools from fetching pages to answer user questions, a use case the current draft does not cover. OpenAI has indicated similar controls are planned.
Why you still need bot management for AI crawlers
robots.txt is a preference signal. A crawler that honours it stops training on your content. A crawler that ignores it keeps going regardless.
Agentic browsers are a separate problem. AI agents that browse on behalf of users, such as ChatGPT's browsing mode and Meta's Muse, mimic real browser sessions rather than announcing themselves as crawlers. robots.txt was not written for them, and __usage_ai-training does not cover them.
Transfon Security handles each case:
- Declared crawlers are matched by name against your robots.txt. Any that attempt to crawl beyond your rules are blocked.
- Crawlers that ignore robots.txt are identified by network origin (data centre vs. residential), user-agent consistency and request patterns.
- Agentic browsers are identified by data centre origin, navigation speed and direct routes to forms, login and checkout pages.
- Per-path controls let you allow AI search crawlers on content pages while keeping them away from account and payment pages, independently of the training preference.
Start blocking AI crawlers today
Add __usage_ai-training to your robots.txt now. Anthropic honours it immediately, and the others follow through early 2027. Keep your per-company entries as a fallback until each confirms support.
For crawlers that ignore robots.txt and agentic browsers that robots.txt does not reach, Transfon Security blocks at the network edge.
- Sign up for Transfon Security to see which crawlers are hitting your site, which ones are bypassing your robots.txt, and how much of your traffic comes from data centres instead of real visitors.
- Download the robots.txt template with all current AI crawler entries and annotations on when to remove each one.
- Talk to our team if you want help auditing your crawler traffic before setting rules.
FAQ
Does __usage_ai-training block AI search results?
No. It blocks training only, specifically updating the parameters of a generative AI model. If an AI assistant retrieves your page to answer a user question, that is grounding or search retrieval, not training. Microsoft has announced a separate grounding preference for that use case.
Does adding this entry affect my search engine ranking?
No. Standard search engine crawlers are controlled by User-Agent: * and company-specific rules. The __usage_ai-training token has no effect on Googlebot, Bingbot or other search engine crawlers.
Do I still need separate entries for GPTBot, ClaudeBot and Google-Extended?
Yes, for now. Anthropic already honours the draft; Microsoft follows on 1 February 2027; Google in early 2027; OpenAI within months. Keep per-company entries until each confirms it reads __usage_ai-training.
What happens to data already collected from my site?
The directive has no retroactive effect. Content crawled before you add this entry may already be in a training dataset. The signal only affects future crawls.
Can I allow training on specific pages?
Yes. Add an Allow: line for any path you want to permit. The longest matching path takes precedence, following standard RFC 9309 rules.
Where can I read the full technical spec?
The IETF draft is draft-kelley-aipref-training-preference-signal-00, published 30 October 2026 and authored by K. Kelley of Anthropic.
Transformation Strategies for Publishers and Marketers
Data Privacy and Security, Performance and User Experience, Regulation Compliance, User and Revenue Growth
Stay ahead of the publishing curve
Privacy, security, monetization, and performance insights. A few emails a month, no spam.