A promise to read, not a rule that binds
On 30 October OpenAI said it will honor a shared signal in robots.txt that website operators can use to state preferences about whether their content is used to train generative models. The signal comes out of the IETF's AI Preferences working group. OpenAI said it aims to implement it in the coming months, that the signal does not replace its existing crawler-specific controls, and that it encourages other providers to honor it. The specification sits in two Internet-Drafts: draft-ietf-aipref-attach-05, dated 19 August 2026, which defines how a preference is attached to content over HTTP and in robots.txt, and draft-ietf-aipref-vocab-08, dated 14 September 2026, which defines what the preference means. Both are Standards Track, and the attachment draft updates RFC 9309 if approved. Neither guarantees that a stated preference is followed; both say so explicitly.
The carrier: one line, an optional path, a token
Two carriers are defined. The first is a response header, Content-Usage: train-ai=n, a Structured Field dictionary per RFC 9651 attached to the representation rather than the resource: it describes the bytes served, not the URL. The second is a new rule inside a robots.txt group, whose grammar reuses the existing rule pattern and adds a directive with an optional path:
content-usage = *WS "content-usage" *WS ":" *WS [ path-pattern 1*WS ] usage-pref EOL
Because the path is optional, one file can carry a site-wide default and per-prefix exceptions. Path matching copies RFC 9309: the longest matching prefix wins, length is counted in encoded bytes, order carries no meaning, and Content-Usage rules can be interleaved with Allow and Disallow. Groups still key on product tokens, so a named group addresses one crawler and User-Agent: * covers the rest. The draft's example resolves to three rows: crawlable paths inherit train-ai=n, /ai-ok/ overrides it to y, and /never/ carries no preference at all.
Three values, and a default the spec refuses to set
The vocabulary assigns every category of use one of three values: allowed, disallowed, or unknown. Absence of a statement means unknown for every category. This revision defines AI Training (train-ai), using an asset to modify the learned parameters of a generative model; AI Use (ai-use), using an asset as input to a generative model when the asset is not directly supplied by the user; and Search (search). Explicit preferences serialize as the tokens y and n, case-sensitive and processed as bytes.
The interesting choice is everything else. A missing key, a value that is not a token, a duplicate key whose last value changed type, or a parse failure all resolve to unknown rather than to an error. A line such as train-ai=y, train-ai, search=n, search="n" yields unknown across the board, because the last value for each key is not a token. And the specification takes no position on what a recipient should do with unknown, which moves the effective default into each implementer's policy. The consequence is that a malformed preference fails silently: the operator believes a restriction is published, the parser reads nothing, and no error surfaces on either side. The only defence is to resolve your own file and compare the tri-state with what you meant.
Acquisition and usage are separate stages
Content-Usage rules apply only to resources that may be crawled. A disallowed path carries no usage preference at all. The split is coherent — fetching and using are different acts — but it inverts the older intuition that blocking a crawler refuses it. Under this model a blanket Disallow: / states something about fetching and nothing about training. Sites with the strongest views about their content would publish the least about its use, unless they keep paths crawlable and state the preference separately.
The mirror of that gap sits on the receiving side. A preference travels only with a fetch from the origin. Content acquired another way, through licensed feeds, purchased corpora or third-party mirrors, arrives without one, and the vocabulary defines no mechanism for reconstructing it. Embedded metadata and a preferences registry are named as alternatives and both are deferred: embedding needs per-format work that plain text lacks, and a registry needs an identity model the drafts do not define. A robots.txt preference also implies only that the server is the source, unauthenticated, so nothing establishes who is speaking.
Two conflict rules in the same file
RFC 9309 resolves an equivalent Allow and Disallow in favour of Allow and treats a URI with no matching rule as allowed. Content-Usage does the opposite: when rules with identical paths carry conflicting preferences, the vocabulary's combination rule applies and any disallow wins. The same document therefore has a permissive default for acquisition and a restrictive one for usage. The asymmetry is defensible, since over-fetching costs less than over-training, but a single code path for both rule types will get it wrong. The reference is also a sign of drift: the attachment draft points at Section 7.1 of the vocabulary, while the current revision puts the combination rule in Section 5.1.
Time, and what has already left the server
A crawler may cache robots.txt and should not use a cached copy for more than 24 hours. Preferences are evaluated against the copy current at fetch time, so a change applies only after the crawler refetches the file and then the resource; there is no retroactive effect. And once content has modified weights, no later line in a text file reaches it. The vocabulary acknowledges this with a section proposing that preferences travel downstream with a distributed model, by listing the preferences tied to training inputs or stating which uses they permit. That section carries an open issue number, so it is a proposal rather than a requirement. Until something travels with the weights, the signal is a statement about future fetches.
What can actually be checked
Three things are testable from outside: the tri-state a reference parser resolves for a given URL and category, the revision of robots.txt current at fetch time, and, on the crawler side, whether the token was read. Only the third is evidence of adoption, and it lives in logs the site does not control. A server log shows that a request arrived; it does not show that a preference was honored. Operators can publish and verify locally, but crawler behaviour remains an observation, not an assumption.
What the drafts settle
They settle the format, the addressing, the resolution procedure and the conflict rule. They do not settle the obligation, the default for unknown, the identity of the declaring party, or how preferences attached to training data follow the model. The vocabulary's sections on the data model and the category list are marked as not yet having consensus in this revision. Today's announcement is therefore a commitment to read a vocabulary whose semantics are defined and whose enforcement is not. The measurable claim is narrower: that a stated preference resolves to the intended value under the reference algorithm, and that a participating crawler records having read it. Anything beyond that is an agreement between parties, and an agreement is not a property of the protocol.
Sources
OpenAI, "Helping websites express AI training preferences" — https://openai.com/index/helping-websites-express-ai-training-preferences
G. Illyes and M. Thomson, "Associating AI Usage Preferences with Content in HTTP", draft-ietf-aipref-attach-05, 19 August 2026 — https://www.ietf.org/archive/id/draft-ietf-aipref-attach-05.txt
P. Keller and M. Thomson (ed.), "A Vocabulary For Expressing AI Usage Preferences", draft-ietf-aipref-vocab-08, 14 September 2026 — https://www.ietf.org/archive/id/draft-ietf-aipref-vocab-08.txt
M. Koster, G. Illyes, H. Zeller and L. Sassman, "Robots Exclusion Protocol", RFC 9309, September 2022 — https://www.rfc-editor.org/rfc/rfc9309.txt
M. Nottingham and P. Kamp, "Structured Field Values for HTTP", RFC 9651, September 2024 — https://www.rfc-editor.org/rfc/rfc9651