OpenAI said it will honor a common signal in robots.txt that website operators can use to state preferences about use of their content for training generative models. The signal comes out of the IETF's AI Preferences working group, which has drafts for a training-preference signal and an accompanying vocabulary; OpenAI said it is joining other working-group participants in the announcement. The point is consolidation: instead of identifying each provider's crawler and keeping a separate rule for each, an operator writes one preference that participating providers read. OpenAI said it aims to implement the signal in the coming months and will publish implementation details then, that the new signal does not replace the crawler controls it already documents, and that it encourages other providers to honor it.
Our takeA preference signal is a vocabulary, not an enforcement mechanism: its effect depends on how many crawlers read the token and what they do when it is absent or ambiguous. The draft's harder questions are granularity - one global token versus per-purpose preferences for search, training and fine-tuning - and what a change of mind means for data already fetched. Until the implementation exists, the checkable part is whether the token appears in a crawler's fetch log; a robots.txt line no parser reads is documentation, not control.