OpenAI’s Shared Training Signal Turns robots.txt Into a Purpose Token From a Same-Day Individual Draft, and a Group That Blocks One Folder Reads as a Preference for Training on Every Other Path

On October 30, OpenAI said it will honor “a common signal” in robots.txt that lets a site operator give the same AI training preference to several providers at once. The announcement is five short paragraphs and never prints the signal. The word “signal” is a hyperlink, and it leads to an Internet-Draft that the IETF datatracker files as an individual submission, last updated the same day, whose author lists Anthropic as an affiliation.

What that draft defines is a robots.txt group whose User-Agent line is __usage_ai-training. We read it against the working group’s own robots.txt draft, ran its lookup rules through three robots.txt parsers, and read 61 live robots.txt files to see what site owners do today. The findings that matter to a site owner:

• The “common signal” is one of two different ways that IETF drafts put a training preference in robots.txt. The working group’s own draft adds a Content-Usage rule inside a crawler’s group. The draft OpenAI links does the opposite and keys a separate group on a token.
• Under the draft’s lookup rules, a token group that blocks one folder expresses a preference in favor of training for every other path, a blanket User-agent: * block expresses no preference at all, and the token does not stop a crawl.
• Python’s urllib.robotparser, Protego and robotspy all agreed with the draft on every allow or block decision we tried, but none of them can answer “no preference”.
• None of the 61 files carries the token or a Content-Usage rule yet. Of the 61, 46 name at least one of six training crawlers, 15 name all six, and 5 treat those crawlers differently at the site root, which a purpose-level signal cannot express by itself.

What the OpenAI post says, and what it leaves out

The commitments are short. OpenAI says it “will honor a common signal” in robots.txt. It calls the effort voluntary: “This voluntary effort makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once.” It describes today’s burden this way: “Previously, website operators would identify individual crawlers and maintain separate rules for each and every provider.” It gives no date: “We aim to implement the signal in the coming months, and will share another update with details about our implementation when we do so.” And it says the new signal “will not replace the controls we already provide through robots.txt.”

That makes the post a commitment, not a change in behavior. The OpenAI crawler documentation we fetched on October 30 does not mention the signal, and its existing guidance is unchanged: GPTBot “is used to crawl content that may be used in training our generative AI foundation models,” and “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” We did not test any crawler, because the signal is not implemented yet.

Four things are missing from the announcement:

• The token and the draft’s status. The post links the draft and does not say whether its syntax is settled.
• The other participants. The post says OpenAI is “joining other participants in the working group in today’s announcement of this shared signal.” We looked for those announcements on Anthropic’s news index, Cloudflare’s blog index and OpenAI’s own crawler documentation and found none at the time of writing, so we cannot say which providers will honor the signal.
• How each state will be treated. A file with no token group, a file whose group allows a path and a file whose group blocks it are three different inputs (see below). The post does not say what OpenAI will do with each.
• A single definition of training. Four documents word the scope four ways.

None of these contradicts another, but they are not interchangeable. “Foundational” appears only in OpenAI’s phrasing, the draft’s definition reaches any generative model whose parameters are updated, and Cloudflare’s covers fine-tuning of any AI model. Which reading OpenAI applies is a question for the implementation update it promised.

The signal is a robots.txt group named __usage_ai-training

What the draft defines

The abstract says an operator binds a preference to paths “by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line,” and that the draft “introduces no new directive names and no new grammar.” Figure 1 in the draft is the whole mechanism:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The draft reads that as a preference against AI training for every path except those under /press-kit/, where it is a preference in favor. The rules that matter:

• A token that never has to appear in a request. It is a product token in RFC 9309’s sense, matched the same way: “Consumers MUST match policy tokens exactly, apart from case, as for any product token.”
• Wildcard groups do not count. “wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token.”
• The presence of the group decides whether there is a preference. If no group names the token, “no preference has been expressed.” If one does, the lookup follows RFC 9309: a disallowed path is a preference against, and for an allowed path “a preference in favor of that use has been expressed.”
• Crawl rules stay separate. The draft calls usage preferences and traffic management signals “independent and orthogonal,” says “This policy token is not a directive for crawling behavior,” and adds that a consumer “MAY use preferences to inform crawling.”
• Conflicts are left to the consumer. “Specific expressions SHOULD take precedence over general ones,” a consumer “MAY document that they interpret rules that match proprietary policy tokens as being more specific than the __usage_ai-training policy token,” and “Private agreements MAY override preferences expressed via this mechanism.”
• One token, one purpose. The draft defines a single token, for training. The vocabulary draft also has “AI Use” and “Search” categories, and neither has a token here.

Where the draft comes from

The datatracker lists the document as an “Active Internet-Draft (individual)” with one author, Kevin Kelley, whose affiliation in the draft’s header is Anthropic. It is revision 00, and the datatracker’s API stamps it at 15:12 UTC on October 30. The header says Standards Track, but the datatracker shows no stream and no intended RFC status, and states: “This I-D is not endorsed by the IETF and has no formal standing in the IETF standards process.” The acknowledgments thank dozens of working-group participants with affiliations that include Google, Microsoft, Meta, OpenAI, Cloudflare, Amazon Web Services and Common Crawl, and the draft is careful about what that list means: “Affiliations are listed for identification only, since IETF participants contribute as individuals.”

OpenAI’s post links the working group’s vocabulary draft as the “ongoing work” it draws on. That draft and the working group’s robots.txt attachment draft (revisions 08 and 05) both carry the datatracker state “WG Document.” The token draft does not; its group is “Individual Submissions.”

How it differs from the working group’s Content-Usage rule

The working group’s attachment draft defines a different robots.txt mechanism for the same vocabulary: a Content-Usage rule that sits inside a crawler’s group. Its own example is a User-Agent: * group containing Content-Usage: train-ai=n. The two drafts disagree on four points.

The last row has a practical consequence. Under the Content-Usage rule, a path that cannot be crawled carries no usage preference. Under the token draft, a site could block every crawler and still publish a token group that welcomes training, because the two statements are independent.

What the lookup rules mean for a site owner

A group that blocks one folder reads as a preference for training everywhere else

RFC 9309 says “If no match is found amongst the rules in a group for a matching user-agent or there are no rules in the group, the URI is allowed.” The token draft then says an allowed path is “a preference in favor of that use.” Put the two together and a group meant to say “do not train on /private/” says something larger:

User-agent: __usage_ai-training
Disallow: /private/

On the draft’s rules, that file expresses a preference against training for /private/ and a preference in favor of training for every other path. An empty group, one User-Agent line and nothing else, is a preference in favor of everything. The partial-group and empty-group rows of the parser table in the next section show both results. On the draft’s rules there is no way to write “against here, silent elsewhere,” because any group that names the token makes every unmatched path an allowed one. The Content-Usage rule is built differently: each rule carries its own path and preference. If you mean “no training except the press kit,” write the shape of Figure 1: block / and list the exceptions.

A blanket crawl block is silence

The reverse trap: a file whose only group is User-agent: * with Disallow: / blocks every crawler that has no group of its own, and by the draft expresses no training preference at all, because wildcard groups must not be read as matching the token. That is the right call, since a crawl block is not a licensing statement. It does mean that a site which blocks everything with a wildcard has said nothing about training until it adds the token group.

The signal does not stop a crawl, and mixing tokens couples the two

The draft says the token “is not a directive for crawling behavior.” GPTBot’s documented meaning is different: for OpenAI, “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models,” and under RFC 9309 a crawler that obeys the rule also does not fetch the disallowed paths. A site that wants both effects has to write both. The draft also allows a group to list a token beside ordinary crawler names: “Policy tokens and other product tokens MAY be mixed in a group.” That ties the two meanings together. Here are three small files, with GPTBot as the crawler and /page as the path:

  token group only    crawl allow  preference against
  GPTBot group only   crawl block  preference none
  one mixed group     crawl block  preference against

In the first file GPTBot may still crawl while the site states a preference against training. In the second GPTBot is blocked and the site has stated no preference. In the third, one group does both.

Timing

The draft ties a preference to the file in effect at retrieval: “The preferences that apply to an asset are those expressed in the robots.txt file in effect when the asset was retrieved.” The working group’s attachment draft says the same in its own words, that updates “do not retroactively apply to resources,” and lets crawlers cache the file for up to 24 hours. Adding a token group on October 30 therefore changes nothing for content OpenAI has already collected, and nothing at all until OpenAI ships the implementation it promised.

Three parsers, one lookup

A consumer that wants to honor the token has to answer the section 4.2 question. The shortcut is to call an existing parser with the token as the user agent and read allow as “in favor” and block as “against.” We tried that with Python 3.13.14’s urllib.robotparser, Protego 0.7.0 and robotspy 0.13.0, and compared each answer with the draft’s lookup, implemented in about 70 lines (listed under Reproduce it). The “draft says” column is that lookup. A rerun of the script printed identical output.

python 3.13.14 | protego 0.7.0 | robotspy 0.13.0
scenario                     path          draft says stdlib protego robotspy
fig1: draft Figure 1         /press-kit/x  in favor   allow  allow   allow
fig1: draft Figure 1         /blog/        against    block  block   block
star: only a crawl rule      /private/x    none       block  block   block
star: only a crawl rule      /open/        none       allow  allow   allow
partial: one folder blocked  /private/x    against    block  block   block
partial: one folder blocked  /open/        in favor   allow  allow   allow
empty: group with no rules   /open/        in favor   allow  allow   allow
two groups name the token    /a/x          against    block  block   block
two groups name the token    /b/x          against    block  block   block
wildcard path in the group   /report.pdf   against    block  block   block
wildcard path in the group   /report.html  in favor   allow  allow   allow
blank line after User-agent  /open/        against    block  block   block
mixed case token             /open/        against    block  block   block

rows where allow/block misreads the draft, out of 13:
  stdlib: 2
  protego: 2
  robotspy: 2

Every parser agreed with the draft on every allow or block decision, including Figure 1, two groups that name the token (RFC 9309 says to merge them), a wildcard path pattern, a blank line after the User-agent line and a mixed-case token. The only mismatch is the state the parsers cannot express. For a file with only User-agent: *, each parser falls back to the wildcard group, exactly as RFC 9309 tells a crawler to, so a consumer built this way reads a crawl rule as a training preference. That is 2 of 13 rows for each parser. The draft’s wildcard exception covers this case, and a consumer has to check that some group names the token before it asks a parser anything.

The draft’s stated reason for avoiding a new directive is that existing parsers “will continue to interpret traffic management signals as they did previously.” We tested the version of that concern that is easiest to test. We put Content-Usage lines first, in the middle and last in a group and compared the crawl decisions for the other rules.

paths: /private/x /private/public/y /tmp/z /open/  (a = allow, b = block)
  stdlib   baba  same in all four variants: True
  protego  baba  same in all four variants: True
  robotspy baba  same in all four variants: True

All three parsers returned the same decisions in all four variants. That does not show the concern is unfounded, since far more than three parsers are deployed. It shows only that these three are not the ones it is about.

What 61 live robots.txt files show

We fixed a list of 64 well-known news, reference and technology sites before reading any file (the list is under Reproduce it) and fetched each site’s robots.txt on October 30 at 19:47 UTC with a desktop browser user agent. Sixty-one returned a readable file. politico.com, stackoverflow.com and sciencedirect.com did not (HTTP 403, 418 and 403). A site counts as naming a token when a User-agent line lists it, whatever the rules say, and “blocked” means the file’s own rules disallow / for that token. This is a fixed list of prominent sites, not a random sample, and a server can send a different file to a different client. The six tokens below all appear in the example managed robots.txt in Cloudflare’s documentation.

• The lists are long and uneven. 46 of the 61 files name at least one of the six tokens. 15 name all six and 15 name none. The rest fall between: 2 name one, 8 name two, 2 name three, 6 name four and 13 name five. The nytimes.com file names all 24 crawler tokens on our list.
• The two best-known tokens often diverge. 30 files name both GPTBot and ClaudeBot. Eight name ClaudeBot but not GPTBot (washingtonpost.com, theguardian.com, ft.com, businessinsider.com, wired.com, arstechnica.com, newyorker.com, bleepingcomputer.com) and three name GPTBot but not ClaudeBot (wsj.com, goodreads.com, and cloudflare.com, which names anthropic-ai and Claude-Web instead).
• Five files treat the six tokens differently at /. theatlantic.com, theverge.com and vox.com allow GPTBot and block the other five. quora.com allows Google-Extended and blocks GPTBot, ClaudeBot and Applebot-Extended. axios.com allows Google-Extended and blocks CCBot.
• Nobody carries a shared signal yet. None of the 61 files has a Content-Usage line or a __usage_ai-training group, which is unsurprising for a draft posted that day. Two files carry Cloudflare’s Content-Signal line: cloudflare.com with ai-train=yes, and theatlantic.com with ai-train=no repeated four times.

The premise of the OpenAI post holds in these files: the per-vendor lists are long, and they are maintained one vendor at a time. They also differ in ways a file cannot explain. An explicit Allow for one token beside Disallow for five others looks like a choice, but we did not ask these publishers. Those five files show the limit of a purpose-level signal, which can say what content may be used for and not which vendor may use it, and the draft leaves room for that: “Private agreements MAY override preferences expressed via this mechanism.” An operator who wants to treat vendors differently keeps the vendor-keyed groups, so for that operator the new signal is one more line to maintain.

Cloudflare’s Content-Signal is the existing attempt to put a purpose in robots.txt, and its own documentation example pairs the line with eight vendor-keyed Disallow groups. Its ai-train definition is wider than the token draft’s (see the scope table above). We covered the enforcement side of that approach in Cloudflare’s Bot Preference Sync.

What to do now, and what to watch

• Keep the vendor-keyed groups. OpenAI says its existing crawler controls “will continue to apply and be respected,” and nothing in the announcement is implemented yet.
• If you add a token group, state the whole intent. Block / first and list exceptions, as Figure 1 does. A group that blocks only one folder reads as a preference for training on the rest.
• Do not rely on a * group to cover it, and do not mix the token into a crawler’s group unless you also want that crawler blocked.
• Do not expect it to stop a crawl. The draft says it is not a crawling directive. User-initiated fetchers are a separate question: OpenAI’s documentation says of ChatGPT-User that “robots.txt rules may not apply,” which we looked at in our earlier post on agent blocks.
• If you write a consumer, check the group first. Test whether any group names the token before you call a parser, and treat a wildcard-only file as no preference.

Four things will decide whether the signal matters. The working group may settle on the Content-Usage rule, the token group or both, and the token draft is revision 00 of an individual submission. OpenAI has promised an implementation update “in the coming months.” The other providers it says are joining the announcement have not yet been named in anything we could find. And the questions the post leaves open (how a missing group, an allowed path and a blocked path are treated, and whether fine-tuning counts) are the ones that determine what a site owner actually gets.

Reproduce it

The lookup is a plain RFC 9309 matcher plus the section 4.2 rule. Save the two listings below as policy_pref.py and check.py in one folder, install the two third-party parsers and run the check. The parser results above came from Python 3.13.14. The survey numbers depend on the files the sites served on October 30.

python -m venv .venv
.venv/bin/pip install protego robotspy
.venv/bin/python check.py

On Windows the folder is .venv\Scripts instead of .venv/bin.

# policy_pref.py
"""A small robots.txt matcher plus the lookup that
draft-kelley-aipref-training-preference-signal-00 describes (sections 4.1 and 4.2)."""
import re

TOKEN = "__usage_ai-training"


def parse_groups(text):
    """Return [(agents, rules)]. Consecutive User-agent lines share one group.
    Unknown fields (Sitemap, Content-Signal, ...) are ignored and do not end a group."""
    groups, agents, rules, in_rules = [], [], [], False
    for raw in text.replace("\r\n", "\n").replace("\r", "\n").split("\n"):
        line = raw.split("#", 1)[0].strip()
        if ":" not in line:
            continue
        field, value = (part.strip() for part in line.split(":", 1))
        field = field.lower()
        if field == "user-agent":
            if in_rules:
                groups.append((agents, rules))
                agents, rules, in_rules = [], [], False
            agents.append(value.lower())
        elif field in ("allow", "disallow") and agents:
            in_rules = True
            rules.append((field, value))
    if agents:
        groups.append((agents, rules))
    return groups


def _regex(pattern):
    anchored = pattern.endswith("$")
    body = pattern[:-1] if anchored else pattern
    return re.compile("^" + ".*".join(re.escape(p) for p in body.split("*")) + ("$" if anchored else ""))


def allowed(rules, path):
    """RFC 9309 section 2.2.2: the longest matching rule wins, Allow wins a tie, no match means allowed."""
    best_len, best_allow = -1, True
    for kind, pattern in rules:
        if pattern and _regex(pattern).match(path):
            size = len(pattern.encode())
            if size > best_len or (size == best_len and kind == "allow"):
                best_len, best_allow = size, kind == "allow"
    return best_allow


def rules_for(groups, token, wildcard):
    """Merge every group that names the token. Fall back to '*' only when the caller says so."""
    token = token.lower()
    named = [rules for agents, rules in groups if token in agents]
    if named:
        return True, [r for rules in named for r in rules]
    if wildcard:
        star = [rules for agents, rules in groups if "*" in agents]
        return bool(star), [r for rules in star for r in rules]
    return False, []


def crawl_allowed(text, crawler, path):
    """The classic question: may this crawler fetch the path? (wildcard group applies)"""
    return allowed(rules_for(parse_groups(text), crawler, wildcard=True)[1], path)


def training_preference(text, path):
    """The draft's question: 'against', 'in favor' or 'none' (wildcard groups never match the token)."""
    found, rules = rules_for(parse_groups(text), TOKEN, wildcard=False)
    if not found:
        return "none"
    return "in favor" if allowed(rules, path) else "against"

# check.py
"""Three checks on the lookup rules in draft-kelley-aipref-training-preference-signal-00.
Needs: pip install protego robotspy    (urllib.robotparser ships with Python)"""
import urllib.robotparser as urp
from importlib.metadata import version

import robots
from protego import Protego

from policy_pref import TOKEN, crawl_allowed, training_preference


def std(text, ua, path):
    parser = urp.RobotFileParser()
    parser.parse(text.splitlines())
    return parser.can_fetch(ua, "https://example.com" + path)


def pro(text, ua, path):
    return Protego.parse(text).can_fetch("https://example.com" + path, ua)


def rbs(text, ua, path):
    return robots.RobotsParser.from_string(text).can_fetch(ua, "https://example.com" + path)


LIBS = (("stdlib", std), ("protego", pro), ("robotspy", rbs))
word = lambda ok: "allow" if ok else "block"

SCENARIOS = [
    ("fig1: draft Figure 1", f"User-Agent: {TOKEN}\nDisallow: /\nAllow: /press-kit/\n", ["/press-kit/x", "/blog/"]),
    ("star: only a crawl rule", "User-agent: *\nDisallow: /private/\n", ["/private/x", "/open/"]),
    ("partial: one folder blocked", f"User-agent: {TOKEN}\nDisallow: /private/\n", ["/private/x", "/open/"]),
    ("empty: group with no rules", f"User-agent: {TOKEN}\n", ["/open/"]),
    ("two groups name the token", f"User-agent: {TOKEN}\nDisallow: /a/\n\nUser-agent: {TOKEN}\nDisallow: /b/\n", ["/a/x", "/b/x"]),
    ("wildcard path in the group", f"User-agent: {TOKEN}\nDisallow: /*.pdf$\n", ["/report.pdf", "/report.html"]),
    ("blank line after User-agent", f"User-agent: {TOKEN}\n\nDisallow: /\n", ["/open/"]),
    ("mixed case token", "User-Agent: __Usage_AI-Training\nDisallow: /\n", ["/open/"]),
]

print("=== 1. the same file, asked can_fetch(token, path) by three parsers")
print("python", __import__("sys").version.split()[0], "| protego", version("protego"), "| robotspy", version("robotspy"))
print(f"{'scenario':<29}{'path':<14}{'draft says':<11}{'stdlib':<7}{'protego':<8}{'robotspy'}")
wrong, rows = {name: 0 for name, _ in LIBS}, 0
for name, text, paths in SCENARIOS:
    for path in paths:
        truth, cells = training_preference(text, path), []
        for lib, fn in LIBS:
            got = fn(text, TOKEN, path)
            wrong[lib] += ("in favor" if got else "against") != truth
            cells.append(word(got))
        rows += 1
        print(f"{name:<29}{path:<14}{truth:<11}{cells[0]:<7}{cells[1]:<8}{cells[2]}")
print(f"\nrows where allow/block misreads the draft, out of {rows}:")
for lib, n in wrong.items():
    print(f"  {lib}: {n}")

BASE = "User-agent: *\nDisallow: /private/\nAllow: /private/public/\nDisallow: /tmp/\n"
VARIANTS = {
    "none": BASE,
    "first": BASE.replace("Disallow: /private/\n", "Content-Usage: train-ai=n\nDisallow: /private/\n", 1),
    "middle": BASE.replace("Allow: /private/public/\n", "Content-Usage: /ai-ok/ train-ai=y\nAllow: /private/public/\n", 1),
    "last": BASE + "Content-Usage: train-ai=n\n",
}
PATHS = ["/private/x", "/private/public/y", "/tmp/z", "/open/"]
print("\n=== 2. crawl decisions with and without Content-Usage lines in the group")
print("paths:", " ".join(PATHS), " (a = allow, b = block)")
for lib, fn in LIBS:
    seen = {name: "".join("a" if fn(text, "ExampleBot", p) else "b" for p in PATHS) for name, text in VARIANTS.items()}
    print(f"  {lib:<9}{seen['none']}  same in all four variants: {len(set(seen.values())) == 1}")

FILES = {
    "token group only": f"User-agent: {TOKEN}\nDisallow: /\n",
    "GPTBot group only": "User-agent: GPTBot\nDisallow: /\n",
    "one mixed group": f"User-agent: GPTBot\nUser-agent: {TOKEN}\nDisallow: /\n",
}
print("\n=== 3. GPTBot crawling /page, and the training preference for /page")
for name, text in FILES.items():
    print(f"  {name:<19} crawl {word(crawl_allowed(text, 'GPTBot', '/page')):<6} preference {training_preference(text, '/page')}")

The 64 sites in the survey, in the order we fetched them: nytimes.com, washingtonpost.com, theguardian.com, bbc.com, cnn.com, reuters.com, apnews.com, bloomberg.com, wsj.com, ft.com, economist.com, forbes.com, businessinsider.com, npr.org, usatoday.com, latimes.com, wired.com, theverge.com, techcrunch.com, arstechnica.com, engadget.com, cnet.com, zdnet.com, theatlantic.com, newyorker.com, time.com, vox.com, axios.com, politico.com, nbcnews.com, foxnews.com, buzzfeed.com, huffpost.com, independent.co.uk, dailymail.co.uk, telegraph.co.uk, en.wikipedia.org, github.com, stackoverflow.com, reddit.com, medium.com, quora.com, nature.com, sciencedirect.com, arxiv.org, imdb.com, yelp.com, tripadvisor.com, goodreads.com, substack.com, bleepingcomputer.com, theregister.com, techradar.com, tomshardware.com, pcmag.com, gizmodo.com, mashable.com, slashdot.org, dev.to, news.ycombinator.com, wordpress.org, cloudflare.com, openai.com, anthropic.com.

The 24 crawler tokens we looked for in User-agent lines (the first six are the training tokens counted above): GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Bytespider, Amazonbot, cohere-ai, Diffbot, Meta-ExternalFetcher, FacebookBot, omgili, Timpibot, ImagesiftBot, YouBot.

Sources

• OpenAI, Making it easier for websites to express AI training preferences (October 30, 2026)
• draft-kelley-aipref-training-preference-signal-00, Machine-Readable Training Preference Signal (IETF datatracker; text)
• draft-ietf-aipref-attach-05, Associating AI Usage Preferences with Content in HTTP
• draft-ietf-aipref-vocab-08, A Vocabulary For Expressing AI Usage Preferences
• IETF AI Preferences (aipref) working group page
• RFC 9309, Robots Exclusion Protocol
• OpenAI crawler documentation
• Cloudflare, managed robots.txt documentation
• Python documentation, urllib.robotparser