Websites Closing Doors to AI Agents Turn robots.txt Into a Request Four Vendors Say Their Fetchers May Skip

On October 17, The Verge’s Emma Roth reported that users of Meta’s Muse (which Amazon blocked in September), xAI’s Grok, OpenAI’s Dots (launched September 29) and Instinct (valued at $10 billion) have run into blocks that stop their agents from doing tasks across popular websites. TechCrunch’s Sarah Perez had mapped the same problem on October 6, with responses from Yelp, Delta, United, eBay, Zillow and Walmart. Neither story settles a question that matters to anyone who runs a site or builds an agent: where does each block live? The answer decides who can lift it.

I checked three places. The first is what five AI vendors say about robots.txt for the fetchers that act when a person asks, and four of the five say it may not apply or is generally ignored. The second is the robots.txt files of the 16 sites the two stories name, 12 of which I could read: five name at least one user-initiated agent token (the product name a vendor gives such a fetcher, which is how a robots.txt file addresses it) and seven name none. The third is Cloudflare’s own documentation and posts, which show the network defaults that have applied to agent traffic since September 15. The pattern in these reports is that robots.txt, which RFC 9309 says is “not a form of access authorization,” is a weak place to read an agent policy, and that the stated policies mostly live in contracts, terms of use, network defaults and generic anti-bot checks.

What the coverage says each site is doing

The two stories describe different mechanisms behind what a user sees as one failure. Table 1 sorts what they report.

A license contract, terms of use and a human-verification challenge are all invisible to robots.txt. TechCrunch puts the user’s side plainly: “What’s not clear to the end user is whether or not the block is accidental on the website owner’s part.” EquiStamp CEO Chris Canal tells The Verge why that is hard to tell: “There are all kinds of things that are set up not even to block agents, but to block in general bots or programs that are trying to scam or hack,” and “These AI agents trigger those same safeguards.”

Yelp’s pay-to-enter route also has a technical cousin in Cloudflare’s Monetization Gateway beta, which lets a seller answer an unpaid request with HTTP 402, but nothing in either story says Yelp uses it.

What robots.txt can promise an agent

RFC 9309, the 2022 standard for the file, describes it as a way for service owners to control how content “may be accessed, if at all, by automatic clients known as crawlers,” and adds: “These rules are not a form of access authorization.” Whether a fetch that a person asked for counts as a crawl is a question each vendor answers in its own documentation, and the answers differ.

I did not test any fetcher, so these are statements of policy, not measurements of behavior. They still change what a Disallow line means. For OpenAI, Perplexity, Google and Meta, a line for the user-initiated token is a request that the vendor says the fetcher may skip or generally skips. Of the five pages, only Anthropic’s ties the user-initiated fetcher to the file. I did not check xAI or Instinct. Meta’s October 6 post on Muse says “Muse acts when a person asks it to,” but it does not say which fetcher Muse uses or whether Muse honors robots.txt, so whether the Meta-ExternalFetcher statement covers Muse is not established.

Perplexity’s page adds a firewall section. Sites that run a web application firewall “may need to explicitly whitelist Perplexity’s bots to ensure they can access your content,” using a rule that combines the user agent with published IP addresses. A fetcher that generally ignores the file and asks to be allowed at the firewall shows where its vendor expects the decision to be made.

What 12 robots.txt files say about five agent tokens

The check

The script below reads saved robots.txt files and reports which of five tokens each one names: ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher and Google-Agent. It groups lines the way RFC 9309 does, so that consecutive User-agent lines share the rules beneath them, and it reports whether a token has its own group, sits in a group with other agents or shares the * group.

# check_agent_tokens.py
# For each saved robots.txt in a folder, report which user-initiated agent tokens the file names
# and what the group that names them says.
# Usage: python check_agent_tokens.py robots_folder
import pathlib
import sys

AGENTS = ["ChatGPT-User", "Claude-User", "Perplexity-User", "Meta-ExternalFetcher", "Google-Agent"]


def groups(text):
    """Split a robots.txt into groups: consecutive User-agent lines share the rules that follow."""
    out, current, last_was_agent = [], None, False
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip()
        if ":" not in line:
            continue
        key, value = (part.strip() for part in line.split(":", 1))
        key = key.lower()
        if key == "user-agent":
            if current is None or not last_was_agent:
                current = {"agents": [], "rules": []}
                out.append(current)
            current["agents"].append(value.lower())
            last_was_agent = True
        elif key in ("allow", "disallow"):
            if current is not None:
                current["rules"].append((key, value))
            last_was_agent = False
        else:
            last_was_agent = False
    return out


def describe(group):
    disallow = sum(1 for k, v in group["rules"] if k == "disallow" and v)
    allow = sum(1 for k, v in group["rules"] if k == "allow" and v)
    if ("disallow", "/") in group["rules"]:
        what = "Disallow: /" + (f" plus {allow} Allow" if allow else "")
    else:
        what = f"{disallow} Disallow, {allow} Allow"
    if "*" in group["agents"]:
        who = "shares the * group"
    elif len(group["agents"]) == 1:
        who = "own group"
    else:
        who = f"group of {len(group['agents'])} agents"
    return f"{what} ({who})"


for path in sorted(pathlib.Path(sys.argv[1]).glob("*.txt")):
    all_groups = groups(path.read_text(encoding="utf-8", errors="replace"))
    found = []
    for token in AGENTS:
        for g in all_groups:
            if token.lower() in g["agents"]:
                found.append(f"{token}: {describe(g)}")
    print(path.stem)
    for line in found or ["none of the five named"]:
        print("  " + line)

I fetched /robots.txt from the 16 sites the two stories name (Amazon, Walmart, Delta, United, Yelp, eBay, Zillow, Pizza Hut, Adidas, Chase, Schwab, Cars.com, Amtrak, Best Buy, Sephora and Gap) on October 17, starting at 19:37 UTC, with Python and a Chrome user agent. A second pass at 19:40 returned the same bytes for all 10 files that loaded. Amtrak and Best Buy timed out on both passes and loaded in headless Edge. United and Pizza Hut returned nothing. Adidas and Cars.com answered with a block page, in Edge as well, which for Cars.com was a Cloudflare page headed “Sorry, you have been blocked.” Those four sites are not in the results. The output for the 12 readable files, saved in a folder named robots:

www.amazon.com
  ChatGPT-User: Disallow: / (own group)
  Claude-User: Disallow: / (own group)
  Perplexity-User: Disallow: / (own group)
  Meta-ExternalFetcher: Disallow: / (own group)
www.amtrak.com
  none of the five named
www.bestbuy.com
  ChatGPT-User: 137 Disallow, 2 Allow (group of 12 agents)
  Claude-User: 137 Disallow, 2 Allow (group of 12 agents)
  Perplexity-User: 137 Disallow, 2 Allow (group of 12 agents)
www.chase.com
  none of the five named
www.delta.com
  none of the five named
www.ebay.com
  ChatGPT-User: 210 Disallow, 13 Allow (group of 4 agents)
  Claude-User: 210 Disallow, 13 Allow (group of 4 agents)
www.gap.com
  ChatGPT-User: 16 Disallow, 1 Allow (shares the * group)
www.schwab.com
  none of the five named
www.sephora.com
  none of the five named
www.walmart.com
  none of the five named
www.yelp.com
  ChatGPT-User: Disallow: / plus 2 Allow (group of 6 agents)
  Claude-User: Disallow: / plus 2 Allow (group of 6 agents)
  Perplexity-User: Disallow: / plus 2 Allow (group of 6 agents)
  Meta-ExternalFetcher: Disallow: / (group of 9 agents)
www.zillow.com
  none of the five named

What the files show

Four readings follow, each limited to what the files show.

The security-framed blocks are not in robots.txt. Delta, Chase and Schwab name none of the five tokens. For the two banks that is expected, since The Verge describes blocks on account access and RFC 9309 says the rules are not access authorization. Delta’s reported steps against unauthorized automated activity are likewise not published as a rule for any of the five.

Amazon and Yelp publish what their stated policies say, and the vendors’ pages say a fetcher may not honor it. Amazon gives each of four tokens its own Disallow: / group. Yelp’s file closes the same four tokens, apart from two article-path Allow lines for three of them, and the data license that TechCrunch says admits paying agents is not something a robots.txt file can show.

Best Buy publishes a policy of the opposite kind. The Verge lists Best Buy among the retailers Muse has partnered with. Its file puts ChatGPT-User, Claude-User and Perplexity-User in a group under a comment that reads “Allows conversational answer engines to securely access clean landing pages.” The group allows /site/ and /discover-learn/ and disallows 137 patterns, including the cart, profile and fulfillment paths. Walmart and Sephora, two other retailers the Verge lists as partners, name none of the five tokens, so a partnership does not show up in the file either, and Gap names ChatGPT-User only inside its * group.

Naming a token is not the same as blocking it. Zillow, which told TechCrunch it allows agents acting for consumers, blocks three other tokens (Meta-ExternalAgent, Amazonbot and Applebot-Extended) but none of the five.

The 16 sites are the ones the stories name, not a sample. A site can serve a different file by hostname, region or client, and I fetched only the www host. A token that is not named is still covered by the * group for any crawler that honors the file, and naming a token shows intent, not enforcement.

Where a block can bite: Cloudflare’s Agent defaults

The defaults

Cloudflare sorts AI traffic into three behaviors that customers can control separately. Its July 1 changelog defines Agent as “automated activity acting in real time on a person’s behalf, such as chat fetch bots and browser-use agents,” which in my reading is the same population the vendors call user-initiated fetchers. Since September 15, “new domains onboarding to Cloudflare receive updated defaults: Bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed.” In the onboarding presets Cloudflare published that day, a site that says it earns money from ads gets Agent set to Block on pages with ads, and a site that does not gets Agent set to Allow.

The stated reason is revenue, not security. In the September 15 post, Cloudflare writes: “Ad revenue depends on a human actually seeing the page. Training replaces that visit with an answer; agents fetch the page with nobody there to see the ads.” Cloudflare’s July 1 report, which we covered, made a related economic argument about agents and publisher traffic.

The migration table

The same post’s “What you need to do” section says: “Nothing, in almost every case. Your current settings carry over on their own.” Its migration table is where the Agent change shows. For domains that never used the granular controls, the new settings follow the legacy Block AI bots toggle:

So a domain that never touched the granular controls but had the legacy toggle on, set either way, ends up with Agent set to Block on pages with ads. Cloudflare’s documentation describes the legacy toggle as one that “blocks verified bots that are classified as crawling for the purpose of AI training, as well as a number of unverified bots that behave similarly,” which does not mention agents. I cannot tell how many sites that affects. Cloudflare said on July 1 that it would keep notifying customers as September 15 approached, so a migrated site may have been warned.

For agents there is enforcement but no published preference. Cloudflare says so directly: “the Internet does not yet have a well-established directive for expressing Disallow preferences to agents. For now, we’re not including a Disallow setting for Agents.” Its new Disallow AI Training setting for training crawlers works by publishing a preference in robots.txt through Bot Preference Sync, the feature we covered in August. For the Agent class the choices are Allow, Block on pages with ads and Block.

TechCrunch reports that some people suspect Cloudflare and other content delivery networks of disrupting Muse traffic, and that Cloudflare “didn’t have any specific data to share at this time.” Nothing I read ties these defaults to any particular Muse failure. Walmart says its failures were not intentional, and TechCrunch points to a human-verification button. What the documents establish is narrower: since September 15, a request that Cloudflare classifies as an agent can be refused at the network edge on a site whose robots.txt says nothing about agents, and for a site migrated from the legacy toggle, without the owner having picked an agent setting.

Cloudflare’s October 16 post on verified bots makes the same point from the enforcement side: “a robots.txt sets the minimum rules for what bots and agents should not do. But there is no implication that anything not stated is allowed.” It also says why identity alone does not settle trust: “verifying who operates it creates accountability, but trust depends on what it does next.”

What the proposed protocol adds, and what it leaves open

Meta’s answer, with Sierra, is a standard. They announced Personal Agent Protocol on October 6 with partners including Genesys, Rocket, Shopify, Stripe and Walmart, and on October 9 Sierra published a draft, nicknamed Poppy, with 35 more design partners including Cloudflare, OpenAI, United Airlines, Gap Inc., Bank of America and Wells Fargo. In the draft, companies publish one file at a standard address on their website, /.well-known/poppy.json. Sierra says it “tells personal agents how to interact with the company, how customers can sign in, and what the agent can do,” and that “The personal agent identifies itself, so the company always knows when an agent is acting for a customer.”

Meta’s post gives the business argument: “Turning away a personal agent means turning away the customer behind it. We’d rather work with businesses to address the underlying concerns than see them lose that customer.” That is a vendor with a product to ship making the case, and it is aimed at sites that want the sale. It does not speak to a site like Yelp, which wants agents to pay for the data.

Three things stay open. First, the posts describe a file that tells agents how to interact and what they can do. I did not find a description of how a site says no, and I did not read the draft specification itself. Second, the design depends on the agent identifying itself, the same kind of promise robots.txt rests on. Google says it is experimenting with Web Bot Auth, a request-signing scheme, using the identity agent.bot.goog, and Cloudflare’s verified-bots post says identity does not settle trust. Third, the sites in Table 1 are mostly absent. Neither Sierra post names Amazon, Delta, JPMorgan Chase, Schwab, Yelp, eBay, Zillow, Google or Anthropic, though the partner lists do include United Airlines, whose terms of use bar robots without its written permission, and banks including Bank of America and Wells Fargo. Sierra said on October 6 that it plans “to publish the v0.1 specification later this month.”

What site owners and agent builders can do now

For site owners:

• Count the tokens in your logs. Search for ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Google-Agent and Claude-User. OpenAI, Perplexity and Google link published IP ranges for their fetchers (chatgpt-user.json, perplexity-user.json and user-triggered-agents.json), and Perplexity’s page recommends checking the user agent and the IP address together.
• Put the decision where it can be enforced. For agents that is the network or the contract, because the file may be skipped. On Cloudflare, the policies are under Security Settings, Configure AI bot policies, and a zone migrated from the legacy toggle should be checked for what its Agent setting became.
• If agents are welcome, say so where they will look. Best Buy’s file shows one pattern, allowing landing pages and disallowing cart and profile paths. The Poppy discovery file is a second option once the specification settles.

For agent builders:

• Expect failures from layers that do not read your documentation. Canal’s point about general anti-bot safeguards applies to agents that drive a browser.
• Publish identity and IP ranges. OpenAI, Perplexity and Google already publish ranges, and Google’s Web Bot Auth identity is an early example of signed requests.
• Treat partnerships as a clearer path, not a guarantee. The Verge reports that “Instead of trying to fight Amazon, the company has partnered with retailers like Walmart, Gap, Sephora, Best Buy, and others, allowing Muse to shop on their sites.” Walmart is one of those partners, and TechCrunch still reports failures there.

What is not established

• Whether the Cloudflare defaults explain any of the Muse failures. Cloudflare told TechCrunch it had no specific data, Walmart says its failures were not intentional, and TechCrunch points to a verification button.
• Whether the fetchers behind Muse, Grok, Dots or Instinct are any of the five tokens. Meta’s post does not say, and I did not check xAI or Instinct.
• How many sites run an Agent preset, or how many were migrated from the legacy toggle.
• How Delta, Chase and Schwab block agents. Their statements do not say, and their robots.txt files name none of the five tokens.
• Whether any vendor’s fetcher behaves as its page says. I tested none.
• Anything about the four sites I could not read.

Sources and method

The sources are The Verge (October 17) and TechCrunch (October 6); the crawler pages of OpenAI, Perplexity, Google, Meta and Anthropic; RFC 9309; Cloudflare’s changelog, documentation and posts of July 1, September 15 and October 16; and Sierra’s October 6 and October 9 posts. Meta’s crawler page and Muse post need JavaScript, so I read them in headless Edge.

I saved the text of each source and checked every quotation in this piece against it. I ran no experiment on any agent or fetcher. The file and token counts are mine, produced by the script above on files fetched on October 17, and robots.txt files can change at any time. The reading of the Cloudflare migration table is mine, taken from the table as published on September 15.