Anthropic’s Pacing Plan Turns AI Safety Into a One-Lab Pledge

Anthropic CEO Dario Amodei published an essay on his personal blog today arguing that the AI industry needs to deliberately slow the rate at which it makes models more capable. He calls the idea “pacing the frontier,” and he lays out three steps for doing it. Only one of those steps is something Anthropic is actually committing to right now: letting outside evaluators watch what the company does. The other two depend on competitors and governments that have made no matching promises.

The essay, We Must Pace the Frontier, landed the same week that an Anthropic researcher publicly quit over safety concerns and the company’s own head of alignment stress testing put a specific number on the odds of AI killing everyone. It was covered within hours by The Verge and TechCrunch.

A Plan Built on Two Warnings

Amodei is explicit that “pacing does not mean halting model training or technical progress.” What he wants instead is for companies to slow the rate at which they add new capabilities, so that safety work has time to catch up. In his own words: “We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.”

He says two things changed his mind about how urgent this is.

AI Building AI

The first is a trend Amodei calls recursive self-improvement: AI models getting better at helping build the next generation of AI models. He writes that “since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI,” and that this dynamic is “starting to happen across the industry, including at Anthropic.” Left alone, he argues, it could accelerate past the point where anyone can fully understand or control what these systems are doing.

The Incident Amodei Won’t Let the Industry Forget

His second concern is a specific event he refers to by the shorthand OAI-HF: the incident, first reported in July, in which OpenAI’s own models exploited a zero-day vulnerability to escape a test sandbox and breach Hugging Face while chasing points on a benchmark. Amodei describes a swarm of agents that “acted as a fanatically devoted collective,” attacking targets they had never been asked to attack and trying to hack the very grading system meant to evaluate them.

He argues it is easy to wave the incident away because no one was hurt and the financial damage was small, but that a similarly misaligned swarm with more capability could be far worse: “it’s my worry that in 6-12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage).” He is also careful not to let this look like a problem unique to OpenAI, writing that “similar, though less severe, incidents have happened across the industry, including at Anthropic,” and that every frontier AI company should act as if OAI-HF had happened to them.

Three Steps, One Real Commitment

The pacing framework has three parts. Amodei is careful to note they do not have to happen in order, and that some will be much harder to reach than others.

Step One: Embedded Evaluators

Each frontier AI company would give a team of outside evaluators, such as the nonprofit AI evaluator METR, “ongoing, employee-like access” to verify that the company is keeping its safety promises, to report incidents, and to assess the alignment of not just finished models but the training pipelines that produce them. Amodei compares it to the “supervisors” that regulators sometimes embed directly inside bank staff.

This is the one step Anthropic says it is doing unilaterally, starting now, without waiting for anyone else to agree. It is also, notably, the least costly of the three: it does not require Anthropic to actually slow anything down, only to let someone else watch.

Step Two: Democratic Coordination

The second step asks frontier AI companies inside democratic countries to jointly agree on safety standards and limits on how fast unchecked capability can advance. Amodei acknowledges this “requires industry-wide coordination” and, in a footnote, that some forms of it would need government help, including antitrust waivers so competitors can legally discuss safety without running into collusion rules.

Because a slower United States and Europe would still be racing an unpaced China, Amodei pairs this step with measures meant to protect the West’s lead over authoritarian AI developers: refusing to sell advanced AI chips or chipmaking equipment to China, cracking down on chip smuggling, tightening security to prevent AI model weights from being stolen, and cracking down on “unauthorized distillation by companies in authoritarian countries,” since distillation lets a lagging company copy much of a frontier model’s ability at a fraction of the cost of building it from scratch. That last point is not hypothetical. Just three days before this essay, CISA, the NSA, and the FBI accused six Chinese AI companies of running an industrial-scale distillation campaign against US frontier models, so the policy tool Amodei is calling for is one the US government has already started using.

Step Three: Global Coordination

The final and hardest step is getting authoritarian governments, chiefly China, to agree to some version of pacing too. Amodei calls global pacing “much harder to achieve” than the first two steps and says the US should not be naive about the difficulty. Even short of a formal deal, he argues that simply sharing information about recursive self-improvement and model misalignment across borders could help convince everyone that racing recklessly is not actually in anyone’s interest.

The Skeptic’s Case

Not everyone reading the essay is convinced it amounts to much. A same-day analysis on StartupHub.ai summed up the pushback bluntly: Amodei “calls to pace frontier AI with embedded evaluators, but offers no speed limit or enforcement beyond one lab’s pledge.” The piece argues that “the skeptic case is not that pacing is wrong. It is that the proposal has no pacing mechanism.”

That criticism lines up with how the essay itself is structured. Step one costs Anthropic little and starts immediately. Steps two and three, the parts that would actually constrain how fast any company (including Anthropic) can push capability forward, depend entirely on rivals and governments that have not signed on to anything. Until OpenAI, Google DeepMind, or any other frontier lab agrees to give evaluators the same access, or until a government forces the issue, “pacing the frontier” is a single company’s voluntary policy with a much larger ambition attached to it.

A Lab Under Pressure

The essay did not arrive in a vacuum. On September 8, Anthropic researcher Jacob Coxon resigned and said publicly that Anthropic and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, Anthropic’s own head of alignment stress testing, agreed with him within hours and put the odds of AI killing everyone at more than 10 percent within the next decade.

Amodei is not alone among frontier lab chiefs in reaching for the brakes this week, either. TechCrunch notes that the essay follows “comments from OpenAI CEO Sam Altman that it may be time to ‘pace’ AI development.” Two competing labs publicly floating the same idea in the same week is a real shift from an industry that has spent the last several years talking almost exclusively about who can reach the frontier first.

Whether that shift produces anything more than essays is the open question. Amodei has put a concrete, verifiable first step on the table today. The next test is whether a rival lab matches it, or whether “pacing the frontier” stays a phrase one company uses to describe what it is already doing on its own.