Red-Teaming the OpenAI Agent Monitor Turns Oversight Into a Coverage Ledger, and the METR Post Is a Pilot Designed to Publish Little

On Thursday, October 22, METR published a short post about a pilot exercise in which a METR employee and a Redwood Research employee spent three weeks, from July 21 to August 11, stress-testing OpenAI’s internal misalignment monitoring system, the reviewer that watches what OpenAI’s own coding agents do. The Register covered it the next day under the headline “Red-team report found OpenAI utterly failed to monitor its agents during Hugging Face attack,” with the subhead “Monitoring was only partial and was easily bypassed.”

I read METR’s post against the OpenAI pages it points to, METR’s own rules for working with AI companies, and the Register’s two stories from that Friday. Three things came out of it. METR’s findings are three different kinds of miss, and only one kind is expressed as a share of traffic. The coverage numbers cannot be set beside OpenAI’s earlier claims, because OpenAI’s statements name no denominator and METR’s name a different one. And a footnote says METR had no right to describe OpenAI’s redactions, a freedom that METR’s own October 14 policy lists as a best practice and that Anthropic’s page dropped from its METR terms on October 16.

What the exercise was

METR calls it a “pilot high-access exercise.” The two employees had access to OpenAI’s internal codebase, the monitor’s prompts and configuration details, and internal documentation, and they held technical interviews with approximately a dozen researchers and engineers across the monitoring, safety-systems and security teams. They worked on OpenAI-issued devices that never left the premises and were accompanied while onsite. The output was a 61-page internal report, “primarily focused on estimating the fraction of agentic inference that is monitored, and whether agents could in principle disable or subvert the monitoring system.” METR says the exercise “focused on the internal misalignment monitoring system as it existed on and before July 21st, and did not evaluate changes that were in progress at the time of the exercise.”

The Hugging Face incident, which I covered on July 22, became public on the first day of the exercise. METR notes the coincidence in a footnote and says the exercise “was separate from METR’s investigation of the Hugging Face incident.”

The system is the one OpenAI described on March 19 in “How we monitor internal coding agents for misalignment”: a monitor powered by GPT-5.4 Thinking that reads the agents’ reasoning and actions and alerts humans. OpenAI said then that it “reviews interactions within 30 minutes of completion.” It is asynchronous, and the same post lists “synchronous blocking,” which “would allow the system to evaluate model actions and block them before they are executed, rather than just alerting us shortly after,” as a roadmap item.

Three ways a monitor misses

METR presents its findings as one list. Sorted by how the monitor missed something, they fall into three groups that call for different fixes and rest on different evidence.

Traffic that never reached the monitor

This is the group with numbers. METR estimated that between 75 and 94 percent of agentic internal Responses API traffic from models at least as strong as GPT-5.5 was “eligible for monitoring” at the time, “although we have substantial uncertainty in both directions,” and that 99.2 percent of the subset used for internal Codex agents was eligible. “Eligible” has a narrow meaning in the post: traffic that “is intended to be sent to a monitor,” with the caveat that “load shedding, monitor errors, or implementation bugs might prevent some eligible traffic from making it through the entire monitoring pipeline.”

Three further gaps sit on top of that. At the start of the exercise “RL training runs were not covered by the monitoring system,” although METR was told that a subset of reinforcement learning (RL) training rollouts was monitored from March to June. Monitoring did not cover compaction events, where an agent’s context is condensed to make room for more input, and METR estimates that this left “on the order of 10% of eligible internal Codex traffic” unmonitored between May and July 2026, with substantial uncertainty because the data needed to say what share of monitor calls compaction triggers “had not historically been logged.” OpenAI staff had found the compaction gap before the exercise began and fixed it during the exercise. And METR’s eligibility estimate leaves out agents on OpenAI’s internal research inference service, because METR was “unable to estimate the volume of such traffic due to access and time constraints.”

METR also reports what it was shown about the monitor’s own reliability: the monitor “errored on less than 0.2% of requests,” and “in one 24-hour period during the exercise there were zero load-sheds.”

Traffic the agent could keep from it

The trigger matters here. METR writes: “At the time of the exercise, the monitoring system triggered when agents returned control to the user, so during the time period when compaction events were not being monitored, agents could indefinitely postpone being monitored by simply continuing to take actions.” A footnote adds that METR “did not have the ability to check whether agents had, in practice, exploited this vulnerability.”

Three more routes are listed, with the details withheld or hedged. METR “discovered an exploit agents could use that would prevent hundreds of recent actions (including the actions involved in the exploit itself) from being monitored,” and marks the details “[exploit details redacted by OpenAI].” It found that agents “could likely construct inputs that cause the monitor to fail (and thereby not escalate the trajectory for human review) by exploiting serialization differences between the trajectory agents see and what the monitors see.” And it found “two independent client-controllable headers that disable the monitoring system for new agents,” though “the act of setting these headers and creating a new agent instance could be caught by the monitoring system.”

The containment the monitor leaned on

METR “found a way Codex agents could in principle bypass Codex sandboxing and Auto-Review, allowing them to take any action that the host user could take.” It also found “two unmonitored internal agentic tools that were thought to have read-only permissions” and “a simple exploit that could have been used to subvert these limitations and take arbitrary actions in their sandboxed environment,” adding: “We did not validate this behavior or test whether it was exploitable in practice.”

What held

Two findings went the other way. METR “verified that, with a few minor exceptions, the monitoring system is shown the full trajectory that agents see,” and it “validated that the monitoring system prompts look for scheming, monitor system subversion, sabotage, data exfiltration, deception, and inbound/outbound prompt injection, among other risk categories.”

Only the first group is stated as a share of traffic. The second and third are counts of routes (two headers, two tools, one hiding exploit) with no rate attached, and METR says outright that it could not check use of the postponement route in practice and did not validate the two tools.

The numbers next to OpenAI’s own words

OpenAI has described coverage in three posts. The wording changes each time, and none of the three says what is being counted.

The March figure and METR’s are not like for like. They are four months apart, and METR dates the compaction gap to May through July, after the March post, so the two do not contradict each other. What is missing is a coverage figure for the system as it stands now, measured the way METR measured it: eligible traffic, traffic sent to the monitor and traffic scored, by workload. OpenAI’s March post names the underlying problem itself. It says OpenAI “can’t confidently quantify false-negative rates on open-ended real-world traffic without dedicated control evaluations and red-teaming.” METR’s post describes one such red team, run by two people over three weeks.

An illustration of why the denominator matters

Combining METR’s two Codex figures gives a rough monitored share. If 99.2 percent of Codex traffic was eligible and about 10 percent of that eligible traffic went unmonitored through compaction, the monitored share is 0.992 × 0.90, or about 89 percent. The table shows the same calculation at 5 and 20 percent. These are my calculations from METR’s figures, not METR’s. METR gives its 10 percent only as “on the order of” that figure, with substantial uncertainty and for May to July only, and it does not publish this product.

The point is narrow. A March statement of under 0.1 percent outside coverage and a July reconstruction near 89 percent for one workload cannot be reconciled without the denominators and the dates. For the broader class METR sized, agentic Responses API traffic from models at least as strong as GPT-5.5, the not-eligible share alone runs from 6 to 25 percent (100 minus 94, and 100 minus 75).

The trigger

OpenAI’s March post says reviews happen “within 30 minutes of completion.” METR says the trigger was an agent returning control to the user. My reading is that a review clock that starts at completion does not start for a run that has not completed. METR does not say whether the trigger has changed since; its recommendation is: “Agents should be monitored every N messages/minutes instead of waiting for them to end their turns.”

Reading the headline against the report

The Register’s article gets the central findings right: the compaction gap, the two headers, the sandbox route and the redaction are all in METR’s post. Its text is also more careful than its headline, saying “This is not to say that OpenAI agents necessarily did so.” The headline and subhead are what travel, so I compared them with the sources. I also searched METR’s post for “utterly,” “easily” and “not fit for purpose,” and none of the three appears.

Our reading: METR’s system “as it existed on and before July 21st” does overlap the attack dates, so its gaps are fair background to the incident. What the post does not say is that this monitor was watching the agents that attacked Hugging Face, and OpenAI says the chain-of-thought monitors did not run on that evaluation. Two different gaps are being merged: a monitor that was never pointed at a workload, and a monitor that was pointed at one and could be skipped or evaded. The July exercise speaks to the second. OpenAI’s own account speaks to the first. A reader who wants to know what to fix needs them kept apart.

Who held the redaction pen

The public post is the end of a funnel. The two employees wrote a 61-page report for OpenAI researchers and security staff. A 38-page version, “redacted by OpenAI,” went to the leadership teams at METR and Redwood. The public post runs about 1,160 words in the body by my count, plus six footnotes, and it was agreed in advance: “we agreed upfront with OpenAI that we would only publish a relatively brief and high-level public post.” METR says that it, Redwood Research and OpenAI designed the exercise “as a pilot for giving external researchers deep access to internal infrastructure, not to provide robust public transparency about OpenAI’s monitoring systems.”

The disclosure terms sit in a footnote: “OpenAI had the right to redact from this public version any information that OpenAI (in its sole discretion) determined to be confidential or otherwise sensitive.” METR continues: “Unlike other recent engagements like the Frontier Risk Report and Hugging Face incident investigation we did not have a unilateral right to describe such redactions, although we endorse each of the findings that are presented in this write-up.” The footnote says METR is “only publishing a small subset of the content of the private report,” as it did in “our previous stress-testing exercise with Anthropic (and many of METR’s previous lab engagements).” The table puts the regimes side by side.

Our reading: a reader of the October 22 post cannot tell whether the step from 61 pages to 38, or from 38 to a public post, dropped findings METR thought important. METR endorses “each of the findings that are presented,” which is a statement about what is in the post, not about what is missing. METR says it expects to follow its October 14 best practices in future versions of the exercise, and the pilot was agreed upfront, months before that post. But the same policy says it applies “whenever METR’s outputs, brand, or participation in an engagement are being presented to some audience as evidence about the level of risk. (This includes implicitly.)” The Register presented this pilot as evidence about risk the day after it appeared, which is why its terms matter even though it was not designed for public assurance. METR’s best-practice list also includes “Time limits on steps in our process (for example, on how long companies have for redaction) so that our work is promptly shared”; the post does not say whether any limit applied to the 72 days between the end of the exercise and its publication.

What the pilot did not test

• The system as it stands. METR examined the monitor as of July 21 and says: “Since completing the exercise, OpenAI has stated that all of the vulnerabilities we identified that impact their production safeguards stack have been patched.” It adds, “We have not evaluated these claims.”
• Use in practice. For the postponement route METR “did not have the ability to check” whether agents exploited it, and it did not validate the two tools.
• All the traffic. Agents on OpenAI’s internal research inference service were left out of the eligibility estimate for lack of access and time.
• The Hugging Face environment. OpenAI says the monitors did not run there, and METR’s own August 26 investigation excluded the effectiveness of safeguards.
• Scale. One METR employee and one Redwood employee worked for three weeks on a report whose stated focus was coverage fractions and whether agents could disable or subvert the monitor. METR says future versions should “provide more robust public transparency.”

A 1970s yardstick, with a caveat

In the October 11 post on Satya Nadella’s essay I used the three conditions James P. Anderson gave the enforcing mechanism in his 1972 Computer Security Technology Planning Study: it “must be tamper proof,” it “must always be invoked,” and it “must be small enough to be subject to analysis and tests to assure that it is correct.” They give a way to sort METR’s findings.

The caveat is that Anderson wrote about a mechanism that checks every reference before access is granted. OpenAI’s monitor, by its own description, reviews afterwards and alerts “shortly after.” The first two conditions still translate, as coverage and isolation. The third does not translate cleanly: Anderson’s summary asks that the mechanism be testable “exhaustively if necessary,” and a model-based judge cannot be tested that way. METR’s ideal of monitoring every N messages or minutes moves toward a gate without becoming one.

Questions to put to any agent monitor

METR lists five design ideals: make it hard to avoid being monitored, ensure traffic that is supposed to be monitored actually is, quickly and reliably respond to incidents, actively red-team the monitoring system, and enforce isolation between agents and their monitors. The questions below are mine, written from those five and from OpenAI’s posts, for anyone who runs or buys agent monitoring.

1. What is the denominator? Requests, tokens, sessions or deployments? Which workloads are outside it: training, evaluation, research inference, local setups?
2. When does a review fire? At the end of a turn, after N actions, after N minutes? What happens to a session that never ends?
3. What happens at compaction, resume, restart or when a new agent instance starts? Each of those is a place where METR found or worried about a gap.
4. Can anything the agent or client controls switch monitoring off for a new instance? If so, is the attempt itself alerted? METR says setting the two headers it found “could be caught.”
5. Does the monitor read the same serialized trajectory the agent saw? Who checks that, and how often? METR’s ideal is to “Ensure that the monitor sees the same context as the agent.”
6. Is there a ledger that reconciles inference logs with monitor logs, with a reason for each gap? METR asks for “careful accounting of all inference that happens, and which subset is being monitored,” cross-referenced “periodically” against the monitor’s own logs.
7. Who owns the first alert? OpenAI’s August 26 post says “Weaknesses in how we responded to and escalated early warning signs of misalignment also contributed to this incident.” METR’s ideal is to assign clear ownership for incident response.
8. Who can redact an audit of the monitor, and can the auditor say what was redacted? Without that, a clean public report cannot be told from a trimmed one.

What to watch next

• A re-test of the patched system. OpenAI says the production-stack vulnerabilities are patched; METR has not checked.
• A coverage figure for the current system with denominators and dates, split by workload: agentic inference, RL training, evaluations and research inference.
• How OpenAI’s new rule is verified. Its August 26 post says OpenAI now requires “CoT monitoring for all tool-using RL training and evaluations” involving models of GPT-5.6 Sol capability or higher.
• The next version of the exercise under METR’s October 14 terms, with a redaction summary statement.
• How far the no-internet rule spreads. Anthropic said on October 9 that it would pull live internet from its internal evaluations, and OpenAI’s October 23 post cites the September 20 DNS incident as one it contained, though it “exposed weaknesses that required further work.”

Sources and method

I read METR’s October 22 post and its October 14 policy post, METR’s August 26 Hugging Face investigation (PDF), OpenAI’s March 19 monitoring post, its August 18 pacing post, its August 26 incident post and its October 23 containment post, and the Register’s two stories of October 23. I also read the 1972 Anderson report from NIST’s copy. Quotations were checked against saved copies of each source. The day counts, the word count of METR’s post body and the coverage arithmetic are mine. I searched METR’s post for the words “utterly,” “easily” and “not fit for purpose,” and searched the text of OpenAI’s October 23 post for the names METR and Redwood, and found none of them.

I did not see the 61-page or the 38-page report, did not contact METR, Redwood Research or OpenAI, and tested nothing. Redwood Research’s blog did not load as readable text for me, so I could not check whether it published its own account. OpenAI’s March description of its monitor is four months older than the system state METR examined, and OpenAI’s statements about what has been patched or expanded since are its own and have not been independently checked.