TRENDING
A person sweeping a metal detector across a wet sandy beach with a sea fort on the horizon
November 22, 2026
How to Write YARA Rules With YARA-X to Hunt Malicious Python .pth Startup Files and Measure False Positives Before You Trust Them
A peeling black-and-white metal door standing ajar, with a dark empty hall and bright windows beyond
November 22, 2026
AWS’s “Found a Zero-Day on Their Own” Post Never Names the Product; the Advisory Behind Its Link Shows a Server That Ships With No Login, a README Now Saying Keep It Off the Internet, and a Fix Merged October 19 That PyPI and Docker Hub Still Do Not Carry
A hand paging through index cards in an open wooden library card-catalog drawer labeled Agriculture, with a second drawer open below to the right
November 22, 2026
Microsoft Tells Customers Unauthorized Parties Read Admin Email Addresses and Tenant Metadata Through an Entra Connect Health Flaw Introduced July 23, but Gives No Tenant Count or Start Date, and Its Own Docs Send Tenant Security Notices to the Same Kinds of Contacts
Swimmers in separate lanes of a blue swimming pool, seen from above the water, with lane lines running across the picture
November 22, 2026
How to Size a PostgreSQL Connection Pool in Python: Find Where Throughput Stops Growing, Split max_connections Across Workers, and Stop Slow Handlers From Hoarding Connections
Workers in caps lean over a 1913 Ford assembly line, each assembling parts as they pass on the moving line
November 22, 2026
OpenAI’s Performance Factory Turns Regression Response Into a Task Queue for Agents, but Its Numbers Stop at 600 Root-Caused Cases in 1,700 Investigations and a 70% Grade on 100 Suggestions, and in a Toy World Each of Its Reviewer’s Two Counterfactual Questions Can Return a Confident Wrong Answer
22 Nov 2026
SXZ.io SXZ.io
  • Home
Search the Site
Popular Searches:
Technology Amazon AI
Recent Posts
The sun setting over the sea behind a pyramid-shaped stone monument at Cape Soya, the northernmost point of Japan
How to Retire an API Endpoint in FastAPI: Send Deprecation and Sunset Headers, Find the Clients Still Calling It, Schedule Brownouts, and Block an Early Removal With oasdiff
November 22, 2026
Brass sextant in an open wooden case inside a museum display, an instrument for measuring the angle to the horizon
MIT Technology Review’s “Most Misunderstood Graph” Update Reports METR’s Judgment Findings Mostly Fairly, but It Never Says Each Point Is a 50% Success Rate or Gives the New 3.6-Month Doubling Time and 2.1 to 4.9 Month Range, and It Credits OpenAI With a Comparison METR Built
November 22, 2026
Brass telescope on a wooden tripod inside a white observatory dome with the roof slit open to the sky
Rackspace Says Leaked Credentials Let an Intruder Act as Legitimate Users in Some Spot Customer Organizations and Reach Their Kubernetes Secrets, a Platform9 Co-Founder Names a Breached Observability System as the Root Cause, and the Status Page Says Only “System Maintenance Operations”
November 22, 2026
SXZ.io SXZ.io
  • Home

Categories

Articles 310 Posts
News 316 Posts
Learning Hub 282 Posts
Home/Articles/MIT Technology Review’s “Most Misunderstood Graph” Update Reports METR’s Judgment Findings Mostly Fairly, but It Never Says Each Point Is a 50% Success Rate or Gives the New 3.6-Month Doubling Time and 2.1 to 4.9 Month Range, and It Credits OpenAI With a Comparison METR Built
Articles

MIT Technology Review’s “Most Misunderstood Graph” Update Reports METR’s Judgment Findings Mostly Fairly, but It Never Says Each Point Is a 50% Success Rate or Gives the New 3.6-Month Doubling Time and 2.1 to 4.9 Month Range, and It Credits OpenAI With a Comparison METR Built

MIT Technology Review’s update on METR’s time-horizon report is mostly fair on the judgment gap, but it omits the 50% success rate and the 3.61-month doubling time, and it credits OpenAI with a...

November 22, 2026 22 Min Read
5

On November 20 MIT Technology Review published “The most misunderstood graph in AI just got a major update,” Grace Huckins’ report on METR’s November capability update, which METR itself titles “AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure.” The article’s message is that AI progress on checkable tasks is still exponential while models fall short on tasks that need judgment. Most of that holds up against METR’s report and against the chart data METR publishes beside each figure.

Table Of Content

  • The article against its sources
  • A point on the graph is a 50% success rate
  • Which doubling time?
  • What the interval does to the extrapolation
  • What METR put first
  • The verifiable tasks run faster, within wide intervals
  • Judgment work is five of 258 tasks
  • The 45 ratings behind “not one”
  • The philosophy task
  • The OpenAI sentence
  • Smaller points
  • What the article gets right
  • Reproduce the numbers
  • What this check does not cover
  • Sources

What does not hold up is mostly omission and attribution. The article never says that a point on the graph is a 50% success rate, never gives the report’s new doubling time or its interval, and credits OpenAI with a comparison that METR built from OpenAI’s figures. It also skips the sentence METR puts in its overview: “We think that current public evidence is insufficient to rule out drastic AI R&D acceleration in the next couple of years.” This post checks the article against the primary documents, recomputes the numbers from METR’s published chart data with two short scripts, and marks the places where the article is right.

The article against its sources

The table lists the article’s main factual sentences, what the sources say, and the result of the check. Details and numbers for each row follow in the sections below.

The article says The sources say Check
“every four months, the length of the tasks that frontier models could successfully complete doubled” First paper: “approximately every seven months since 2019.” January 2026 post: 131 days after 2023 on TH1.1. November report: 3.61 months (95% CI 2.08-4.94) Depends on the window; the new figure and the 50% success rate are absent
Each task “had also been completed by humans” “For most tasks in our original time horizon suite, we measured this directly” Slightly stronger than the source
“the completion times are mostly estimates” “Human baselines are directly used for only 9% of our estimates” Accurate
“Not one of the AI-designed evaluations was good enough for METR employees to consider it useful for their research.” Figure 8 data: 0 of 45 runs clearly above the bar, 7 borderline above it; METR’s own label is “borderline-useful” Accurate; a later paragraph gives the other half
“just one person rated most of the arguments” “The LMCA dataset’s main expert rater” Accurate
“On this task, too, models fell short of human performance.” Many models are above the 5-minute expert level and all public models are below the 15-minute level Half of the finding
“better than almost every human at software engineering” 96th to 98th percentile in optimization contests; “In the median contest, roughly 4 humans beat the strongest models we tested” Wider than the evidence
“OpenAI reported that its models don’t do as well on the real-world tasks its employees assign them as they do on the easier, more verifiable tasks in METR’s original suite” OpenAI’s page names METR once, in a footnote about Anthropic, and never uses the word “verifiable”; METR’s Figure 7 builds the comparison Misattributed
Preprint: “AI models are not yet able to reproduce human computer science research on their own” “Early evidence from two case studies”; “two unpublished NeurIPS 2026 submissions” Fair, but it rests on two papers
Not mentioned 3.61-month doubling and interval; 1.65 months on the verifiable subset; the METR Capability Index; Lifecycle Cost; the sentence on drastic acceleration Omitted

A point on the graph is a 50% success rate

METR’s first paper defined the metric in its abstract: “50%-task-completion time horizon. This is the time humans typically take to complete tasks that AI models can complete with 50% success rate.” A model with a 40-hour horizon is therefore expected to fail about half of the tasks that take a skilled human 40 hours. The article describes the metric as “the length of the tasks that frontier models could successfully complete,” and a search of its text finds no “50”, no reliability wording and no horizon in hours for any model. For a plot the article itself says was “widely misinterpreted,” the success threshold is the first thing worth stating.

The threshold is also a choice. METR reports 80% time horizons as well, and one appendix chart is titled “When separated, 80% time horizon doubling times on TH2 (verifiable) appear shorter than on TH1.1.”

Which doubling time?

The article says the original plot showed that “every four months, the length of the tasks that frontier models could successfully complete doubled.” METR has published many doubling times, and the one a reader gets depends on the task set and the window. The first paper’s abstract says the horizon has been “doubling approximately every seven months since 2019, though the trend may have accelerated in 2024.” The table collects the figures I found, each with its source.

Source Task set and window Doubling time
First paper and the January 29, 2026 post TH1, 2019 to 2025 196 days (7 months)
January 2026 post TH1, after 2023 165 days
January 2026 post TH1.1, after 2023 131 days
January 2026 post TH1, since 2024 109 days
January 2026 post TH1.1, since 2024 89 days
November report, text “the previous TH1.1 doubling trend” 105 days
November report, Figure 11 TH1.1 refit, 16 frontier models, March 2024 to March 2026 3.07 months (95% interval 1.98 to 4.18)
November report, Figure 10 TH1.1 and TH2 combined, 12 frontier models, May 2024 to September 2026 3.61 months (2.08 to 4.94), or 110 days
November report, Figure 11 TH2 verifiable, 7 frontier models, February to September 2026 1.65 months (0.79 to 3.99)
November report, Figure 11 MirrorCode, 5 frontier models, August 2025 to February 2026 1.79 months (0.19 to 4.94)

Four months is about 122 days, which sits between the 131-day TH1.1 fit for the period after 2023 and the 109-day TH1 fit since 2024. It is neither the seven months in the first paper’s abstract nor the new report’s 3.61 months, which the article never states. METR’s sentence is “time horizons are doubling roughly every 3.61 months (95% CI 2.08-4.94), or 110 days.” Refitting the twelve frontier models in the Figure 10 chart data reproduces it:

12 frontier models, doubling every 3.613 months, R2 0.9857
METR prints 3.613 months, R2 0.9857, 95% interval 2.08 to 4.94

What the interval does to the extrapolation

METR adds a forecast: “naive extrapolation would imply that models reach a 50% time horizon equal to the longest score threshold on our task set (200 hours) in July 2027.” The script reproduces the July date. It then swaps the ends of the 95% interval for the doubling time while holding the fitted line’s September 3 value fixed. This is my what-if, not an interval METR publishes for the date, because it leaves out the uncertainty in the line’s height:

fitted horizon on 2026-09-03: 27.4 hours, 2.87 doublings short of 200 hours
point estimate  3.61 months per doubling -> 2027-07-15
95% low end     2.08 months per doubling -> 2027-03-03
95% high end    4.94 months per doubling -> 2027-11-07

The newest points are weakly pinned down. The three newest frontier-flagged points have 95% intervals whose high ends are 9, 26 and 200 times their low ends, because the suite has few tasks near their level. TH1.1 “includes tasks up to 30 hours,” and the combined suite’s longest score threshold is 200 hours:

GPT-5.6 Sol       11.9 h  95% interval   6.6 to    20.7 h  (  3.2x wide)
Claude Opus 5     19.9 h  95% interval   9.4 to    86.2 h  (  9.1x wide)
Claude Fable 5.1  30.2 h  95% interval  12.0 to   307.7 h  ( 25.7x wide)
GPT-6 Astra       40.3 h  95% interval  13.2 to  2645.0 h  (200.5x wide)
Claude Opus 5.5   39.0 h  95% interval  11.9 to  1252.9 h  (105.2x wide)

None of this makes the trend line wrong. It means that a claim such as “GPT-6 Astra handles 40-hour tasks” would be a much weaker statement than the plot suggests: the point estimate is 40.3 hours, the success rate behind it is 50%, and the 95% interval runs from 13.2 hours to 2,645 hours.

What METR put first

The article tells one of METR’s findings in detail and skips others that METR lists first or alongside it. Beyond the sentence on drastic acceleration, the report introduces a second instrument, the METR Capability Index, an item-response model that METR calls “very similar in spirit to the Epoch Capabilities Index (ECI).” Its heading on the trend result reads: “We do not observe a strong acceleration in capability across public frontier models, as measured by METR Capability Index (MCI) on our tasks. However, we do see uncertain evidence of developer-specific trend changes in some conditions.” The article’s closing question is whether and how quickly automated AI R&D might arrive, and this result bears on it directly.

The first limit METR lists is not judgment but cost: “models perform somewhat worse on tasks that we estimate are costlier to turn into RL training environments, even after controlling for how long these tasks would take humans.” METR quantifies it: “a doubling of lifecycle cost for a task of a given length imposes a difficulty penalty that is very roughly equivalent to halving the time horizon of the agent.” Judgment is the second limit, in Section 3.B.2, and it is the one the article tells.

The verifiable tasks run faster, within wide intervals

The article says that “progress on verifiable tasks doesn’t seem to be slowing down.” METR’s data say more: a doubling time of 1.65 months on the verifiable tasks of the new suite, with the comment that “This faster doubling time could be taken as evidence of an acceleration in model capabilities in 2026.” The caveats are in the same section. The fit covers seven frontier models in seven months, its interval (0.79 to 3.99) overlaps the combined and TH1.1 intervals, METR adds short TH1.1 tasks to TH2 for the figure to constrain the fits, and “we already observe similarly quick doubling times on MirrorCode in the second half of 2025.”

TH1.1             2024-03-04 to 2026-03-05, 16 frontier models: doubling 3.07 months (95% 1.98 to 4.18)
TH2 (Verifiable)  2026-02-05 to 2026-09-03,  7 frontier models: doubling 1.65 months (95% 0.79 to 3.99)
MirrorCode        2025-08-05 to 2026-02-05,  5 frontier models: doubling 1.79 months (95% 0.19 to 4.94)

Judgment work is five of 258 tasks

The article says the updated plot adds tasks that are “more open-ended,” and later that “METR includes few of them in its report.” The second statement is right, and the census makes it concrete. TH1.1 has 222 tasks. TH2 has 36 tasks in 12 families: eight verifiable families with 31 tasks, and four hard-to-verify and judgment families with five tasks, one each for Making METR Tasks, Investigate Wiki Incident and LMCA and two for Judge Run.

TH1.1 222 + TH2 36 = 258 tasks
hard-to-verify and judgment: 5 of 36 TH2 tasks (13.9%), 5 of 258 overall (1.9%)

The report’s time-horizon trend charts cover TH1.1, TH2 (verifiable), MirrorCode, the combined suite and the lower- and higher-cost subsets. I found no trend chart for the five hard-to-verify tasks on their own, so the doubling times above say little about judgment directly. The report’s judgment evidence is mostly the LMCA agreement scores, the ratings of evaluation tasks that models built (Figure 8), and a comparison of model judges with a human scorer (Figure 9, “AI judges are overgenerous on tasks made by stronger models”).

The 45 ratings behind “not one”

In the evaluation-design task, METR asks a model to build a new task for its suite and then rates the result. Figure 8 plots one rating per run, three runs for each of 15 models, on a 0 to 1 scale split into four bands. The top band, “Clearly above the bar,” means “Informative for METR to run in risk assessments or predeployment evaluations, with under 10 hours of further staff work.” The next, “Borderline above the bar,” means “A strong idea whose implementation needs more work, or a strong implementation of a lower-signal idea.”

Released Model Run 1 Run 2 Run 3 Mean
2024-05-13 GPT-4o 0.00 0.01 0.03 0.013
2024-12-05 o1 0.00 0.01 0.00 0.003
2025-08-07 GPT-5 0.05 0.01 0.02 0.027
2025-11-24 Claude Opus 4.5 0.05 0.15 0.15 0.117
2026-02-05 Claude Opus 4.6 0.07 0.10 0.35 0.173
2026-03-05 GPT-5.4 0.03 0.27 0.15 0.150
2026-04-16 Claude Opus 4.7 0.05 0.08 0.05 0.060
2026-04-23 GPT-5.5 0.30 0.37 0.24 0.303
2026-05-28 Claude Opus 4.8 0.35 0.35 0.33 0.343
2026-06-09 Claude Fable 5 0.47 0.58 0.48 0.510
2026-07-09 GPT-5.6 Sol 0.20 0.33 0.35 0.293
2026-07-24 Claude Opus 5 0.30 0.37 0.48 0.383
2026-09-01 Claude Fable 5.1 0.53 0.26 0.51 0.433
2026-09-03 GPT-6 Astra 0.55 0.30 0.40 0.417
2026-09-22 Claude Opus 5.5 0.54 0.52 0.55 0.537

Bold marks the seven ratings of 0.50 or more. The script counts the bands and runs a small permutation test:

Clearly above the bar      0.75 to 1.00:  0 of 45 runs
Borderline above the bar   0.50 to 0.75:  7 of 45 runs
Borderline below the bar   0.25 to 0.50: 17 of 45 runs
Clearly below the bar      0.00 to 0.25: 21 of 45 runs
highest single rating: 0.58
Claude Fable 5   1 of 3 runs at 0.50 or more [0.47, 0.48, 0.58]
Claude Fable 5.1 2 of 3 runs at 0.50 or more [0.26, 0.51, 0.53]
GPT-6 Astra      1 of 3 runs at 0.50 or more [0.3, 0.4, 0.55]
Claude Opus 5.5  3 of 3 runs at 0.50 or more [0.52, 0.54, 0.55]
released by 2025-11-24: 4 models, 12 runs, highest 0.15
Fable 5 mean 0.510, Opus 5.5 mean 0.537, difference 0.027; exact permutation test: 10 of 20 splits are as extreme

The article’s “Not one of the AI-designed evaluations was good enough for METR employees to consider it useful for their research” matches the top band: 0 of 45 runs, with a highest rating of 0.58. A later paragraph of the article reports the other half, that Opus 5.5 reaches “borderline above the bar,” so both facts are in the piece. METR’s own label for the result is “borderline-useful,” and its text says “None of the tasks agents produced were sufficiently useful to us to warrant inclusion in this post.”

Three details sharpen the picture. “Just a year ago, every model was scoring” in the bottom band, as the article says: the four models released by November 24, 2025 have 12 runs with a highest rating of 0.15. But the first run at 0.50 or more belongs to Claude Fable 5 (0.58, released June 9, 2026), not to Opus 5.5. Opus 5.5 is the only model with all three runs in the band, yet its mean of 0.537 is 0.027 above Fable 5’s, and 10 of the 20 ways to split those six runs are as extreme. Fable 5.1’s own three runs span 0.26 to 0.53. On this chart the newest models are not distinguishable from the model that first reached the band three and a half months earlier.

The second detail is who rates. The chart’s subtitle says “We show a METR researcher’s rating of evaluation tasks built by models,” and the text refers to “our human scorer.” The article says “human METR employees.” If the subtitle is literal, the one-rater limit the article correctly raises about the philosophy task applies here too.

The philosophy task

The article says the philosophical-argument task shows that “models fell short of human performance.” METR’s sentence is: “Models have shown steady improvement, and many have agreement above the estimated agreement level of expert humans who spent 5 minutes on the task, but all public models are still below the estimated agreement level of expert humans spending 15 minutes.” The article keeps the second clause and drops the first.

The article is right that “just one person rated most of the arguments.” METR’s appendix takes the task’s difficulty from “The LMCA dataset’s main expert rater’s estimate of how long an average rating took,” so the score and the difficulty estimate both lean on one rater. METR’s footnote adds a point the article compresses: “the authors of LMCA believe the benchmark isn’t very relevant to estimating AI R&D automation risk. We mostly don’t mean to take a very strong position on that question (and we have some internal disagreements on this).” The article says METR employees disagree; METR says the benchmark’s authors doubt its relevance and that METR itself has “some internal disagreements” about it.

The OpenAI sentence

The article says: “And in a Wednesday blog post, OpenAI reported that its models don’t do as well on the real-world tasks its employees assign them as they do on the easier, more verifiable tasks in METR’s original suite.” The Wednesday post is OpenAI’s “Research acceleration: The jagged frontier,” which we audited when it appeared. Its first takeaway is “Agents now complete nearly half of two-to-four-hour research tasks on their own,” its breakdown is by task category (79% for monitoring training runs, 35% for launching them, 25% for writing code for inference and serving stacks), and its conclusion is “Overall, uplift is jagged, and humans remain a core part of the AI research loop.” A script that counts words in the page’s visible text finds this:

# openai_metr_mentions.py
# How often does OpenAI's "Research acceleration: The jagged frontier" page name METR,
# and does its visible text contain the words the article's sentence relies on?
# Needs the requests package (pip install requests).
import html
import re

import requests

URL = "https://openai.com/index/research-acceleration-jagged-frontier/"
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

page = requests.get(URL, headers=HEADERS, timeout=60)
page.raise_for_status()
visible = re.sub(r"(?is)<(script|style|noscript).*?</\1>", " ", page.text)
visible = html.unescape(re.sub(r"(?s)<[^>]+>", " ", visible))
visible = re.sub(r"\s+", " ", visible)
print(f"status {page.status_code}, {len(visible):,} characters of visible text")

for label, pattern, flags in (("METR", r"\bMETR\b", 0), ("verifiable", r"verifiable", re.I),
                              ("time horizon", r"time horizon", re.I), ("original suite", r"original suite", re.I)):
    print(f"{label!r}: {len(re.findall(pattern, visible, flags))} hits")

for match in re.finditer(r"\bMETR\b", visible):
    context = visible[max(0, match.start() - 90):match.start() + 110]
    print("context:", context.encode("ascii", "replace").decode())  # the page has invisible characters
status 200, 26,365 characters of visible text
'METR': 1 hits
'verifiable': 0 hits
'time horizon': 0 hits
'original suite': 0 hits
context:  share of AI-written code internally at Google DeepMind. In collaboration with Anthropic, METR released an acceleration report ? (opens in a new window) estimating the amount that internal Claude mode

METR is named once, in a footnote about Anthropic’s acceleration report. The page’s main text never uses the term “time horizon” (it appears only in the methods drawers, where it means an agent’s estimate of how long a task would take a person) and makes no comparison with METR’s task suites. The comparison is in METR’s Figure 7, whose subtitle says: “For OpenAI’s internal research tasks, we fit one ‘model’ per month and infer a log-normal distribution from its reported CIs.” The text states the result in careful terms: “compared to the lower lifecycle cost subset, model performance on the higher lifecycle cost subset is closer to performance reported by OpenAI on their own internal research usage.” It adds: “We don’t have many task distributions to compare here.”

The numbers in the chart data show what “closer” means:

2026-03    36.7 min  95% interval  23.7 to   58.2
2026-04    42.4 min  95% interval  27.3 to   66.8
2026-05    47.2 min  95% interval  29.7 to   77.3
2026-06    61.8 min  95% interval  34.7 to  112.1
2026-07    81.9 min  95% interval  42.9 to  159.4
2026-08    91.0 min  95% interval  41.5 to  196.1
2026-09   101.4 min  95% interval  51.3 to  195.3
2026-10   131.9 min  95% interval  51.3 to  320.5
OpenAI trend 2026-03-15 to 2026-10-15: 35.5 to 129.7 min, doubling 3.76 months (95% 2.42 to 8.99)
Lower relative lifecycle cost   41.4 h on 2026-09-03: 19.1x the OpenAI figure, doubling 3.77 months (95% 1.77 to 7.04)
Higher relative lifecycle cost  15.8 h on 2026-09-03: 7.3x the OpenAI figure, doubling 3.39 months (95% 0.85 to 4.46)

The costlier subset’s fitted value, 15.8 hours on September 3, is still 7.3 times METR’s reconstruction of OpenAI’s figure for October 15, and the cheaper subset is 19.1 times above it. The intervals are wide, and the dates differ, so the ratios are rough. The units also differ. OpenAI’s durations are estimates (“tasks estimated to require two to four hours of human work”), and the grader prompt in the page’s methods asks an agent to “Estimate how many hours a competent professional human would need to complete the task without AI.” METR’s durations for the new TH2 tasks are mostly its own estimates too (“TH2 being mostly composed of our own time estimates”), while for most tasks in its original suite “we measured this directly.” The article’s sentence is a reasonable gloss of METR’s Figure 7, but it is METR’s reconstruction and METR’s comparison, not something OpenAI reported.

Smaller points

  • Baselines. Appendix A says “For most tasks in our original time horizon suite, we measured this directly,” and a footnote says “The default time estimates are a mixture of human baselines and our own time estimates, with TH2 being mostly composed of our own time estimates.” For the new tasks, METR “defined (score, time) points for 34 of the 36 new tasks using a mix of successful human baselines (3 tasks) and researcher estimates of score given time (31 tasks).” The article is right that the times are mostly estimates. It does not mention that baseliners could use Sonnet 5 for implementation, which “differs from TH1, where baseliners mostly couldn’t use AI,” so the human yardstick changed too.
  • “Better than almost every human at software engineering.” METR’s claim is narrower: “Models usually match or outperform skilled humans on our coding and optimization tasks.” The evidence is a median placement at the 96th to 98th percentile across 15 Al Zimmermann optimization contests (“In the median contest, roughly 4 humans beat the strongest models we tested”) and three baselined tasks. Of its best baseliners, METR guesses “they’d probably do worse than a typical frontier AI researcher in their own research area,” and it notes that “some aspects of our tasks are artificially unfriendly to humans.”
  • The preprint. The Kirgis, Kapoor and colleagues paper was first posted July 29 and revised August 7. Its title is “Can AI agents conduct open-ended AI research? Early evidence from two case studies.” Frontier agents got six days on two unpublished NeurIPS 2026 submissions and “completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions.” That supports the article’s direction, with a sample of two papers.
  • The Anthropic report. The article says METR’s earlier report claimed AI “might be speeding up the pace of AI progress at Anthropic by a factor of two.” Our post on that report found the 1.9x slope ratio reproduces from its charts but runs from 1.1x to 2.9x depending on the fit function.

What the article gets right

Several of the article’s harder claims check out. The single-rater limit on the philosophy task is real and comes from METR’s own appendix. The completion times are mostly estimates, with human baselines behind 9% of METR’s estimates. METR does include few judgment tasks, five of 258. No AI-designed evaluation reached the top band in 45 runs, and a year ago every model sat in the bottom band. Opus 5.5’s three ratings are all in “borderline above the bar.” The closing caution, that experts “agree that we are still far from achieving any definitive answers about whether, and how quickly, automated AI R&D might be coming,” matches METR’s own statement that it remains “significantly uncertain.” The problems are the missing success threshold and doubling time, the OpenAI attribution, and a few sentences that are a notch broader than their sources.

Reproduce the numbers

The first script uses only the Python 3.10+ standard library. It downloads four of the chart files that METR links from its report (Figures 7, 8, 10 and 11, each a JSON file with the rows behind the plot), refits the combined trend, and recomputes the rating counts. Run it with python -I metr_check.py. The task counts in section 6 are typed in from the report’s task table, because the report gives them only in the text. The second script needs pip install requests. Both were run on Python 3.13.14 on November 22, 2026.

# metr_check.py
# Recomputes figures from METR's November 20, 2026 capability update using the
# chart data METR publishes beside each figure. Standard library only (Python 3.10+).
import datetime as dt
import itertools
import json
import math
import statistics
import urllib.request

BASE = "https://metr.org/assets/images/q4-2026-capability-update/"
UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
FIG10 = "cad6b5d98261f80c.json"  # 50% time horizon, TH1.1 and TH2 combined
FIG11 = "8835b7b420aed38c.json"  # trends by task set
FIG8 = "663232d1c7eb4379.json"   # one rating per run for evaluation tasks made by models
FIG7 = "0b82c649db0a5059.json"   # high and low lifecycle-cost subsets, plus METR's reconstruction of OpenAI's figures
MONTH = 30.4375                   # days per month


def chart(name):
    req = urllib.request.Request(BASE + name, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=60) as response:
        return json.load(response)


def table(data, panel, name):
    for t in data["panels"][panel]["tables"]:
        if t["name"] == name:
            return t["rows"]
    raise KeyError(name)


def months(a, b):
    return (dt.date.fromisoformat(b) - dt.date.fromisoformat(a)).days / MONTH


print("=== 1. Refit the combined trend (Figure 10)")
fig10 = chart(FIG10)
models = table(fig10, 0, "models")
trend = table(fig10, 0, "frontier trends")[0]
frontier = [m for m in models if m["frontier"]]
xs = [months(frontier[0]["released"], m["released"]) for m in frontier]
ys = [math.log2(m["horizon_minutes"]) for m in frontier]
fit = statistics.linear_regression(xs, ys)
r2 = statistics.correlation(xs, ys) ** 2
print(f"{len(frontier)} frontier models, doubling every {1 / fit.slope:.3f} months, R2 {r2:.4f}")
print(f"METR prints {trend['doubling_months']:.3f} months, R2 {trend['r_squared']:.4f}, "
      f"95% interval {trend['doubling_months_low_95']:.2f} to {trend['doubling_months_high_95']:.2f}")

print("\n=== 2. Where the fitted line meets the longest task (200 hours = 12,000 minutes)")
needed = math.log2(12000 / trend["horizon_minutes_to"])
print(f"fitted horizon on {trend['to']}: {trend['horizon_minutes_to'] / 60:.1f} hours, {needed:.2f} doublings short of 200 hours")
for label, key in (("point estimate", "doubling_months"), ("95% low end", "doubling_months_low_95"),
                   ("95% high end", "doubling_months_high_95")):
    when = dt.date.fromisoformat(trend["to"]) + dt.timedelta(days=needed * trend[key] * MONTH)
    print(f"{label:<15} {trend[key]:.2f} months per doubling -> {when}")

print("\n=== 3. How wide are the newest points?")
for m in models:
    if m["released"] >= "2026-07-01":
        low, high = m["horizon_minutes_low_95"] / 60, m["horizon_minutes_high_95"] / 60
        print(f"{m['model']:<16} {m['horizon_minutes'] / 60:5.1f} h  95% interval {low:5.1f} to {high:7.1f} h  ({high / low:5.1f}x wide)")

print("\n=== 4. Trends by task set (Figure 11)")
fig11 = chart(FIG11)
all_models = table(fig11, 0, "models")
for row in table(fig11, 0, "frontier trends"):
    n = sum(1 for m in all_models if m["series"] == row["series"] and m["frontier"])
    print(f"{row['series']:<17} {row['from']} to {row['to']}, {n:>2} frontier models: doubling {row['doubling_months']:.2f} months "
          f"(95% {row['doubling_months_low_95']:.2f} to {row['doubling_months_high_95']:.2f})")

print("\n=== 5. Ratings of evaluation tasks made by models (Figure 8)")
fig8 = chart(FIG8)
runs = table(fig8, 0, "runs")
for band in table(fig8, 0, "rating bands"):
    count = sum(1 for r in runs if band["human_rating_from"] <= r["human_rating"] <= band["human_rating_to"]
                and not (r["human_rating"] == band["human_rating_to"] and band["human_rating_to"] < 1.0))
    print(f"{band['band']:<26} {band['human_rating_from']:.2f} to {band['human_rating_to']:.2f}: {count:>2} of {len(runs)} runs")
print("highest single rating:", max(r["human_rating"] for r in runs))
by_model = {}
for r in runs:
    by_model.setdefault(r["model"], []).append(r["human_rating"])
for model, values in by_model.items():
    if max(values) >= 0.5:
        print(f"{model:<16} {sum(v >= 0.5 for v in values)} of {len(values)} runs at 0.50 or more {sorted(values)}")
early = [r for r in runs if r["released"] <= "2025-11-24"]
print("released by 2025-11-24:", len({r['model'] for r in early}), "models,", len(early), "runs, highest", max(r["human_rating"] for r in early))
a, b = by_model["Claude Fable 5"], by_model["Claude Opus 5.5"]
pool, observed = a + b, statistics.mean(b) - statistics.mean(a)
splits = list(itertools.combinations(range(6), 3))
extreme = 0
for chosen in splits:
    g1 = [pool[i] for i in chosen]
    g2 = [pool[i] for i in range(6) if i not in chosen]
    if abs(statistics.mean(g1) - statistics.mean(g2)) >= abs(observed) - 1e-12:
        extreme += 1
print(f"Fable 5 mean {statistics.mean(a):.3f}, Opus 5.5 mean {statistics.mean(b):.3f}, difference {observed:.3f}; "
      f"exact permutation test: {extreme} of {len(splits)} splits are as extreme")

print("\n=== 6. Task census (counts transcribed from the report's task table)")
TH11 = 222
TH2 = {
    "verifiable": {"AZsPCs": 15, "Budget NanoGPT": 2, "Minecraft Bot": 3, "Reverse Conway": 2, "Train a Program": 1,
                   "Earthborne Rangers": 2, "Slay the Spire": 4, "Dominion-like Game": 2},
    "hard-to-verify and judgment": {"Making METR Tasks": 1, "Investigate Wiki Incident": 1, "LMCA": 1, "Judge Run": 2},
}
th2_total = sum(sum(f.values()) for f in TH2.values())
hard = sum(TH2["hard-to-verify and judgment"].values())
print(f"TH1.1 {TH11} + TH2 {th2_total} = {TH11 + th2_total} tasks")
print(f"hard-to-verify and judgment: {hard} of {th2_total} TH2 tasks ({hard / th2_total:.1%}), {hard} of {TH11 + th2_total} overall ({hard / (TH11 + th2_total):.1%})")

print("\n=== 7. METR's reconstruction of OpenAI's internal research horizon (Figure 7)")
fig7 = chart(FIG7)
subsets = {r["series"]: r for r in table(fig7, 0, "frontier trends")}
name = "OpenAI internal research tasks with zero human interventions"
openai_rows = table(fig7, 0, name)
openai = table(fig7, 0, name + " trend")[0]
for r in openai_rows:
    print(f"{r['month']}  {r['horizon_minutes']:6.1f} min  95% interval {r['horizon_minutes_low_95']:5.1f} to {r['horizon_minutes_high_95']:6.1f}")
print(f"OpenAI trend {openai['from']} to {openai['to']}: {openai['horizon_minutes_from']:.1f} to {openai['horizon_minutes_to']:.1f} min, "
      f"doubling {openai['doubling_months']:.2f} months (95% {openai['doubling_months_low_95']:.2f} to {openai['doubling_months_high_95']:.2f})")
for label, row in subsets.items():
    print(f"{label:<30} {row['horizon_minutes_to'] / 60:5.1f} h on {row['to']}: {row['horizon_minutes_to'] / openai['horizon_minutes_to']:.1f}x the OpenAI figure, "
          f"doubling {row['doubling_months']:.2f} months (95% {row['doubling_months_low_95']:.2f} to {row['doubling_months_high_95']:.2f})")
=== 1. Refit the combined trend (Figure 10)
12 frontier models, doubling every 3.613 months, R2 0.9857
METR prints 3.613 months, R2 0.9857, 95% interval 2.08 to 4.94

=== 2. Where the fitted line meets the longest task (200 hours = 12,000 minutes)
fitted horizon on 2026-09-03: 27.4 hours, 2.87 doublings short of 200 hours
point estimate  3.61 months per doubling -> 2027-07-15
95% low end     2.08 months per doubling -> 2027-03-03
95% high end    4.94 months per doubling -> 2027-11-07

=== 3. How wide are the newest points?
GPT-5.6 Sol       11.9 h  95% interval   6.6 to    20.7 h  (  3.2x wide)
Claude Opus 5     19.9 h  95% interval   9.4 to    86.2 h  (  9.1x wide)
Claude Fable 5.1  30.2 h  95% interval  12.0 to   307.7 h  ( 25.7x wide)
GPT-6 Astra       40.3 h  95% interval  13.2 to  2645.0 h  (200.5x wide)
Claude Opus 5.5   39.0 h  95% interval  11.9 to  1252.9 h  (105.2x wide)

=== 4. Trends by task set (Figure 11)
TH1.1             2024-03-04 to 2026-03-05, 16 frontier models: doubling 3.07 months (95% 1.98 to 4.18)
TH2 (Verifiable)  2026-02-05 to 2026-09-03,  7 frontier models: doubling 1.65 months (95% 0.79 to 3.99)
MirrorCode        2025-08-05 to 2026-02-05,  5 frontier models: doubling 1.79 months (95% 0.19 to 4.94)

=== 5. Ratings of evaluation tasks made by models (Figure 8)
Clearly above the bar      0.75 to 1.00:  0 of 45 runs
Borderline above the bar   0.50 to 0.75:  7 of 45 runs
Borderline below the bar   0.25 to 0.50: 17 of 45 runs
Clearly below the bar      0.00 to 0.25: 21 of 45 runs
highest single rating: 0.58
Claude Fable 5   1 of 3 runs at 0.50 or more [0.47, 0.48, 0.58]
Claude Fable 5.1 2 of 3 runs at 0.50 or more [0.26, 0.51, 0.53]
GPT-6 Astra      1 of 3 runs at 0.50 or more [0.3, 0.4, 0.55]
Claude Opus 5.5  3 of 3 runs at 0.50 or more [0.52, 0.54, 0.55]
released by 2025-11-24: 4 models, 12 runs, highest 0.15
Fable 5 mean 0.510, Opus 5.5 mean 0.537, difference 0.027; exact permutation test: 10 of 20 splits are as extreme

=== 6. Task census (counts transcribed from the report's task table)
TH1.1 222 + TH2 36 = 258 tasks
hard-to-verify and judgment: 5 of 36 TH2 tasks (13.9%), 5 of 258 overall (1.9%)

=== 7. METR's reconstruction of OpenAI's internal research horizon (Figure 7)
2026-03    36.7 min  95% interval  23.7 to   58.2
2026-04    42.4 min  95% interval  27.3 to   66.8
2026-05    47.2 min  95% interval  29.7 to   77.3
2026-06    61.8 min  95% interval  34.7 to  112.1
2026-07    81.9 min  95% interval  42.9 to  159.4
2026-08    91.0 min  95% interval  41.5 to  196.1
2026-09   101.4 min  95% interval  51.3 to  195.3
2026-10   131.9 min  95% interval  51.3 to  320.5
OpenAI trend 2026-03-15 to 2026-10-15: 35.5 to 129.7 min, doubling 3.76 months (95% 2.42 to 8.99)
Lower relative lifecycle cost   41.4 h on 2026-09-03: 19.1x the OpenAI figure, doubling 3.77 months (95% 1.77 to 7.04)
Higher relative lifecycle cost  15.8 h on 2026-09-03: 7.3x the OpenAI figure, doubling 3.39 months (95% 0.85 to 4.46)

What this check does not cover

  • I did not re-run any task or re-estimate METR’s models. The bootstrap intervals, the Lifecycle Cost estimates, the MCI fits and the reconstruction of OpenAI’s figures are METR’s, and only the Figure 10 trend is refit here.
  • I read METR’s main text in full and the parts of Appendices A and M cited above, and searched the rest for specific numbers and phrases. The report may be revised after November 20.
  • The article’s interview quotes (Kapoor, Kim, Putz) cannot be checked against a document. A news article compresses a report of roughly 190,000 characters, appendices included, and some of the differences above may come from that compression.
  • The rater count for Figure 8 rests on the chart’s subtitle and the text’s “our human scorer.” METR’s Appendix M also refers to “human manual scorers” in general terms.

Sources

  • MIT Technology Review, “The most misunderstood graph in AI just got a major update” (November 20, 2026)
  • METR, “AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure” (November 20, 2026), with the chart data files linked from the page
  • METR, Time Horizon 1.1 (January 29, 2026)
  • Kwa et al., “Measuring AI Ability to Complete Long Software Tasks” (arXiv 2503.14499)
  • Kirgis et al., “Can AI agents conduct open-ended AI research? Early evidence from two case studies” (arXiv 2607.27191)
  • OpenAI, “Research acceleration: The jagged frontier” (November 18, 2026)
  • Earlier posts: METR’s Anthropic acceleration report, OpenAI’s jagged-frontier report and MIT Technology Review’s agentic-web essay

Tags:

AI BenchmarksAI EvaluationMETRMIT Technology ReviewOpenAI

Share

Brass telescope on a wooden tripod inside a white observatory dome with the roof slit open to the sky
Previous Post

Rackspace Says Leaked Credentials Let an Intruder Act as Legitimate Users in Some Spot Customer Organizations and Reach Their Kubernetes Secrets, a Platform9 Co-Founder Names a Breached Observability System as the Root Cause, and the Status Page Says Only “System Maintenance Operations”

The sun setting over the sea behind a pyramid-shaped stone monument at Cape Soya, the northernmost point of Japan
Next Post

How to Retire an API Endpoint in FastAPI: Send Deprecation and Sunset Headers, Find the Clients Still Calling It, Schedule Brownouts, and Block an Early Removal With oasdiff

No Comment! Be the first one.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Latest
22 Nov
How to Retire an API Endpoint in FastAPI: Send Deprecation and Sunset Headers, Find the Clients Still Calling It, Schedule Brownouts, and Block an Early Removal With oasdiff
22 Nov
MIT Technology Review’s “Most Misunderstood Graph” Update Reports METR’s Judgment Findings Mostly Fairly, but It Never Says Each Point Is a 50% Success Rate or Gives the New 3.6-Month Doubling Time and 2.1 to 4.9 Month Range, and It Credits OpenAI With a Comparison METR Built
Trending
November 22, 2026
How to Retire an API Endpoint in FastAPI: Send Deprecation and Sunset Headers, Find the Clients Still Calling It, Schedule Brownouts, and Block an Early Removal With oasdiff
November 22, 2026
MIT Technology Review’s “Most Misunderstood Graph” Update Reports METR’s Judgment Findings Mostly Fairly, but It Never Says Each Point Is a 50% Success Rate or Gives the New 3.6-Month Doubling Time and 2.1 to 4.9 Month Range, and It Credits OpenAI With a Comparison METR Built
November 22, 2026
Rackspace Says Leaked Credentials Let an Intruder Act as Legitimate Users in Some Spot Customer Organizations and Reach Their Kubernetes Secrets, a Platform9 Co-Founder Names a Breached Observability System as the Root Cause, and the Status Page Says Only “System Maintenance Operations”
November 21, 2026
How to Write YARA Rules With YARA-X to Hunt Malicious Python .pth Startup Files and Measure False Positives Before You Trust Them
November 21, 2026
AWS’s “Found a Zero-Day on Their Own” Post Never Names the Product; the Advisory Behind Its Link Shows a Server That Ships With No Login, a README Now Saying Keep It Off the Internet, and a Fix Merged October 19 That PyPI and Docker Hub Still Do Not Carry
November 21, 2026
Microsoft Tells Customers Unauthorized Parties Read Admin Email Addresses and Tenant Metadata Through an Entra Connect Health Flaw Introduced July 23, but Gives No Tenant Count or Start Date, and Its Own Docs Send Tenant Security Notices to the Same Kinds of Contacts

Related Posts

Blue-lit server racks in a modern data center, illustrating the compute infrastructure behind the AI boom.
Articles

The AI Boom Is Spending Real Money Before Proving Real Returns

June 7, 2026
Technician working with a laptop beside server racks, representing enterprise AI retrieval infrastructure
Articles

Google’s Agentic RAG Push Makes Enterprise AI Less of a One-Shot Guess

June 7, 2026
A person with a laptop and smartphone, representing digital attention and AI-assisted work
Articles

AI Chatbots Are Making Attention a Design Problem

June 7, 2026
A customer-support representative wearing a headset against a dark studio background.
Articles

The Meta AI Support Hack Was a Plain Old Authorization Failure

June 7, 2026
SXZ.io SXZ.io
  • [email protected]

Categories

Articles
Learning Hub
News

All Rights Reserved by SXZ.io ©2026