MIT Technology Review’s “Most Misunderstood Graph” Update Reports METR’s Judgment Findings Mostly Fairly, but It Never Says Each Point Is a 50% Success Rate or Gives the New 3.6-Month Doubling Time and 2.1 to 4.9 Month Range, and It Credits OpenAI With a Comparison METR Built
MIT Technology Review’s update on METR’s time-horizon report is mostly fair on the judgment gap, but it omits the 50% success rate and the 3.61-month doubling time, and it credits OpenAI with a...
On November 20 MIT Technology Review published “The most misunderstood graph in AI just got a major update,” Grace Huckins’ report on METR’s November capability update, which METR itself titles “AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure.” The article’s message is that AI progress on checkable tasks is still exponential while models fall short on tasks that need judgment. Most of that holds up against METR’s report and against the chart data METR publishes beside each figure.
Table Of Content
- The article against its sources
- A point on the graph is a 50% success rate
- Which doubling time?
- What the interval does to the extrapolation
- What METR put first
- The verifiable tasks run faster, within wide intervals
- Judgment work is five of 258 tasks
- The 45 ratings behind “not one”
- The philosophy task
- The OpenAI sentence
- Smaller points
- What the article gets right
- Reproduce the numbers
- What this check does not cover
- Sources
What does not hold up is mostly omission and attribution. The article never says that a point on the graph is a 50% success rate, never gives the report’s new doubling time or its interval, and credits OpenAI with a comparison that METR built from OpenAI’s figures. It also skips the sentence METR puts in its overview: “We think that current public evidence is insufficient to rule out drastic AI R&D acceleration in the next couple of years.” This post checks the article against the primary documents, recomputes the numbers from METR’s published chart data with two short scripts, and marks the places where the article is right.
The article against its sources
The table lists the article’s main factual sentences, what the sources say, and the result of the check. Details and numbers for each row follow in the sections below.
| The article says | The sources say | Check |
|---|---|---|
| “every four months, the length of the tasks that frontier models could successfully complete doubled” | First paper: “approximately every seven months since 2019.” January 2026 post: 131 days after 2023 on TH1.1. November report: 3.61 months (95% CI 2.08-4.94) | Depends on the window; the new figure and the 50% success rate are absent |
| Each task “had also been completed by humans” | “For most tasks in our original time horizon suite, we measured this directly” | Slightly stronger than the source |
| “the completion times are mostly estimates” | “Human baselines are directly used for only 9% of our estimates” | Accurate |
| “Not one of the AI-designed evaluations was good enough for METR employees to consider it useful for their research.” | Figure 8 data: 0 of 45 runs clearly above the bar, 7 borderline above it; METR’s own label is “borderline-useful” | Accurate; a later paragraph gives the other half |
| “just one person rated most of the arguments” | “The LMCA dataset’s main expert rater” | Accurate |
| “On this task, too, models fell short of human performance.” | Many models are above the 5-minute expert level and all public models are below the 15-minute level | Half of the finding |
| “better than almost every human at software engineering” | 96th to 98th percentile in optimization contests; “In the median contest, roughly 4 humans beat the strongest models we tested” | Wider than the evidence |
| “OpenAI reported that its models don’t do as well on the real-world tasks its employees assign them as they do on the easier, more verifiable tasks in METR’s original suite” | OpenAI’s page names METR once, in a footnote about Anthropic, and never uses the word “verifiable”; METR’s Figure 7 builds the comparison | Misattributed |
| Preprint: “AI models are not yet able to reproduce human computer science research on their own” | “Early evidence from two case studies”; “two unpublished NeurIPS 2026 submissions” | Fair, but it rests on two papers |
| Not mentioned | 3.61-month doubling and interval; 1.65 months on the verifiable subset; the METR Capability Index; Lifecycle Cost; the sentence on drastic acceleration | Omitted |
A point on the graph is a 50% success rate
METR’s first paper defined the metric in its abstract: “50%-task-completion time horizon. This is the time humans typically take to complete tasks that AI models can complete with 50% success rate.” A model with a 40-hour horizon is therefore expected to fail about half of the tasks that take a skilled human 40 hours. The article describes the metric as “the length of the tasks that frontier models could successfully complete,” and a search of its text finds no “50”, no reliability wording and no horizon in hours for any model. For a plot the article itself says was “widely misinterpreted,” the success threshold is the first thing worth stating.
The threshold is also a choice. METR reports 80% time horizons as well, and one appendix chart is titled “When separated, 80% time horizon doubling times on TH2 (verifiable) appear shorter than on TH1.1.”
Which doubling time?
The article says the original plot showed that “every four months, the length of the tasks that frontier models could successfully complete doubled.” METR has published many doubling times, and the one a reader gets depends on the task set and the window. The first paper’s abstract says the horizon has been “doubling approximately every seven months since 2019, though the trend may have accelerated in 2024.” The table collects the figures I found, each with its source.
| Source | Task set and window | Doubling time |
|---|---|---|
| First paper and the January 29, 2026 post | TH1, 2019 to 2025 | 196 days (7 months) |
| January 2026 post | TH1, after 2023 | 165 days |
| January 2026 post | TH1.1, after 2023 | 131 days |
| January 2026 post | TH1, since 2024 | 109 days |
| January 2026 post | TH1.1, since 2024 | 89 days |
| November report, text | “the previous TH1.1 doubling trend” | 105 days |
| November report, Figure 11 | TH1.1 refit, 16 frontier models, March 2024 to March 2026 | 3.07 months (95% interval 1.98 to 4.18) |
| November report, Figure 10 | TH1.1 and TH2 combined, 12 frontier models, May 2024 to September 2026 | 3.61 months (2.08 to 4.94), or 110 days |
| November report, Figure 11 | TH2 verifiable, 7 frontier models, February to September 2026 | 1.65 months (0.79 to 3.99) |
| November report, Figure 11 | MirrorCode, 5 frontier models, August 2025 to February 2026 | 1.79 months (0.19 to 4.94) |
Four months is about 122 days, which sits between the 131-day TH1.1 fit for the period after 2023 and the 109-day TH1 fit since 2024. It is neither the seven months in the first paper’s abstract nor the new report’s 3.61 months, which the article never states. METR’s sentence is “time horizons are doubling roughly every 3.61 months (95% CI 2.08-4.94), or 110 days.” Refitting the twelve frontier models in the Figure 10 chart data reproduces it:
12 frontier models, doubling every 3.613 months, R2 0.9857
METR prints 3.613 months, R2 0.9857, 95% interval 2.08 to 4.94
What the interval does to the extrapolation
METR adds a forecast: “naive extrapolation would imply that models reach a 50% time horizon equal to the longest score threshold on our task set (200 hours) in July 2027.” The script reproduces the July date. It then swaps the ends of the 95% interval for the doubling time while holding the fitted line’s September 3 value fixed. This is my what-if, not an interval METR publishes for the date, because it leaves out the uncertainty in the line’s height:
fitted horizon on 2026-09-03: 27.4 hours, 2.87 doublings short of 200 hours
point estimate 3.61 months per doubling -> 2027-07-15
95% low end 2.08 months per doubling -> 2027-03-03
95% high end 4.94 months per doubling -> 2027-11-07
The newest points are weakly pinned down. The three newest frontier-flagged points have 95% intervals whose high ends are 9, 26 and 200 times their low ends, because the suite has few tasks near their level. TH1.1 “includes tasks up to 30 hours,” and the combined suite’s longest score threshold is 200 hours:
GPT-5.6 Sol 11.9 h 95% interval 6.6 to 20.7 h ( 3.2x wide)
Claude Opus 5 19.9 h 95% interval 9.4 to 86.2 h ( 9.1x wide)
Claude Fable 5.1 30.2 h 95% interval 12.0 to 307.7 h ( 25.7x wide)
GPT-6 Astra 40.3 h 95% interval 13.2 to 2645.0 h (200.5x wide)
Claude Opus 5.5 39.0 h 95% interval 11.9 to 1252.9 h (105.2x wide)
None of this makes the trend line wrong. It means that a claim such as “GPT-6 Astra handles 40-hour tasks” would be a much weaker statement than the plot suggests: the point estimate is 40.3 hours, the success rate behind it is 50%, and the 95% interval runs from 13.2 hours to 2,645 hours.
What METR put first
The article tells one of METR’s findings in detail and skips others that METR lists first or alongside it. Beyond the sentence on drastic acceleration, the report introduces a second instrument, the METR Capability Index, an item-response model that METR calls “very similar in spirit to the Epoch Capabilities Index (ECI).” Its heading on the trend result reads: “We do not observe a strong acceleration in capability across public frontier models, as measured by METR Capability Index (MCI) on our tasks. However, we do see uncertain evidence of developer-specific trend changes in some conditions.” The article’s closing question is whether and how quickly automated AI R&D might arrive, and this result bears on it directly.
The first limit METR lists is not judgment but cost: “models perform somewhat worse on tasks that we estimate are costlier to turn into RL training environments, even after controlling for how long these tasks would take humans.” METR quantifies it: “a doubling of lifecycle cost for a task of a given length imposes a difficulty penalty that is very roughly equivalent to halving the time horizon of the agent.” Judgment is the second limit, in Section 3.B.2, and it is the one the article tells.
The verifiable tasks run faster, within wide intervals
The article says that “progress on verifiable tasks doesn’t seem to be slowing down.” METR’s data say more: a doubling time of 1.65 months on the verifiable tasks of the new suite, with the comment that “This faster doubling time could be taken as evidence of an acceleration in model capabilities in 2026.” The caveats are in the same section. The fit covers seven frontier models in seven months, its interval (0.79 to 3.99) overlaps the combined and TH1.1 intervals, METR adds short TH1.1 tasks to TH2 for the figure to constrain the fits, and “we already observe similarly quick doubling times on MirrorCode in the second half of 2025.”
TH1.1 2024-03-04 to 2026-03-05, 16 frontier models: doubling 3.07 months (95% 1.98 to 4.18)
TH2 (Verifiable) 2026-02-05 to 2026-09-03, 7 frontier models: doubling 1.65 months (95% 0.79 to 3.99)
MirrorCode 2025-08-05 to 2026-02-05, 5 frontier models: doubling 1.79 months (95% 0.19 to 4.94)
Judgment work is five of 258 tasks
The article says the updated plot adds tasks that are “more open-ended,” and later that “METR includes few of them in its report.” The second statement is right, and the census makes it concrete. TH1.1 has 222 tasks. TH2 has 36 tasks in 12 families: eight verifiable families with 31 tasks, and four hard-to-verify and judgment families with five tasks, one each for Making METR Tasks, Investigate Wiki Incident and LMCA and two for Judge Run.
TH1.1 222 + TH2 36 = 258 tasks
hard-to-verify and judgment: 5 of 36 TH2 tasks (13.9%), 5 of 258 overall (1.9%)
The report’s time-horizon trend charts cover TH1.1, TH2 (verifiable), MirrorCode, the combined suite and the lower- and higher-cost subsets. I found no trend chart for the five hard-to-verify tasks on their own, so the doubling times above say little about judgment directly. The report’s judgment evidence is mostly the LMCA agreement scores, the ratings of evaluation tasks that models built (Figure 8), and a comparison of model judges with a human scorer (Figure 9, “AI judges are overgenerous on tasks made by stronger models”).
The 45 ratings behind “not one”
In the evaluation-design task, METR asks a model to build a new task for its suite and then rates the result. Figure 8 plots one rating per run, three runs for each of 15 models, on a 0 to 1 scale split into four bands. The top band, “Clearly above the bar,” means “Informative for METR to run in risk assessments or predeployment evaluations, with under 10 hours of further staff work.” The next, “Borderline above the bar,” means “A strong idea whose implementation needs more work, or a strong implementation of a lower-signal idea.”
| Released | Model | Run 1 | Run 2 | Run 3 | Mean |
|---|---|---|---|---|---|
| 2024-05-13 | GPT-4o | 0.00 | 0.01 | 0.03 | 0.013 |
| 2024-12-05 | o1 | 0.00 | 0.01 | 0.00 | 0.003 |
| 2025-08-07 | GPT-5 | 0.05 | 0.01 | 0.02 | 0.027 |
| 2025-11-24 | Claude Opus 4.5 | 0.05 | 0.15 | 0.15 | 0.117 |
| 2026-02-05 | Claude Opus 4.6 | 0.07 | 0.10 | 0.35 | 0.173 |
| 2026-03-05 | GPT-5.4 | 0.03 | 0.27 | 0.15 | 0.150 |
| 2026-04-16 | Claude Opus 4.7 | 0.05 | 0.08 | 0.05 | 0.060 |
| 2026-04-23 | GPT-5.5 | 0.30 | 0.37 | 0.24 | 0.303 |
| 2026-05-28 | Claude Opus 4.8 | 0.35 | 0.35 | 0.33 | 0.343 |
| 2026-06-09 | Claude Fable 5 | 0.47 | 0.58 | 0.48 | 0.510 |
| 2026-07-09 | GPT-5.6 Sol | 0.20 | 0.33 | 0.35 | 0.293 |
| 2026-07-24 | Claude Opus 5 | 0.30 | 0.37 | 0.48 | 0.383 |
| 2026-09-01 | Claude Fable 5.1 | 0.53 | 0.26 | 0.51 | 0.433 |
| 2026-09-03 | GPT-6 Astra | 0.55 | 0.30 | 0.40 | 0.417 |
| 2026-09-22 | Claude Opus 5.5 | 0.54 | 0.52 | 0.55 | 0.537 |
Bold marks the seven ratings of 0.50 or more. The script counts the bands and runs a small permutation test:
Clearly above the bar 0.75 to 1.00: 0 of 45 runs
Borderline above the bar 0.50 to 0.75: 7 of 45 runs
Borderline below the bar 0.25 to 0.50: 17 of 45 runs
Clearly below the bar 0.00 to 0.25: 21 of 45 runs
highest single rating: 0.58
Claude Fable 5 1 of 3 runs at 0.50 or more [0.47, 0.48, 0.58]
Claude Fable 5.1 2 of 3 runs at 0.50 or more [0.26, 0.51, 0.53]
GPT-6 Astra 1 of 3 runs at 0.50 or more [0.3, 0.4, 0.55]
Claude Opus 5.5 3 of 3 runs at 0.50 or more [0.52, 0.54, 0.55]
released by 2025-11-24: 4 models, 12 runs, highest 0.15
Fable 5 mean 0.510, Opus 5.5 mean 0.537, difference 0.027; exact permutation test: 10 of 20 splits are as extreme
The article’s “Not one of the AI-designed evaluations was good enough for METR employees to consider it useful for their research” matches the top band: 0 of 45 runs, with a highest rating of 0.58. A later paragraph of the article reports the other half, that Opus 5.5 reaches “borderline above the bar,” so both facts are in the piece. METR’s own label for the result is “borderline-useful,” and its text says “None of the tasks agents produced were sufficiently useful to us to warrant inclusion in this post.”
Three details sharpen the picture. “Just a year ago, every model was scoring” in the bottom band, as the article says: the four models released by November 24, 2025 have 12 runs with a highest rating of 0.15. But the first run at 0.50 or more belongs to Claude Fable 5 (0.58, released June 9, 2026), not to Opus 5.5. Opus 5.5 is the only model with all three runs in the band, yet its mean of 0.537 is 0.027 above Fable 5’s, and 10 of the 20 ways to split those six runs are as extreme. Fable 5.1’s own three runs span 0.26 to 0.53. On this chart the newest models are not distinguishable from the model that first reached the band three and a half months earlier.
The second detail is who rates. The chart’s subtitle says “We show a METR researcher’s rating of evaluation tasks built by models,” and the text refers to “our human scorer.” The article says “human METR employees.” If the subtitle is literal, the one-rater limit the article correctly raises about the philosophy task applies here too.
The philosophy task
The article says the philosophical-argument task shows that “models fell short of human performance.” METR’s sentence is: “Models have shown steady improvement, and many have agreement above the estimated agreement level of expert humans who spent 5 minutes on the task, but all public models are still below the estimated agreement level of expert humans spending 15 minutes.” The article keeps the second clause and drops the first.
The article is right that “just one person rated most of the arguments.” METR’s appendix takes the task’s difficulty from “The LMCA dataset’s main expert rater’s estimate of how long an average rating took,” so the score and the difficulty estimate both lean on one rater. METR’s footnote adds a point the article compresses: “the authors of LMCA believe the benchmark isn’t very relevant to estimating AI R&D automation risk. We mostly don’t mean to take a very strong position on that question (and we have some internal disagreements on this).” The article says METR employees disagree; METR says the benchmark’s authors doubt its relevance and that METR itself has “some internal disagreements” about it.
The OpenAI sentence
The article says: “And in a Wednesday blog post, OpenAI reported that its models don’t do as well on the real-world tasks its employees assign them as they do on the easier, more verifiable tasks in METR’s original suite.” The Wednesday post is OpenAI’s “Research acceleration: The jagged frontier,” which we audited when it appeared. Its first takeaway is “Agents now complete nearly half of two-to-four-hour research tasks on their own,” its breakdown is by task category (79% for monitoring training runs, 35% for launching them, 25% for writing code for inference and serving stacks), and its conclusion is “Overall, uplift is jagged, and humans remain a core part of the AI research loop.” A script that counts words in the page’s visible text finds this:
# openai_metr_mentions.py
# How often does OpenAI's "Research acceleration: The jagged frontier" page name METR,
# and does its visible text contain the words the article's sentence relies on?
# Needs the requests package (pip install requests).
import html
import re
import requests
URL = "https://openai.com/index/research-acceleration-jagged-frontier/"
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
}
page = requests.get(URL, headers=HEADERS, timeout=60)
page.raise_for_status()
visible = re.sub(r"(?is)<(script|style|noscript).*?</\1>", " ", page.text)
visible = html.unescape(re.sub(r"(?s)<[^>]+>", " ", visible))
visible = re.sub(r"\s+", " ", visible)
print(f"status {page.status_code}, {len(visible):,} characters of visible text")
for label, pattern, flags in (("METR", r"\bMETR\b", 0), ("verifiable", r"verifiable", re.I),
("time horizon", r"time horizon", re.I), ("original suite", r"original suite", re.I)):
print(f"{label!r}: {len(re.findall(pattern, visible, flags))} hits")
for match in re.finditer(r"\bMETR\b", visible):
context = visible[max(0, match.start() - 90):match.start() + 110]
print("context:", context.encode("ascii", "replace").decode()) # the page has invisible characters
status 200, 26,365 characters of visible text
'METR': 1 hits
'verifiable': 0 hits
'time horizon': 0 hits
'original suite': 0 hits
context: share of AI-written code internally at Google DeepMind. In collaboration with Anthropic, METR released an acceleration report ? (opens in a new window) estimating the amount that internal Claude mode
METR is named once, in a footnote about Anthropic’s acceleration report. The page’s main text never uses the term “time horizon” (it appears only in the methods drawers, where it means an agent’s estimate of how long a task would take a person) and makes no comparison with METR’s task suites. The comparison is in METR’s Figure 7, whose subtitle says: “For OpenAI’s internal research tasks, we fit one ‘model’ per month and infer a log-normal distribution from its reported CIs.” The text states the result in careful terms: “compared to the lower lifecycle cost subset, model performance on the higher lifecycle cost subset is closer to performance reported by OpenAI on their own internal research usage.” It adds: “We don’t have many task distributions to compare here.”
The numbers in the chart data show what “closer” means:
2026-03 36.7 min 95% interval 23.7 to 58.2
2026-04 42.4 min 95% interval 27.3 to 66.8
2026-05 47.2 min 95% interval 29.7 to 77.3
2026-06 61.8 min 95% interval 34.7 to 112.1
2026-07 81.9 min 95% interval 42.9 to 159.4
2026-08 91.0 min 95% interval 41.5 to 196.1
2026-09 101.4 min 95% interval 51.3 to 195.3
2026-10 131.9 min 95% interval 51.3 to 320.5
OpenAI trend 2026-03-15 to 2026-10-15: 35.5 to 129.7 min, doubling 3.76 months (95% 2.42 to 8.99)
Lower relative lifecycle cost 41.4 h on 2026-09-03: 19.1x the OpenAI figure, doubling 3.77 months (95% 1.77 to 7.04)
Higher relative lifecycle cost 15.8 h on 2026-09-03: 7.3x the OpenAI figure, doubling 3.39 months (95% 0.85 to 4.46)
The costlier subset’s fitted value, 15.8 hours on September 3, is still 7.3 times METR’s reconstruction of OpenAI’s figure for October 15, and the cheaper subset is 19.1 times above it. The intervals are wide, and the dates differ, so the ratios are rough. The units also differ. OpenAI’s durations are estimates (“tasks estimated to require two to four hours of human work”), and the grader prompt in the page’s methods asks an agent to “Estimate how many hours a competent professional human would need to complete the task without AI.” METR’s durations for the new TH2 tasks are mostly its own estimates too (“TH2 being mostly composed of our own time estimates”), while for most tasks in its original suite “we measured this directly.” The article’s sentence is a reasonable gloss of METR’s Figure 7, but it is METR’s reconstruction and METR’s comparison, not something OpenAI reported.
Smaller points
- Baselines. Appendix A says “For most tasks in our original time horizon suite, we measured this directly,” and a footnote says “The default time estimates are a mixture of human baselines and our own time estimates, with TH2 being mostly composed of our own time estimates.” For the new tasks, METR “defined (score, time) points for 34 of the 36 new tasks using a mix of successful human baselines (3 tasks) and researcher estimates of score given time (31 tasks).” The article is right that the times are mostly estimates. It does not mention that baseliners could use Sonnet 5 for implementation, which “differs from TH1, where baseliners mostly couldn’t use AI,” so the human yardstick changed too.
- “Better than almost every human at software engineering.” METR’s claim is narrower: “Models usually match or outperform skilled humans on our coding and optimization tasks.” The evidence is a median placement at the 96th to 98th percentile across 15 Al Zimmermann optimization contests (“In the median contest, roughly 4 humans beat the strongest models we tested”) and three baselined tasks. Of its best baseliners, METR guesses “they’d probably do worse than a typical frontier AI researcher in their own research area,” and it notes that “some aspects of our tasks are artificially unfriendly to humans.”
- The preprint. The Kirgis, Kapoor and colleagues paper was first posted July 29 and revised August 7. Its title is “Can AI agents conduct open-ended AI research? Early evidence from two case studies.” Frontier agents got six days on two unpublished NeurIPS 2026 submissions and “completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions.” That supports the article’s direction, with a sample of two papers.
- The Anthropic report. The article says METR’s earlier report claimed AI “might be speeding up the pace of AI progress at Anthropic by a factor of two.” Our post on that report found the 1.9x slope ratio reproduces from its charts but runs from 1.1x to 2.9x depending on the fit function.
What the article gets right
Several of the article’s harder claims check out. The single-rater limit on the philosophy task is real and comes from METR’s own appendix. The completion times are mostly estimates, with human baselines behind 9% of METR’s estimates. METR does include few judgment tasks, five of 258. No AI-designed evaluation reached the top band in 45 runs, and a year ago every model sat in the bottom band. Opus 5.5’s three ratings are all in “borderline above the bar.” The closing caution, that experts “agree that we are still far from achieving any definitive answers about whether, and how quickly, automated AI R&D might be coming,” matches METR’s own statement that it remains “significantly uncertain.” The problems are the missing success threshold and doubling time, the OpenAI attribution, and a few sentences that are a notch broader than their sources.
Reproduce the numbers
The first script uses only the Python 3.10+ standard library. It downloads four of the chart files that METR links from its report (Figures 7, 8, 10 and 11, each a JSON file with the rows behind the plot), refits the combined trend, and recomputes the rating counts. Run it with python -I metr_check.py. The task counts in section 6 are typed in from the report’s task table, because the report gives them only in the text. The second script needs pip install requests. Both were run on Python 3.13.14 on November 22, 2026.
# metr_check.py
# Recomputes figures from METR's November 20, 2026 capability update using the
# chart data METR publishes beside each figure. Standard library only (Python 3.10+).
import datetime as dt
import itertools
import json
import math
import statistics
import urllib.request
BASE = "https://metr.org/assets/images/q4-2026-capability-update/"
UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
FIG10 = "cad6b5d98261f80c.json" # 50% time horizon, TH1.1 and TH2 combined
FIG11 = "8835b7b420aed38c.json" # trends by task set
FIG8 = "663232d1c7eb4379.json" # one rating per run for evaluation tasks made by models
FIG7 = "0b82c649db0a5059.json" # high and low lifecycle-cost subsets, plus METR's reconstruction of OpenAI's figures
MONTH = 30.4375 # days per month
def chart(name):
req = urllib.request.Request(BASE + name, headers={"User-Agent": UA})
with urllib.request.urlopen(req, timeout=60) as response:
return json.load(response)
def table(data, panel, name):
for t in data["panels"][panel]["tables"]:
if t["name"] == name:
return t["rows"]
raise KeyError(name)
def months(a, b):
return (dt.date.fromisoformat(b) - dt.date.fromisoformat(a)).days / MONTH
print("=== 1. Refit the combined trend (Figure 10)")
fig10 = chart(FIG10)
models = table(fig10, 0, "models")
trend = table(fig10, 0, "frontier trends")[0]
frontier = [m for m in models if m["frontier"]]
xs = [months(frontier[0]["released"], m["released"]) for m in frontier]
ys = [math.log2(m["horizon_minutes"]) for m in frontier]
fit = statistics.linear_regression(xs, ys)
r2 = statistics.correlation(xs, ys) ** 2
print(f"{len(frontier)} frontier models, doubling every {1 / fit.slope:.3f} months, R2 {r2:.4f}")
print(f"METR prints {trend['doubling_months']:.3f} months, R2 {trend['r_squared']:.4f}, "
f"95% interval {trend['doubling_months_low_95']:.2f} to {trend['doubling_months_high_95']:.2f}")
print("\n=== 2. Where the fitted line meets the longest task (200 hours = 12,000 minutes)")
needed = math.log2(12000 / trend["horizon_minutes_to"])
print(f"fitted horizon on {trend['to']}: {trend['horizon_minutes_to'] / 60:.1f} hours, {needed:.2f} doublings short of 200 hours")
for label, key in (("point estimate", "doubling_months"), ("95% low end", "doubling_months_low_95"),
("95% high end", "doubling_months_high_95")):
when = dt.date.fromisoformat(trend["to"]) + dt.timedelta(days=needed * trend[key] * MONTH)
print(f"{label:<15} {trend[key]:.2f} months per doubling -> {when}")
print("\n=== 3. How wide are the newest points?")
for m in models:
if m["released"] >= "2026-07-01":
low, high = m["horizon_minutes_low_95"] / 60, m["horizon_minutes_high_95"] / 60
print(f"{m['model']:<16} {m['horizon_minutes'] / 60:5.1f} h 95% interval {low:5.1f} to {high:7.1f} h ({high / low:5.1f}x wide)")
print("\n=== 4. Trends by task set (Figure 11)")
fig11 = chart(FIG11)
all_models = table(fig11, 0, "models")
for row in table(fig11, 0, "frontier trends"):
n = sum(1 for m in all_models if m["series"] == row["series"] and m["frontier"])
print(f"{row['series']:<17} {row['from']} to {row['to']}, {n:>2} frontier models: doubling {row['doubling_months']:.2f} months "
f"(95% {row['doubling_months_low_95']:.2f} to {row['doubling_months_high_95']:.2f})")
print("\n=== 5. Ratings of evaluation tasks made by models (Figure 8)")
fig8 = chart(FIG8)
runs = table(fig8, 0, "runs")
for band in table(fig8, 0, "rating bands"):
count = sum(1 for r in runs if band["human_rating_from"] <= r["human_rating"] <= band["human_rating_to"]
and not (r["human_rating"] == band["human_rating_to"] and band["human_rating_to"] < 1.0))
print(f"{band['band']:<26} {band['human_rating_from']:.2f} to {band['human_rating_to']:.2f}: {count:>2} of {len(runs)} runs")
print("highest single rating:", max(r["human_rating"] for r in runs))
by_model = {}
for r in runs:
by_model.setdefault(r["model"], []).append(r["human_rating"])
for model, values in by_model.items():
if max(values) >= 0.5:
print(f"{model:<16} {sum(v >= 0.5 for v in values)} of {len(values)} runs at 0.50 or more {sorted(values)}")
early = [r for r in runs if r["released"] <= "2025-11-24"]
print("released by 2025-11-24:", len({r['model'] for r in early}), "models,", len(early), "runs, highest", max(r["human_rating"] for r in early))
a, b = by_model["Claude Fable 5"], by_model["Claude Opus 5.5"]
pool, observed = a + b, statistics.mean(b) - statistics.mean(a)
splits = list(itertools.combinations(range(6), 3))
extreme = 0
for chosen in splits:
g1 = [pool[i] for i in chosen]
g2 = [pool[i] for i in range(6) if i not in chosen]
if abs(statistics.mean(g1) - statistics.mean(g2)) >= abs(observed) - 1e-12:
extreme += 1
print(f"Fable 5 mean {statistics.mean(a):.3f}, Opus 5.5 mean {statistics.mean(b):.3f}, difference {observed:.3f}; "
f"exact permutation test: {extreme} of {len(splits)} splits are as extreme")
print("\n=== 6. Task census (counts transcribed from the report's task table)")
TH11 = 222
TH2 = {
"verifiable": {"AZsPCs": 15, "Budget NanoGPT": 2, "Minecraft Bot": 3, "Reverse Conway": 2, "Train a Program": 1,
"Earthborne Rangers": 2, "Slay the Spire": 4, "Dominion-like Game": 2},
"hard-to-verify and judgment": {"Making METR Tasks": 1, "Investigate Wiki Incident": 1, "LMCA": 1, "Judge Run": 2},
}
th2_total = sum(sum(f.values()) for f in TH2.values())
hard = sum(TH2["hard-to-verify and judgment"].values())
print(f"TH1.1 {TH11} + TH2 {th2_total} = {TH11 + th2_total} tasks")
print(f"hard-to-verify and judgment: {hard} of {th2_total} TH2 tasks ({hard / th2_total:.1%}), {hard} of {TH11 + th2_total} overall ({hard / (TH11 + th2_total):.1%})")
print("\n=== 7. METR's reconstruction of OpenAI's internal research horizon (Figure 7)")
fig7 = chart(FIG7)
subsets = {r["series"]: r for r in table(fig7, 0, "frontier trends")}
name = "OpenAI internal research tasks with zero human interventions"
openai_rows = table(fig7, 0, name)
openai = table(fig7, 0, name + " trend")[0]
for r in openai_rows:
print(f"{r['month']} {r['horizon_minutes']:6.1f} min 95% interval {r['horizon_minutes_low_95']:5.1f} to {r['horizon_minutes_high_95']:6.1f}")
print(f"OpenAI trend {openai['from']} to {openai['to']}: {openai['horizon_minutes_from']:.1f} to {openai['horizon_minutes_to']:.1f} min, "
f"doubling {openai['doubling_months']:.2f} months (95% {openai['doubling_months_low_95']:.2f} to {openai['doubling_months_high_95']:.2f})")
for label, row in subsets.items():
print(f"{label:<30} {row['horizon_minutes_to'] / 60:5.1f} h on {row['to']}: {row['horizon_minutes_to'] / openai['horizon_minutes_to']:.1f}x the OpenAI figure, "
f"doubling {row['doubling_months']:.2f} months (95% {row['doubling_months_low_95']:.2f} to {row['doubling_months_high_95']:.2f})")
=== 1. Refit the combined trend (Figure 10)
12 frontier models, doubling every 3.613 months, R2 0.9857
METR prints 3.613 months, R2 0.9857, 95% interval 2.08 to 4.94
=== 2. Where the fitted line meets the longest task (200 hours = 12,000 minutes)
fitted horizon on 2026-09-03: 27.4 hours, 2.87 doublings short of 200 hours
point estimate 3.61 months per doubling -> 2027-07-15
95% low end 2.08 months per doubling -> 2027-03-03
95% high end 4.94 months per doubling -> 2027-11-07
=== 3. How wide are the newest points?
GPT-5.6 Sol 11.9 h 95% interval 6.6 to 20.7 h ( 3.2x wide)
Claude Opus 5 19.9 h 95% interval 9.4 to 86.2 h ( 9.1x wide)
Claude Fable 5.1 30.2 h 95% interval 12.0 to 307.7 h ( 25.7x wide)
GPT-6 Astra 40.3 h 95% interval 13.2 to 2645.0 h (200.5x wide)
Claude Opus 5.5 39.0 h 95% interval 11.9 to 1252.9 h (105.2x wide)
=== 4. Trends by task set (Figure 11)
TH1.1 2024-03-04 to 2026-03-05, 16 frontier models: doubling 3.07 months (95% 1.98 to 4.18)
TH2 (Verifiable) 2026-02-05 to 2026-09-03, 7 frontier models: doubling 1.65 months (95% 0.79 to 3.99)
MirrorCode 2025-08-05 to 2026-02-05, 5 frontier models: doubling 1.79 months (95% 0.19 to 4.94)
=== 5. Ratings of evaluation tasks made by models (Figure 8)
Clearly above the bar 0.75 to 1.00: 0 of 45 runs
Borderline above the bar 0.50 to 0.75: 7 of 45 runs
Borderline below the bar 0.25 to 0.50: 17 of 45 runs
Clearly below the bar 0.00 to 0.25: 21 of 45 runs
highest single rating: 0.58
Claude Fable 5 1 of 3 runs at 0.50 or more [0.47, 0.48, 0.58]
Claude Fable 5.1 2 of 3 runs at 0.50 or more [0.26, 0.51, 0.53]
GPT-6 Astra 1 of 3 runs at 0.50 or more [0.3, 0.4, 0.55]
Claude Opus 5.5 3 of 3 runs at 0.50 or more [0.52, 0.54, 0.55]
released by 2025-11-24: 4 models, 12 runs, highest 0.15
Fable 5 mean 0.510, Opus 5.5 mean 0.537, difference 0.027; exact permutation test: 10 of 20 splits are as extreme
=== 6. Task census (counts transcribed from the report's task table)
TH1.1 222 + TH2 36 = 258 tasks
hard-to-verify and judgment: 5 of 36 TH2 tasks (13.9%), 5 of 258 overall (1.9%)
=== 7. METR's reconstruction of OpenAI's internal research horizon (Figure 7)
2026-03 36.7 min 95% interval 23.7 to 58.2
2026-04 42.4 min 95% interval 27.3 to 66.8
2026-05 47.2 min 95% interval 29.7 to 77.3
2026-06 61.8 min 95% interval 34.7 to 112.1
2026-07 81.9 min 95% interval 42.9 to 159.4
2026-08 91.0 min 95% interval 41.5 to 196.1
2026-09 101.4 min 95% interval 51.3 to 195.3
2026-10 131.9 min 95% interval 51.3 to 320.5
OpenAI trend 2026-03-15 to 2026-10-15: 35.5 to 129.7 min, doubling 3.76 months (95% 2.42 to 8.99)
Lower relative lifecycle cost 41.4 h on 2026-09-03: 19.1x the OpenAI figure, doubling 3.77 months (95% 1.77 to 7.04)
Higher relative lifecycle cost 15.8 h on 2026-09-03: 7.3x the OpenAI figure, doubling 3.39 months (95% 0.85 to 4.46)
What this check does not cover
- I did not re-run any task or re-estimate METR’s models. The bootstrap intervals, the Lifecycle Cost estimates, the MCI fits and the reconstruction of OpenAI’s figures are METR’s, and only the Figure 10 trend is refit here.
- I read METR’s main text in full and the parts of Appendices A and M cited above, and searched the rest for specific numbers and phrases. The report may be revised after November 20.
- The article’s interview quotes (Kapoor, Kim, Putz) cannot be checked against a document. A news article compresses a report of roughly 190,000 characters, appendices included, and some of the differences above may come from that compression.
- The rater count for Figure 8 rests on the chart’s subtitle and the text’s “our human scorer.” METR’s Appendix M also refers to “human manual scorers” in general terms.
Sources
- MIT Technology Review, “The most misunderstood graph in AI just got a major update” (November 20, 2026)
- METR, “AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure” (November 20, 2026), with the chart data files linked from the page
- METR, Time Horizon 1.1 (January 29, 2026)
- Kwa et al., “Measuring AI Ability to Complete Long Software Tasks” (arXiv 2503.14499)
- Kirgis et al., “Can AI agents conduct open-ended AI research? Early evidence from two case studies” (arXiv 2607.27191)
- OpenAI, “Research acceleration: The jagged frontier” (November 18, 2026)
- Earlier posts: METR’s Anthropic acceleration report, OpenAI’s jagged-frontier report and MIT Technology Review’s agentic-web essay








No Comment! Be the first one.