OpenAI’s Performance Factory Turns Regression Response Into a Task Queue for Agents, but Its Numbers Stop at 600 Root-Caused Cases in 1,700 Investigations and a 70% Grade on 100 Suggestions, and in a Toy World Each of Its Reviewer’s Two Counterfactual Questions Can Return a Confident Wrong Answer
OpenAI’s Performance Factory post says Codex agents find and fix ChatGPT regressions, but its outcome numbers are 600-plus root-caused cases among 1,700-plus investigations and a 70% grade on 100...
OpenAI’s November 19 engineering post, How OpenAI’s Performance Factory keeps ChatGPT fast by Okay Zed and Cyrus Forbes, describes a system in which Codex agents pick up a ChatGPT performance alert, investigate it, propose a fix for an engineer to review, and later check whether the metric came back. Its subhead reads “Our Codex-powered system automates finding and fixing performance issues.” The design is described in enough detail to copy: task states kept in configuration files, a dispatch loop, cloud Codex turns and event subscriptions.
Table Of Content
- The short version
- What the post describes
- Every number the post prints
- The throughput figures agree with each other
- Investigations and cases are different units
- The 70 percent grade has a wide interval and no date
- Five of the seven quantities that would show find and fix have no number
- The headline against the body
- The reviewer’s two questions, tested in a toy request world
- The world
- The checks
- The runner
- The mean rose in every case, and the split says why
- How often each check blames the deploy
- A placebo class flags the broken comparison but cannot repair it
- What the toy world does not show
- What the post leaves out about the agent’s own exposure
- What you can build from public Codex pieces
- What to measure if you copy the pattern
- The bottom line
- Sources and method
The evidence for how well it works is thinner than the design. Only four sentences in the post body carry a number, and none of them is about a fix. This article recomputes those numbers, shows what they can and cannot support, and then tests the two counterfactual questions the post says its reviewer asks, in a synthetic request world where the true cause of every alert is known. I did not run or access any OpenAI system. The toy world’s settings are my own assumptions, so its percentages show how the checks behave and say nothing about OpenAI’s data.
The short version
- The post’s outcome numbers are more than 1,700 tracked investigations, over 600 root-caused cases, 17 investigations a day and a 70% grade on 100 reviewed suggestions. They are consistent with each other. No sentence counts fixes proposed, merged or verified, and none gives a time from alert to diagnosis.
- The 70% grade rests on 100 reviews, which gives an exact 95% interval of 60.0% to 78.8%, and the post does not say whether the reviews came before or after the adversarial reviewer was added.
- The lead says the first workflow “has automated the entire process of fixing regressions.” The body says an engineer decides whether to merge, and that the merge may be the bottleneck.
- In my toy world, a reviewer question about unexposed requests blamed a harmless deploy in 100% of incidents when the rollout started in a slow region, and a rollback-recovery question blamed a harmless deploy in 95% of incidents when traffic happened to revert at the same moment. Matching on the hidden variable, and splitting the change into within-class and mix parts, fixed both.
What the post describes
The post walks through six stages in how its team’s use of Codex evolved: manual investigation, Codex as a partner, a reusable skill called perf-forensics, an engineer acting as “The human proxy” between alerts and Codex, automated investigations that post to Slack, and finally the factory. The first workflow, the Regression Workflow, opens a task when a configured performance alert fires. The task moves through investigation, a proposed fix, an engineer’s review and deployment, and an agent’s check that performance returned to its baseline. If there is no improvement, the agent sends the regression back for investigation, and if recovery is partial, it resolves the original task and opens a new one for the remainder.
Four components do the work. The post lists “a task tracker with configurable workflows, a dispatch loop that assigns ready work to agents, an API that runs Codex in the cloud, and subscriptions that wake agents when something changes”. Each task “keeps its state and findings, so work can pass between agents and engineers and resume after a code review or deployment.” The Codex turn “runs on an environment inside our network, with access to our code repositories and the observability tools our engineers use, such as metrics, logs, traces, CPU profiles, and memory snapshots.” The agent “runs under a service identity, so we can scope its permissions and see which actions came from our performance agents.”
Two details of the root-cause step matter later in this article. First, the agent “compares groups of requests: did similar requests actually get slower, or were we serving more requests that were already slow? Those findings point to different causes.” Second, a separate reviewer tries to disprove the first diagnosis. The post says “Early on, agents sometimes stopped at the first plausible explanation they found, and some of those diagnoses turned out to be wrong.” So it added “an adversarial review: a separate Codex session that tries to disprove the proposed diagnosis.” The reviewer “rechecks the assumptions and tests counterfactuals. If a rollout caused the slowdown, did similar requests that weren’t exposed stay healthy? If the suspected change was rolled back, did the affected metric recover?”
The post gives one example. An agent traced failing ChatGPT conversations to errors in image downloads and marked the cause as found, but “The Codex reviewer noticed that the internal service responsible for locating files had already recovered while image requests were still failing.” The reviewer reopened the investigation. That is a third kind of check, a timeline comparison between the suspected cause and the symptom, and the lab below does not cover it.
The post also describes how it tests the investigator, and the controls are sound. It backtests against past production incidents, and “Each case starts with a fresh Codex investigator and the original signal, stripped of details that might give away the answer.” The agent “cannot use saved memories (notes from earlier investigations), read the incident writeup, or see results from previous runs.” The team also inspects each transcript afterward for spoilers. It scores “when the problem began, how large it was, and which requests were affected”, and scores the cause separately: “An agent can get the timing right and still blame the wrong deployment, so we score those separately.” The post prints no score from these backtests.
Every number the post prints
The script below fetches the post, lists each sentence that carries a number (list numbering and code-listing line numbers are skipped, and em dashes in the source are shown as hyphens), then counts a few words in the body. It needs the requests package and sends a browser User-Agent, because earlier checks of openai.com pages with other scripted clients returned HTTP 403.
# post_census.py
"""Fetch the Performance Factory post, list every sentence that carries a number, then count a few words."""
import html
import re
import sys
import requests
sys.stdout.reconfigure(encoding="utf-8") # the post has curly quotes; keep Windows consoles from choking
URL = "https://openai.com/index/how-openai-performance-factory-keeps-chatgpt-fast"
UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
raw = requests.get(URL, headers={"User-Agent": UA, "Accept-Language": "en-US,en;q=0.9"}, timeout=40).content.decode("utf-8")
text = re.sub(r"(?is)<(script|style|noscript|svg).*?</\1>", " ", raw)
text = re.sub(r"(?is)</(p|li|h\d|div)>", "\n", text)
text = html.unescape(re.sub(r"(?s)<[^>]+>", " ", text))
text = re.sub(r"[ \t ]+", " ", text)
start = text.index("The staggering pace of agentic development")
body = text[start:text.index("Authors", start)].replace(chr(0x2014), " - ") # em dashes shown as hyphens
print(f"body: {len(body.split()):,} words")
print("\n== Sentences that carry a number (list numbering such as '1.' ignored)")
for sentence in re.split(r"(?<=[.!?])\s+|\n", body):
plain = re.sub(r"^\s*\d\.\s+", "", sentence)
numbers = re.findall(r"(?<![\w.])\d[\d,]*(?:\.\d+)?\+?%?(?!\))", plain)
if numbers and len(plain.split()) >= 5:
print(f"{' '.join(numbers):14} | {' '.join(plain.split())[:150]}")
print("\n== Word counts in the body")
for word in ("inject", "untrusted", "secret", "credential", "permission", "service identity", "sandbox",
"median", "merge", "minutes", "hours", "baseline", "unprecedented", "human in the loop"):
print(f"{word:18} {len(re.findall(word, body, flags=re.I))}")
The output on November 19:
body: 3,303 words
== Sentences that carry a number (list numbering such as '1.' ignored)
1,700 600 | In just over three months, the Regression Workflow tracked more than 1,700 investigations and root caused over 600 cases.
17 | That’s 17 automated investigations every day.
30 | Over time, we accumulated more than 30 performance tools.
100 70% 30% | In a manual review of 100 root cause suggestions, we graded 70% as accurate, with 30% needing further refinement or not fully establishing causality.
== Word counts in the body
inject 0
untrusted 0
secret 0
credential 0
permission 2
service identity 1
sandbox 0
median 0
merge 4
minutes 0
hours 1
baseline 2
unprecedented 1
human in the loop 1
The body is 3,303 words, and four sentences carry numbers: the 1,700 and the 600, the 17 a day, the “more than 30 performance tools,” and the 100 reviews graded 70% and 30%. None concerns a fix proposed, a fix merged, a time to diagnosis, a cost or a count of engineer hours. The word “unprecedented” appears once, in the lead’s “find and fix regressions at an unprecedented speed,” and “baseline” appears twice, both times for the metric the agent checks after a deployment. “Inject”, “untrusted”, “secret” and “credential” appear zero times.
The throughput figures agree with each other
The next script recomputes the post’s arithmetic. The sentences are “In just over three months, the Regression Workflow tracked more than 1,700 investigations and root caused over 600 cases. That’s 17 automated investigations every day.”
# factory_numbers.py
"""Recompute the figures OpenAI's Performance Factory post prints, and what they can and cannot imply."""
from math import comb, sqrt
TRACKED = 1700 # "tracked more than 1,700 investigations"
ROOT_CAUSED = 600 # "root caused over 600 cases"
PER_DAY = 17 # "That's 17 automated investigations every day"
GRADED = 100 # "a manual review of 100 root cause suggestions"
ACCURATE = 70 # "we graded 70% as accurate"
def wilson(k, n, z=1.96):
"""Wilson score interval for k successes in n trials."""
p = k / n
scale = 1 + z * z / n
centre = (p + z * z / (2 * n)) / scale
half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / scale
return centre - half, centre + half
def binom_cdf(k, n, p):
"""P(X <= k) for X ~ Binomial(n, p)."""
return sum(comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(k + 1))
def clopper_pearson(k, n, alpha=0.05):
"""Exact two-sided interval, found by bisection on the binomial tails."""
lo, hi = 0.0, 1.0
for _ in range(60): # lower end: P(X >= k) = alpha / 2
mid = (lo + hi) / 2
lo, hi = (mid, hi) if 1 - binom_cdf(k - 1, n, mid) < alpha / 2 else (lo, mid)
lower = lo
lo, hi = 0.0, 1.0
for _ in range(60): # upper end: P(X <= k) = alpha / 2
mid = (lo + hi) / 2
lo, hi = (mid, hi) if binom_cdf(k, n, mid) > alpha / 2 else (lo, mid)
return lower, lo
print("== Arithmetic inside the post")
days = TRACKED / PER_DAY
print(f"days implied by {TRACKED:,} investigations at {PER_DAY} a day: {days:.1f} days = {days / 30.44:.2f} months")
print(f"17 a day for exactly 3 months (91.3 days): {PER_DAY * 91.3:,.0f} investigations")
print(f"{TRACKED:,} over a 92-day quarter: {TRACKED / 92:.1f} a day")
print(f"root-caused share: {ROOT_CAUSED / TRACKED:.1%} ({ROOT_CAUSED} of {TRACKED:,}), {ROOT_CAUSED / days:.1f} cases a day")
print(f"if both figures are floors rounded down to the hundred: "
f"{ROOT_CAUSED / 1799:.1%} to {699 / TRACKED:.1%}")
print("\n== The 70 percent grade")
k, n = ACCURATE, GRADED
w_lo, w_hi = wilson(k, n)
c_lo, c_hi = clopper_pearson(k, n)
print(f"{k} of {n} graded accurate: Wilson 95% {w_lo:.1%} to {w_hi:.1%}; exact 95% {c_lo:.1%} to {c_hi:.1%}")
need = 1.96**2 * 0.7 * 0.3 / 0.05**2
print(f"reviews needed to pin 70% to plus or minus 5 points: {need:.0f}")
print("if the grade applied to the 600 root-caused cases:")
for label, share in (("low", c_lo), ("point", k / n), ("high", c_hi)):
good = ROOT_CAUSED * share
print(f" {label:5} {share:.1%} -> {good:.0f} accurate root causes = {good / TRACKED:.1%} of {TRACKED:,} tracked")
print("\n== What the stages named in the post have counts for")
for stage, count in (
("tracked investigations", f"{TRACKED:,}+"),
("root caused", f"{ROOT_CAUSED}+"),
("fix proposed", "none printed"),
("fix merged by an engineer", "none printed"),
("recovery verified", "none printed"),
("time from alert to diagnosis", "none printed"),
("accuracy after adversarial review", "none printed"),
):
print(f" {stage:34} {count}")
== Arithmetic inside the post
days implied by 1,700 investigations at 17 a day: 100.0 days = 3.29 months
17 a day for exactly 3 months (91.3 days): 1,552 investigations
1,700 over a 92-day quarter: 18.5 a day
root-caused share: 35.3% (600 of 1,700), 6.0 cases a day
if both figures are floors rounded down to the hundred: 33.4% to 41.1%
== The 70 percent grade
70 of 100 graded accurate: Wilson 95% 60.4% to 78.1%; exact 95% 60.0% to 78.8%
reviews needed to pin 70% to plus or minus 5 points: 323
if the grade applied to the 600 root-caused cases:
low 60.0% -> 360 accurate root causes = 21.2% of 1,700 tracked
point 70.0% -> 420 accurate root causes = 24.7% of 1,700 tracked
high 78.8% -> 473 accurate root causes = 27.8% of 1,700 tracked
== What the stages named in the post have counts for
tracked investigations 1,700+
root caused 600+
fix proposed none printed
fix merged by an engineer none printed
recovery verified none printed
time from alert to diagnosis none printed
accuracy after adversarial review none printed
1,700 investigations at 17 a day is exactly 100 days, or 3.29 months, which fits “just over three months.” If the window had been exactly three months (91.3 days), 17 a day would give 1,552 investigations, so the window must have been about 100 days. The root-caused share is 35.3% (600 of 1,700), or six cases a day. Because the post gives floors rather than counts, the share could be anywhere from 33.4% to 41.1% if both figures were rounded down to the hundred.
Investigations and cases are different units
The sentence counts “investigations” in its first half and “cases” in its second. The post also says “One regression can also appear as several incidents across alerts, monitors, and channels.” It adds that the workflow now “links related incidents.” A reader cannot tell whether the 600 is counted after that linking and the 1,700 before it. If it is, the 35.3% divides one unit by another.
The 70 percent grade has a wide interval and no date
The post says “In a manual review of 100 root cause suggestions, we graded 70% as accurate, with 30% needing further refinement or not fully establishing causality.” With 70 of 100, the Wilson 95% interval is 60.4% to 78.1% and the exact (Clopper-Pearson) interval is 60.0% to 78.8%, about nine points either side. Pinning 70% to within five points at 95% confidence would take about 323 reviews. The post does not say who graded, what “accurate” required, or whether the 100 suggestions came before or after the adversarial reviewer was added. Its next sentence is “In the Regression Workflow, different turns investigate regressions, challenge diagnoses, and propose fixes to increase accuracy numbers through checks and balances.” It prints no accuracy number for the version with those checks.
If the grade applied to the 600 root-caused cases, 70% would be 420 accurate root causes, 24.7% of the 1,700 tracked (21.2% to 27.8% across the exact interval). The post does not say that it does.
Five of the seven quantities that would show find and fix have no number
The last block of the output above lists seven quantities that would show “find and fix” working and what the post prints for each. Two have numbers. Fix proposed, fix merged, recovery verified, time from alert to diagnosis and accuracy after adversarial review do not, and those are where the headline claim would be measured.
The headline against the body
The lead and the body describe the same system with different emphasis. The table sets the statements that carry the “find and fix” claim against what the body says about them.
| In the post | What the body adds |
|---|---|
| “find and fix regressions at an unprecedented speed” (lead) | No time to diagnosis, to a fix or to recovery is printed for the factory or for the manual process it replaced. |
| “has automated the entire process of fixing regressions” (lead) | “An engineer reviews the PR and decides whether to merge it.” Later: “Today, the bottleneck may be waiting for an engineer to review and merge the fix.” |
| “That’s 17 automated investigations every day.” (lead) | The unit is investigations, not fixes. The post says “Getting fixes right has been harder than finding the cause.” |
| “there’s a good chance the Performance Factory is already figuring out why” (closing) | No share of ChatGPT slowdowns that open a task or reach a root cause is given. |
Read charitably, “automated the entire process” means every step has an automated actor and the merge decision stays with an engineer by design. The post says it is exploring how agents could use other agentic review tools to “identify low-risk changes and eventually ship those fixes without a human in the loop.” The gap is in the counts: a reader cannot tell how many of the 600 root-caused cases led to a merged fix, or how long any of it took. Our earlier audit of Trail of Bits’ Codex backporting had the same shape, an automated step feeding a human review queue, and there a public tracker let the counts be recomputed. Here the post links no tracker, and its figure captions describe diagrams and one example task report.
The reviewer’s two questions, tested in a toy request world
The post does not say how its reviewer decides that requests are “similar” or that a metric “recovered.” Those choices decide whether the two counterfactual questions give right answers. To see how, I built a synthetic request world in which the true cause of each alert is known. It uses only the standard library, and every script is printed here.
The world
| Setting | Value in the toy world |
|---|---|
| Request classes and median latency | chat-short 300 ms, chat-long 520 ms, image 900 ms |
| Spread of one request | lognormal, sigma 0.35 |
| Hidden variable | region: half of all traffic is eu, and eu requests are 25% slower |
| Traffic mix before | 50% chat-short, 30% chat-long, 20% image |
| Traffic mix after a shift | 35% chat-short, 35% chat-long, 30% image |
| The bad build | image requests served by the new build are 30% slower |
| Rollout | half of requests on the new build at random, or 80% of eu requests and 20% of us requests |
| Window and alert | 4,000 requests per window; the alert fires when the overall mean rises more than 2% |
Four truths are simulated, so that the right answer to “did the deploy cause it?” is known each time.
| Truth | The build | Traffic during the incident | Traffic after the rollback |
|---|---|---|---|
| deploy only | image requests 30% slower | unchanged | unchanged |
| mix shift only | harmless | shifted toward heavier requests | still shifted |
| both | image requests 30% slower | shifted | still shifted |
| mix shift that reverts | harmless | shifted | back to normal, like a daily peak ending |
The function window() draws 4,000 requests: a class from the mix, a region at random, whether the request is on the new build, and a lognormal latency, and the bad build multiplies image latency by 1.3. The function incident() builds four windows: a baseline, an incident window with a random rollout, an incident window with a rollout that started in the slow region, and a recovery window after the rollback.
# world.py
"""A synthetic request world: three request classes, a hidden slow region, a bad build and a traffic shift."""
import math
import random
CLASSES = ("chat-short", "chat-long", "image")
MEDIAN_MS = {"chat-short": 300.0, "chat-long": 520.0, "image": 900.0}
SIGMA = 0.35 # lognormal spread of one request's latency
REGION_FACTOR = {"us": 1.00, "eu": 1.25} # hidden variable: eu requests are 25% slower
MIX_BEFORE = (0.50, 0.30, 0.20) # share of chat-short, chat-long, image
MIX_AFTER = (0.35, 0.35, 0.30) # traffic shifts toward heavier requests
SLOWDOWN = 0.30 # the bad build makes image requests 30% slower
N = 4000 # requests per window
def window(rng, mix, exposure, slowdown):
"""Rows (class, region, on_new_build, ms). exposure(cls, region) is P(request is served by the new build)."""
cum = [sum(mix[: i + 1]) for i in range(len(mix))]
rows = []
for cls in rng.choices(CLASSES, cum_weights=cum, k=N):
region = "eu" if rng.random() < 0.5 else "us"
new = rng.random() < exposure(cls, region)
ms = MEDIAN_MS[cls] * REGION_FACTOR[region] * math.exp(rng.gauss(0.0, SIGMA))
if new and cls == "image":
ms *= 1.0 + slowdown
rows.append((cls, region, new, ms))
return rows
NONE = lambda cls, region: 0.0 # nobody is on the new build
RANDOM = lambda cls, region: 0.5 # half the requests, picked at random
SLOW_FIRST = lambda cls, region: 0.8 if region == "eu" else 0.2 # the rollout started in the slow region
TRUTHS = { # name: (slowdown of the build, mix during the incident, mix after the rollback)
"deploy only": (SLOWDOWN, MIX_BEFORE, MIX_BEFORE),
"mix shift only": (0.0, MIX_AFTER, MIX_AFTER),
"both": (SLOWDOWN, MIX_AFTER, MIX_AFTER),
"mix shift that reverts": (0.0, MIX_AFTER, MIX_BEFORE),
}
def incident(seed, truth):
"""Four windows: baseline, incident with a random rollout, incident with a slow-region-first rollout, recovery."""
slow, mix_in, mix_out = TRUTHS[truth]
rng = random.Random(seed)
before = window(rng, MIX_BEFORE, NONE, 0.0)
random_rollout = window(rng, mix_in, RANDOM, slow)
slow_first = window(rng, mix_in, SLOW_FIRST, slow)
recovery = window(rng, mix_out, NONE, 0.0)
return before, random_rollout, slow_first, recovery
The checks
Each check answers one question from the post. The within-class split answers “did similar requests actually get slower, or were we serving more requests that were already slow?” The change in the overall mean equals a within-class part, the baseline class shares times each class’s change in mean, plus a mix part, the change in class shares times each class’s new mean. The two parts add up exactly, so a mean that rose only because heavier requests became more common lands in the mix part. The functions gap() and matched_gap() answer “did similar requests that weren’t exposed stay healthy?”, the second one inside each region. The function placebo_fails() compares exposed and unexposed chat-short requests, a class the bad build cannot touch. The function recovered() answers “did the affected metric recover?” by asking whether the mean is back within 2% of the baseline.
# checks.py
"""The checks: a within-class split, exposed-versus-unexposed cohorts, a placebo class, and a rollback test."""
import math
from world import CLASSES
def stats(values):
"""Mean, variance of the mean, count."""
n = len(values)
mean = sum(values) / n
return mean, sum((v - mean) ** 2 for v in values) / (n - 1) / n, n
def decompose(before, after):
"""Split the change in overall mean latency into a within-class part and a traffic-mix part."""
within = mix = var = 0.0
for cls in CLASSES:
m0, v0, c0 = stats([r[3] for r in before if r[0] == cls])
m1, v1, c1 = stats([r[3] for r in after if r[0] == cls])
w0, w1 = c0 / len(before), c1 / len(after)
within += w0 * (m1 - m0)
mix += (w1 - w0) * m1
var += w0 * w0 * (v0 + v1)
base = sum(r[3] for r in before) / len(before)
return within, mix, math.sqrt(var), base
def decomp_blames(before, after):
"""Blame the deploy only if the within-class part is clearly positive and at least 1% of the baseline."""
within, _, se, base = decompose(before, after)
return within > 1.96 * se and within > 0.01 * base
def gap(rows, cls, region=None):
"""Exposed minus unexposed mean latency for one request class, optionally inside one region."""
pick = lambda flag: [r[3] for r in rows if r[0] == cls and r[2] == flag and (region is None or r[1] == region)]
(ma, va, _), (mb, vb, _) = stats(pick(True)), stats(pick(False))
return ma - mb, math.sqrt(va + vb)
def matched_gap(rows, cls):
"""The same gap computed inside each region and averaged, so a rollout that favours one region cannot fake it."""
total = se2 = 0.0
for region in ("us", "eu"):
g, se = gap(rows, cls, region)
total, se2 = total + g / 2, se2 + (se / 2) ** 2
return total, math.sqrt(se2)
def cohort_blames(rows, matched=False):
g, se = (matched_gap if matched else gap)(rows, "image")
return g > 1.96 * se
def placebo_fails(rows):
"""chat-short is untouched by the bad build, so a gap there means exposed and unexposed requests differ."""
g, se = gap(rows, "chat-short")
return g > 1.96 * se
def recovered(before, recovery):
mean = lambda rows: sum(r[3] for r in rows) / len(rows)
return mean(recovery) <= 1.02 * mean(before)
The runner
The runner simulates 300 incidents for each truth, applies seven checks, and prints three tables. Save world.py, checks.py and run_rollout.py in one folder and run python run_rollout.py on Python 3.13; it takes about 16 seconds.
# run_rollout.py
"""Run every check against simulated incidents and print the tables."""
from checks import cohort_blames, decomp_blames, decompose, placebo_fails, recovered
from world import N, TRUTHS, incident
mean = lambda rows: sum(r[3] for r in rows) / len(rows)
SHORT = {"deploy only": "deploy", "mix shift only": "mix", "both": "both", "mix shift that reverts": "reverting"}
print("== One incident per truth (seed 1): the alert and its decomposition, ms of mean latency")
print(f"{'truth':24} {'baseline':>8} {'change':>8} {'within':>8} {'mix':>8}")
for truth in TRUTHS:
before, random_rollout, _, _ = incident(1, truth)
within, mix, se, base = decompose(before, random_rollout)
total = mean(random_rollout) - base
print(f"{truth:24} {base:8.1f} {total:+8.1f} {within:+8.1f} {mix:+8.1f} "
f"({total / base:+.1%} overall, within {within / base:+.1%} +/- {1.96 * se / base:.1%})")
RUNS = 300
CHECKS = [ # each check gets (baseline, random rollout, slow-region-first rollout, recovery)
("1 blame the deploy whenever the mean rose", lambda b, r, s, c: mean(r) > 1.02 * mean(b)),
("2 within-class split of the change", lambda b, r, s, c: decomp_blames(b, r)),
("3 exposed vs unexposed, random rollout", lambda b, r, s, c: cohort_blames(r)),
("4 exposed vs unexposed, slow region first", lambda b, r, s, c: cohort_blames(s)),
("5 same, matched on class and region", lambda b, r, s, c: cohort_blames(s, matched=True)),
("6 rollback: metric within 2% of baseline", lambda b, r, s, c: recovered(b, c)),
("7 split: effect with build, gone after rollback",
lambda b, r, s, c: decomp_blames(b, r) and not decomp_blames(b, c)),
]
hits = {name: {t: 0 for t in TRUTHS} for name, _ in CHECKS}
placebo = {t: [0, 0, 0] for t in TRUTHS} # check 4 with a placebo class: blame, clear, inconclusive
alerts = {t: 0 for t in TRUTHS}
for t_index, truth in enumerate(TRUTHS):
for i in range(RUNS):
b, r, s, c = incident(1000 * (t_index + 1) + i, truth)
alerts[truth] += mean(r) > 1.02 * mean(b)
for name, check in CHECKS:
hits[name][truth] += bool(check(b, r, s, c))
if placebo_fails(s):
placebo[truth][2] += 1
else:
placebo[truth][0 if cohort_blames(s) else 1] += 1
print(f"\n== Share of {RUNS} simulated incidents in which each check blames the deploy")
print(f"{'check':50}" + "".join(f"{SHORT[t]:>11}" for t in TRUTHS))
print(f"{'(alert fired: overall mean up more than 2%)':50}" + "".join(f"{alerts[t] / RUNS:>11.0%}" for t in TRUTHS))
for name, _ in CHECKS:
print(f"{name:50}" + "".join(f"{hits[name][t] / RUNS:>11.0%}" for t in TRUTHS))
print("\n== Check 4 with a placebo class added: blame / clear / inconclusive")
for t in TRUTHS:
print(f"{SHORT[t]:10} {placebo[t][0] / RUNS:6.0%} {placebo[t][1] / RUNS:6.0%} {placebo[t][2] / RUNS:6.0%}")
false_alarms = sum(placebo_fails(incident(5000 + i, "deploy only")[1]) for i in range(RUNS))
print(f"placebo class flags a properly random rollout in {false_alarms / RUNS:.0%} of {RUNS} incidents")
The mean rose in every case, and the split says why
== One incident per truth (seed 1): the alert and its decomposition, ms of mean latency
truth baseline change within mix
deploy only 585.2 +42.7 +42.3 +0.4 (+7.3% overall, within +7.2% +/- 2.0%)
mix shift only 585.2 +90.7 +9.0 +81.7 (+15.5% overall, within +1.5% +/- 1.8%)
both 585.2 +137.7 +41.5 +96.2 (+23.5% overall, within +7.1% +/- 1.9%)
mix shift that reverts 585.2 +90.7 +9.0 +81.7 (+15.5% overall, within +1.5% +/- 1.8%)
In the “mix shift only” incident the mean rose by 90.7 ms, 15.5% of the baseline. The within-class part is +9.0 ms, +1.5% with a 95% margin of 1.8%, which is inside the noise, and the mix part is +81.7 ms. In “deploy only” the within-class part is +42.3 ms (+7.2% with a margin of 2.0%) and the mix part is +0.4 ms. In “both” the parts are +41.5 ms and +96.2 ms. The last row repeats the second because the first three windows are identical; only the recovery window differs.
How often each check blames the deploy
In the table below the column “deploy” means deploy only, “mix” means mix shift only, and “reverting” means the mix shift that reverts. A perfect check would read 100%, 0%, 100% and 0%. Values are rounded to the nearest percent.
== Share of 300 simulated incidents in which each check blames the deploy
check deploy mix both reverting
(alert fired: overall mean up more than 2%) 99% 100% 100% 100%
1 blame the deploy whenever the mean rose 99% 100% 100% 100%
2 within-class split of the change 100% 2% 100% 4%
3 exposed vs unexposed, random rollout 100% 3% 100% 3%
4 exposed vs unexposed, slow region first 100% 100% 100% 100%
5 same, matched on class and region 100% 4% 100% 3%
6 rollback: metric within 2% of baseline 93% 0% 0% 95%
7 split: effect with build, gone after rollback 99% 2% 96% 3%
Row 1 is a stand-in for the “first plausible explanation” the post says early agents stopped at: it blames the deploy whenever the mean rose, so it is wrong in 100% of the “mix” and “reverting” incidents. Row 2, the within-class split, reads 100%, 2%, 100% and 4%. Row 3, the unexposed-cohort question with a random rollout, reads 100%, 3%, 100% and 3%, close to the 2.5% false-alarm rate of a one-sided 1.96 standard-error rule.
Row 4 asks the same question when the rollout started in the slow region, and it blames the deploy in 100% of incidents in every column, including the “mix” and “reverting” incidents where the build is harmless. The hidden region is the reason: the exposed requests include more slow eu requests, so the “similar requests” were not similar. Row 5 matches on class and region and returns to 100%, 4%, 100% and 3%.
Row 6, the rollback question, reads 93%, 0%, 0% and 95%. It passes the clean deploy-only case, but it misses the “both” case almost every time, because the mean stays elevated while the traffic shift remains, and it blames the harmless deploy in 95% of the “reverting” incidents, because the mean returns to baseline when the traffic does. Row 7 asks for a within-class effect while the build is live and no effect after the rollback, and it reads 99%, 2%, 96% and 3%.
A placebo class flags the broken comparison but cannot repair it
== Check 4 with a placebo class added: blame / clear / inconclusive
deploy 0% 0% 100%
mix 0% 0% 100%
both 0% 0% 100%
reverting 0% 0% 100%
placebo class flags a properly random rollout in 3% of 300 incidents
The class chat-short is untouched by the bad build, so a gap between exposed and unexposed chat-short requests means the two groups are not comparable. In the slow-region-first rollout the placebo flagged the comparison in 100% of incidents in every column, so check 4 stopped blaming and answered “inconclusive” every time. With a properly random rollout the placebo flagged 3% of 300 incidents, about the expected false-alarm rate. The placebo cannot say what the answer is. Matching on the hidden variable, as in row 5, does.
What the toy world does not show
Every setting is mine: a 30% slowdown on one class, a hidden region with a 25% penalty, an 80% and 20% exposure split. The post’s reviewer is an LLM session that may use many signals beyond these two questions, and its image-download example used a timeline check. The checks here use means, where the split is exact. For a percentile alert the split is not additive, and a team would reweight the old window to the new traffic mix instead; our tutorial on merging latency percentiles explains why percentiles cannot be averaged. Nothing here estimates how often OpenAI’s reviewer is right. The lab shows only that each question has a condition under which it returns a confident wrong answer, and the post does not say how its reviewer establishes those conditions.
What the post leaves out about the agent’s own exposure
As the census showed, “inject”, “untrusted”, “secret” and “credential” appear zero times in the post body, and “permission” appears twice. The post says the agent runs under a service identity so that its permissions can be scoped, and that it reads metrics, logs, traces, CPU profiles and memory snapshots. In many systems, logs and traces carry text that users influence, such as error messages, file names and request fields. The agent reads that text, proposes code changes and opens a pull request. The post does not say how the agent or the review step treats instructions found in that text, and I cannot tell from the post whether controls exist. I am noting only that none is described. For context, our test of the public Codex sandbox on Windows covers what that sandbox does and does not protect, and a report we covered from Forever Security says planted AGENTS.md and CLAUDE.md files steered Codex and Claude Code.
What you can build from public Codex pieces
OpenAI describes its cloud API as internal, but the public Codex documentation covers several of the same building blocks. I read three pages on November 19: skills, non-interactive mode and the SDK.
| Part of the post | Public counterpart | What the page says |
|---|---|---|
A skill that encodes the investigation playbook (perf-forensics) |
Codex skills | “A skill is a directory with a SKILL.md file plus optional scripts and references.” Also: “Skills use progressive disclosure to manage context efficiently.” |
| Start a Codex turn from an alert or a script | codex exec |
“By default, codex exec runs in a read-only sandbox.” With --json, “stdout becomes a JSON Lines (JSONL) stream”, whose events include thread.started, turn.started, turn.completed and turn.failed. |
| Resume a task after a review comment or a deployment | Codex SDK threads | “The TypeScript library lets your application start, continue, and resume local Codex threads.” Also: “Call run() again to continue on the same thread, or resume a past thread by providing a thread ID.” |
| Task tracker, dispatch loop, event subscriptions | none in these three pages | The non-interactive page lists “scheduled jobs” among its uses. The rest you would write yourself. |
A reader who builds the missing parts would also need a lease so that two dispatch loops cannot assign one task, and a limit on how many follow-up tasks a partly recovered regression may spawn; the post does not discuss either. I did not run an agent. The documentation navigation also lists an Agents API with sessions, scheduled automations and webhooks; I did not read those pages and cannot say whether they match the internal API the post describes.
What to measure if you copy the pattern
- Count every stage and time it. Record timestamps for alert, investigation opened, root cause found, fix proposed, fix merged and recovery verified, and publish counts plus median and 90th-percentile times. A rate such as 17 a day describes arrivals, not outcomes.
- Fix the unit once. Define an investigation as one agent run started by an alert and a case as one deduplicated regression, and report both with the ratio between them, because the post’s own text says one regression can show up as several incidents.
- Grade after the reviewer, on enough cases. A grade on 100 cases carries about nine points of uncertainty either way, and about 323 reviews are needed for five. Say who graded and what counted as accurate.
- Match the cohorts and keep a placebo. Compare exposed and unexposed requests inside strata of every variable that moves latency, such as region, request class, model and hardware, and include a class the change cannot touch. In the lab, the unmatched comparison blamed a harmless deploy in 100% of incidents, to the nearest percent, and the placebo caught it.
- Pair the rollback test with the split. Recovery alone could not separate a rollback that worked from traffic that changed back (95% false blame in the lab), and it missed the case where both causes were present (0% detection).
- Track what happens to agent-written fixes. Count merged unchanged, merged after edits and discarded. The post says fixes are harder than diagnosis, so that split is where the factory’s value would show.
- Use reweighting for percentile alerts. The additive split is exact for means only.
The bottom line
The Performance Factory post is a useful design document and an incomplete result. The architecture is concrete: states in configuration files, a dispatch loop, resumable cloud turns and subscriptions. The outcome evidence is four numbers that agree with each other and stop before the fix. The strongest claim in the lead is that the first workflow “has automated the entire process of fixing regressions,” while the body keeps a human at the merge and calls fixes the harder part. Read it as a description of how OpenAI organizes agent work on regressions, not as evidence of how often the agents are right or how much faster the work became. The toy lab adds one practical point: the two questions the reviewer asks are sound only when the exposed and unexposed requests are truly comparable and nothing else changed at the rollback, and the post does not say how those conditions are checked.
Sources and method
- How OpenAI’s Performance Factory keeps ChatGPT fast, OpenAI, November 19, 2026: every quotation about the factory comes from this page, read in full on November 19.
- Codex documentation read on November 19: skills, non-interactive mode and the SDK.
- Our earlier coverage used for context: OpenAI’s Control Room post, the Codex sandbox audit, the Trail of Bits backporting audit and the percentile tutorial.
- Method: the number census and arithmetic ran on Python 3.13.14; the toy world uses the standard library only and fixed random seeds, so the tables above reproduce exactly on the same Python version. No OpenAI system was run, queried or tested.








No Comment! Be the first one.