Red Hat’s Faster-Coding Warning Turns AI Productivity Into a Measurement Problem

A Red Hat engineer’s new blog post argues that AI coding agents never fixed software delivery’s real bottleneck: writing syntax was never the slow part. Independent research on whether AI actually speeds up delivery backs up that diagnosis, then goes a step further. The field’s own flagship study on the question has become so contaminated by how normal AI-assisted coding has become that the researchers say they can no longer trust it.

In a post published today on the Red Hat Blog, Senior Software Engineer Andriana Theodorakopoulou describes what she calls an AI productivity paradox. An AI coding agent, she writes, “can implement features in minutes that might have taken an engineer an hour.” Yet that speed does not translate cleanly into faster delivery once the code has to be reviewed, integrated, and maintained by a team. Her diagnosis is direct: “Code generation has become remarkably inexpensive. Engineering hasn’t.”

The Bottleneck Was Never Typing

Theodorakopoulou’s central claim is that “the real bottlenecks have always been system design, debugging, coordination, testing, and maintainability,” not the mechanical act of writing code. AI removes friction from the one step that was rarely the constraint, and the effects compound downstream: a model can produce 300 lines of plausible-looking code in seconds, but a reviewer still needs roughly the same amount of time as before to check its edge cases, confirm it fits the rest of the system, and catch logic it may have silently duplicated. Her framing of the outcome is blunt: “metrics like lines of code written or PRs merged go way up, even while actual system reliability and feature quality stay flat or decline.”

The post walks through several concrete failure patterns, including AI-generated pull requests large enough that nobody reviews them line by line, and a pattern she nicknames the “Memento problem” after the film about a man who cannot form new long-term memories: an agent partway through a multi-step task loses track of decisions it made earlier and starts contradicting its own prior work. Her practical recommendations include writing explicit repo rules into an AGENTS.md or .cursorrules file so a model does not have to infer architectural constraints on its own, using AI for fast, disposable prototypes rather than production code, routing heavy reasoning tasks to more capable models while reserving cheap models for boilerplate, and deliberately protecting less experienced engineers from skipping the step of understanding a problem before asking AI to solve it. Her conclusion: “AI doesn’t replace engineering discipline. It amplifies it.”

What Happens When Someone Actually Measures It

Theodorakopoulou’s post is a practitioner’s essay, built on experience rather than data. The most direct test of her thesis comes from METR, the AI evaluation nonprofit, which ran a randomized controlled trial in early 2025 to measure the effect directly. Sixteen experienced open-source developers, each with roughly five years of history on the specific mature codebase they were assigned, completed 246 real coding tasks. Each task was randomly assigned to allow or forbid the use of AI tools, mainly Cursor Pro paired with Claude 3.5 or 3.7 Sonnet, models that were current at the time.

The result inverted expectations at every level. Before the study, the developers predicted AI would speed them up by about 24 percent. Professional forecasters, including economists and machine learning researchers, predicted a gain of roughly 38 to 39 percent. After finishing their tasks, the developers still believed they had been sped up, estimating a 20 percent improvement. The actual, measured outcome, drawn from the published paper: developers took 19 percent longer to finish the same class of task when they were allowed to use AI. They were not only slower, they could not tell they were slower.

The Study Researchers Now Say They Can’t Trust

METR itself does not treat that 19 percent figure as settled. On its own site, the organization now flags the 2025 result as dated: “These results are out of date. We have released results that are current as of early 2026.” That follow-up, however, is not a clean confirmation or reversal. It is an admission that the experiment can no longer produce a reliable number at all.

For the subset of original developers who took part again, METR’s 2026 update now estimates a speedup of about negative 18 percent, with a confidence interval stretching from roughly negative 38 percent to positive 9 percent. That range comfortably includes zero, which means the study can no longer distinguish “AI made these developers slower” from “AI made no measurable difference” from “AI made them somewhat faster.” The reason is not noisy data. It is that the control group has stopped being a fair comparison. METR reports that an increased share of developers now say they would not want to do half of their work without AI, and, more specifically, that “30% to 50% of developers told us that they were choosing not to submit some tasks because they did not want to do them without AI.” That is true even though the study pays its participants $50 an hour to work on tasks of their own choosing. Developers who have grown reliant on AI assistance are opting out of the very trials designed to measure life without it, and METR says this selection effect likely means its own estimate is a floor, not a true reading. The organization is now redesigning the experiment from scratch rather than trust the numbers its current design produces.

Read one way, that is a research failure. Read another way, it is itself a finding: in under a year, AI-assisted coding became so embedded in daily practice that a well-funded, methodologically rigorous attempt to isolate its effect could not hold a clean control group together, even for pay. The honest state of the evidence is not “AI speeds developers up” or “AI slows them down.” It is that nobody currently has a trustworthy number either way.

DORA’s Verification Tax and the J-Curve

Google Cloud’s DORA research team, the group behind the long-running Accelerate State of DevOps research, reaches a compatible conclusion from a different angle in its “ROI of AI-Assisted Software Development (2026.01)” report. Rather than asking whether AI helps or hurts, DORA models AI adoption as a J-curve: a temporary dip in measured productivity before any net gain shows up, driven by three overlapping costs, according to reporting from InfoQ. The first is an ordinary learning curve as teams adapt their workflow. The second is what DORA calls the verification tax: the added effort of checking whether AI-generated code is reliable, secure, and architecturally consistent, precisely the review burden Theodorakopoulou describes from the inside. The third is strain on downstream processes, since a higher volume of code moving faster can overwhelm deployment pipelines and manual review gates built for a slower era.

According to InfoQ’s account of the report, DORA also cites Stanford University research to explain why any single productivity number is close to meaningless without qualification: AI yields a 35 to 40 percent productivity gain on simple, greenfield tasks, but its effect on complex legacy code is often 10 percent or less. InfoQ describes the report’s own illustrative ROI model, built around a 500-person engineering organization with a fully loaded salary of $176,000 per head, which works out to roughly $11.6 million in first-year returns against an $8.4 million investment, a 39 percent ROI. That figure is a scenario built around the best-case, greenfield end of the range DORA itself documents, not a guarantee, and it says nothing about the legacy, high-verification-cost work that dominates most enterprise codebases.

What Actually Helps

None of this argues against using AI coding agents. It argues against measuring their value by how much code they produce. The practices Theodorakopoulou recommends map closely onto the specific costs DORA identifies. Writing constraints into a file the model reads before it starts, of the kind covered in sxz.io’s own guide to AGENTS.md files, reduces the learning-curve cost by giving the model architectural context it would otherwise have to guess at. Treating fast AI output as a disposable prototype rather than a shippable feature, the same distinction that separates safe experimentation from the production-security gaps covered in sxz.io’s reporting on vibe-coded apps reaching the public internet, keeps unverified code out of the pipeline DORA warns is already strained. And treating code review as the real gate on delivery, not a formality after the “real work” is done, is the same argument sxz.io has made in covering how code review has become the actual production gate for AI-assisted engineering teams.

The throughline across a practitioner’s blog post, a contaminated research trial, and an industry ROI model is the same: none of the available evidence supports treating code-generation speed as a proxy for delivery speed. Until the research community has a design that can actually isolate the effect, the more reliable signal for any individual team is its own delivery data: deployment frequency, lead time for changes, change failure rate, and time to restore service, the four metrics DORA has tracked for a decade, measured before and after AI adoption rather than estimated from how fast the work felt. The one data point this whole episode nails down is that developers’ own sense of how much faster they are moving cannot be trusted on its own. METR’s participants were wrong about their own speed by roughly 39 percentage points, in the wrong direction, without knowing it.