A Harvard Study of 718 Firms Turns AI Coding Agents Into a Code-Review Queue Problem

AI coding agents are very good at producing code. A Harvard job market paper, covered by Ars Technica on October 9, asks what happens after the code is written. Fiona Chen and James Stratton used engineering analytics data from 718 firms and estimate that once a firm adopts AI coding agents, lines of code added per worker-month rise 30 percent, commits 20 percent and pull requests 23 percent. Jira issue resolution, their measure of finished work, does not change by a statistically significant amount. The place where the difference piles up is code review: the time between opening a pull request and merging it rises 49 percent on the paper’s point estimate.

I read the paper itself, “Artificial Intelligence in the Firm: Bottlenecks in Software Production” (current version dated August 4, 2026, first version January 7, 2026), and checked every figure below against it. Where I add arithmetic of my own, I say so. The paper’s central finding is about the pipeline, not about jobs: the authors write that “while AI tools substantially raise coding productivity, these gains do not translate into proportional changes in software output or employment.”

What the study measured

A vendor’s customer data covering 718 firms

The data come from Jellyfish, whose main product, in the authors’ description, is an analytics platform that lets companies see what their engineering teams are doing. The analysis sample is the Jellyfish clients that opted in to research use and have data across the whole window: 718 firms and 725,938 workers from January 2021 to March 2026. The introduction rounds this to 700 firms and 700,000 workers, and Ars Technica used the rounded figures. The records combine GitHub activity, Jira issues and Google Calendar events, about 300 million work events in all, plus usage data for GitHub Copilot, Cursor and Claude Code. Version control data covers 99.6 percent of the firms and task management data 94.4 percent. The average firm had 1,652 workers and 241 active engineers in January 2023 and was 19.3 years old, so this is a sample of established companies, 52 percent of which sell a software license directly.

How the authors date adoption

A firm counts as adopting AI coding assistants on the earliest seat activation recorded in the GitHub Copilot or Cursor integrations. It counts as adopting AI coding agents on the earliest signal across the Claude Code integration and GitHub activity: bot accounts matched by login and display name, plus tool signatures in commit messages and pull request bodies such as “Co-authored-by: Claude Code”. The paper dates the first commercial agent to November 2024, Cursor’s agent tool, and reports that the share of firms with any agent rose from near zero in October 2024 to over 95 percent by January 2026. Assistant adoption is followed for 16 months.

The design in one paragraph

The authors run a staggered difference-in-differences regression: each firm’s monthly outcomes are compared before and after its own adoption month against firms that adopt earlier or later, with firm and month fixed effects, controls for baseline firm size, and standard errors clustered by firm. One detail changes how to read the agent numbers. In the authors’ words, “the estimated effect of AI coding agents represents the added effect of agent adoption, relative to AI assistant adoption.” The identifying assumption is that adoption timing, set by legal, security and budget approvals that took as little as a day at one interviewed firm and weeks to months at another, is unrelated to how a firm’s output was already trending. The authors checked pre-period coefficients, compared their two-way fixed effects against four newer estimators for one outcome (lines of code added) and report no significant differences, and interviewed 17 engineers and managers.

The numbers, with baselines

The table lists the paper’s estimated effects of agent adoption next to each outcome’s baseline mean. The first four rows are per worker-month; the rest are per pull request or shares, as labeled. The last column is my arithmetic: the estimate plus or minus 1.96 standard errors, a normal approximation that the paper does not print.

More code, no detectable change in finished work

The coding gains hold across three measures, which matters because lines of code alone could reflect verbose output. Pull requests are a coarser unit that another person must review, and they rose 23 percent. Finished work did not follow. The estimated change in Jira issues resolved is +0.12 per worker-month on a baseline of 3.67, with a standard error of 0.17. My arithmetic on those two published numbers gives an interval of roughly -0.21 to +0.45 issues, or about -6 percent to +12 percent of baseline, which matches the authors’ statement that the estimates rule out an output gain larger than 12 percent. Jira epics, which group issues into whole features or projects, also show no significant effect. The authors also checked whether firms shifted to bigger issues that a flat count would hide, using an LLM-assigned length label and a predicted cycle time, and found no change in average issue size.

Employment shows no detectable movement either. For agents the authors estimate “a precise zero effect on overall employment, ruling out a decline greater than 2.9%; and a noisier zero effect on engineering employment, ruling out a decline greater than 13.8%.” They are careful about what that means. It is a short-run estimate, it is partial equilibrium for one firm at a time, and in their words “it is plausible that this null partial equilibrium effect masks a negative general equilibrium effect,” for example if an adopting firm’s gains come at the expense of rivals. The finding is that adopting firms did not shed staff, not that AI leaves labor demand untouched.

Where the extra code goes: the review queue

The paper’s explanation starts with how little of an engineer’s time goes to writing code. Imputed from the Jellyfish signals the figure is 42 percent, and in a survey of 100 engineers on Prolific it is 30 percent. The rest goes to complementary work such as planning, review and testing. My back-of-envelope arithmetic: if coding is 30 to 42 percent of the work and everything else stays fixed, even an infinitely fast coder would shorten the total by at most 30 to 42 percent, a throughput ceiling of about 1.4× to 1.7×. That is an Amdahl’s-law calculation, not a result from the paper.

After agent adoption, the review stage is where the paper finds the strain. The average time from submitting a pull request to merging it rises by 3.45 days on a 7.03-day baseline, which is 49 percent. The share of pull requests with at least one formal request for changes goes up by 0.12 on a baseline of 0.13, so from about 13 percent to about 25 percent. Review comments per pull request rise by 0.58 on a baseline of 1.66, to about 2.2. The share of workers who comment on or approve at least one pull request rises 14 percent, from 29 percent to about 33 percent, while the share who write code does not change significantly. Pull requests show no significant change in size: lines added per pull request moved by -68 on a baseline of 1,168, with a standard error of 127, so larger submissions do not obviously explain the extra comments and change requests. AI assistants, by contrast, show no significant change in review time, changes requested or comments.

The 49 percent deserves a caution that the headline omits. With a standard error of 1.60 days, my approximate 95 percent interval for the increase runs from about 0.3 to 6.6 days, roughly 4 to 94 percent of the baseline. The estimate is significant but loose. The change-request and comment estimates are much tighter, with standard errors of 0.01 on 0.12 and 0.11 on 0.58, so the clearest evidence of review strain is that reviewers push back more often and say more, and the exact length of the added delay is less certain.

Here is a second piece of my own arithmetic. By Little’s law, the average number of items waiting in a stable queue equals the arrival rate times the average wait. Take the paper’s baseline of 5.34 pull requests per worker-month, about 0.175 a day, and 7.03 days to merge: roughly 1.2 pull requests open per worker at any moment. Add the estimated effects, 6.56 pull requests a month and 10.48 days, and the figure is roughly 2.3. Taken at face value, each worker has about 1.8 times as many pull requests in flight. If the time-to-merge estimate takes the ends of its interval instead, the ratio runs from about 1.3 to about 2.4. This treats nearly every pull request as eventually merged and multiplies averages, so read it as an illustration of scale rather than a measurement.

An interviewee quoted in the paper, a junior software engineer, describes the effect on the ground: “People could make a huge number of commits quickly, but code review was still the bottleneck.” The same interviewee says that after outages caused by AI-generated commits, “there were also new rules requiring senior engineer approval for AI-generated code, which created even more review bottlenecks.”

AI review tools have not drained the queue yet

A natural objection is that AI can review too. The authors looked: nearly 80 percent of firms had used AI tools in the code review process by March 2026, yet “only 23.3% of all review comments are generated by AI, and only 10.8% of pull requests receive at least one AI-generated comment.” The tools they track include CodeRabbit, Greptile, GitHub Copilot Reviewer, Cursor Bug Bot, Graphite and Bito. One wording point if you cite the figures: Ars Technica wrote that AI agents were “only responsible for 23.3 percent of all review comments and 10.8 percent of all pull requests,” while the paper’s second figure counts pull requests that receive at least one AI-generated comment, and its subject is AI code review tools, not coding agents.

What the evidence can and cannot support

Firm-level adoption dilutes the effect

A firm is treated from its first adoption signal, however few engineers use the tool at first. By month 12 the median firm has about 20 percent of its engineers using agents, the top quintile about 60 percent and the bottom quintile close to none. The authors say this is why their coding gains sit toward the low end of the literature: “the estimate aggregates heterogeneity in take-up.” I searched the full text, appendix included, for a split of the output or review estimates by take-up or firm size and did not find one. The only heterogeneity cut the paper reports on its estimates is by worker seniority for employment, which found no significant effects. Whether firms with heavy agent use see a longer review delay than firms with light use is therefore not something this version of the paper answers.

A job market paper built on one vendor’s customers

This is a working paper, listed as a job market paper, and the version I read notes no journal publication. Every firm in the sample is a Jellyfish client that opted in to research use. The authors disclose that Jellyfish reviewed the manuscript “solely to verify that it does not disclose proprietary information” and that it was “not reviewed for its scientific content,” and they state that “the conclusions of this study are those of the authors and do not represent the views of Jellyfish.” The work was funded by the Stripe Economics of AI Fellowship, Harvard Kennedy School’s James M. and Cathleen D. Stone Program in Wealth Distribution, Inequality, and Social Policy, and the Schmidt Sciences AI at Work Program. On bias, the authors expect any remaining confounding to “produce upward bias,” because “factors that accelerate adoption would also tend to produce higher outcome growth, leading us to overstate treatment effects.” That makes the flat output result harder to dismiss.

Review signals stand in for quality

The paper’s quality evidence is indirect. Its model has a “bug rate” channel, and the authors read the rise in changes requested and comments per review as evidence for it. I found no production defect or incident rates in the paper; the only mention of outages is the interviewee’s account above. Change requests are a reasonable proxy, since reviewers ask for changes when something looks wrong, but they also reflect team norms, such as the senior sign-off rules that interviewee describes, so the review numbers mix a quality signal with a policy response.

The window closes in March 2026

The data end in March 2026. Ars Technica points out that agents have been updated considerably since then and that many firms are still learning when to use them, so review costs could shrink with experience. And because over 95 percent of firms had adopted an agent by January 2026, identification leans on differences in timing among adopters rather than on a large group that never adopted.

What an engineering team can do with this

Measure the pipeline, not the keystrokes

The authors’ advice to managers is direct: “Many managers rely on intermediate productivity metrics, such as lines of code or commits; however, these may not accurately capture the true gains from AI adoption.” The review measures the paper uses can be computed from GitHub. Track them before and after an agent rollout, per team:

• Time to merge. The paper counts the days between a pull request being submitted and merged: merged_at minus created_at on the pull request object.
• Change-request rate. The share of pull requests with at least one review whose state is CHANGES_REQUESTED, from the pull request reviews endpoint.
• Review comments per pull request. The pull request object exposes review_comments (inline) and comments (conversation) separately, so pick one definition and keep it fixed.
• Who reviews. The share of active people who comment on or approve at least one pull request in the month, which is the paper’s definition.
• Finished work. Issues resolved per person per month, the paper’s measure of output, so you can see whether the extra code arrives as shipped work.

I confirmed the field names above on a public pull request through GitHub’s REST API, and the CHANGES_REQUESTED state in the reviews documentation.

Budget review capacity before buying more generation

The paper’s own implication is stated plainly: “Realizing the benefits of AI adoption requires complementary investments in downstream review capacity, rather than simply increasing code output.” Its model says the right investment depends on how the tools change production. If AI works only through coding productivity, reallocating people to review can relieve much of the bottleneck, and the data show firms starting to do that. If the effect runs through the quality of the draft code, the authors write that “alleviation would require improvements in the technology itself.” The interviewee’s story about senior sign-off shows the risk in between: a policy that routes more AI-written code through fewer approvers can lengthen the queue it was meant to protect.

The same constraint outside company walls

Task-level evidence adds a different angle. In its 2025 randomized trial, METR found that 16 experienced open-source developers took 19 percent longer on 246 real issues when they were allowed to use AI tools, although they expected a 24 percent speed-up and, afterward, still believed AI had sped them up by 20 percent. Our earlier piece on Red Hat’s productivity warning covers that study and DORA’s data. The designs differ, a task-level experiment with early-2025 tools against firm-level adoption across 718 companies, so I would not call them confirmation of each other. Both are reasons to measure outcomes rather than assume them.

The queue also runs between companies and open-source projects. Anthropic’s OSS Scanner, which we covered, emails maintainers findings that Anthropic says “will be fully model-generated, without human review or triage.” In the launch post, Anthropic says it has discovered over 29,000 candidate vulnerabilities but has manually reviewed and triaged approximately 6,000 of them, and that “we remain bottlenecked on our human capacity to validate these findings.” Google, for its part, stopped accepting product vulnerability reports to its open-source bounty on October 1, citing automated submissions. I read these as the same arithmetic at ecosystem scale: generation is cheap, review is the scarce resource, and someone has to pay for it. Our tutorial on reviewing AI-generated Python code shows what that review has to catch: in its lab, Ruff and mypy flagged style and typing issues, Bandit flagged nothing, and none of the three caught the model-written module’s functional bugs.

The bottom line

The Harvard paper does not show that AI coding agents fail. It shows, on one vendor’s customer base through March 2026, that more code per worker arrived without a detectable rise in finished Jira work, and that the extra code spent its time in the review queue, where pull requests came back for changes about twice as often and, on the point estimate, waited about 3.5 days longer. For teams, the number to watch is not how much code the agent writes but how long a pull request waits and how often it bounces. For researchers, the open questions are the ones this version leaves open: whether heavy-use firms see larger review delays, whether production defects follow the review signals, and whether the pattern survives newer agents and a longer window.

Sources and method

I read the paper’s introduction, data description, results, bottleneck analysis, model discussion and conclusion, checked its appendix tables, and searched the full text, appendix included, for terms such as heterogeneity, defect and incident. Every figure in the table and every quotation was checked against that text. I did not work through the model’s derivations or the interview transcripts in the appendix. I also read the Ars Technica report of October 9, 2026, METR’s 2025 post, Anthropic’s OSS Scanner launch post and GitHub’s REST documentation, and queried a public pull request through GitHub’s REST API to confirm field names. The approximate intervals, the review-queue arithmetic and the Amdahl ceiling are my own calculations from the paper’s published estimates and inherit its assumptions. The figures are the authors’ estimates from a working paper; I did not have access to the underlying Jellyfish data and could not rerun the regressions.