Google’s Wet-Lab Agents Turn Safety Into a Simulation Gate, but the Errors That Skewed Results Were Found at the Bench

On October 16, Google Research published “Grounding AI agents in biological reality: Building blocks for autonomous wet labs”, a post by Pooja Rao, Petar Sirkovic and Annalisa Pawlosky that summarizes three research projects and links a paper for each. It frames them as steps toward agents that run lab experiments, learn from the results and end up “discovering single-digit nanomolar protein binders for cancer targets,” and its safety section says “Every protocol is dry-run in simulation and validated against the digital twin before touching reagents.”

I read the three PDFs against the post. Most of its figures match the papers, and the papers are more careful than the post about which check did which job. One project puts a simulation gate between an agent and a liquid-handling robot, and it was tested only on food dye and water. A second runs a design loop for DNA aptamers in which human operators ran the assay, and what protected it was a set of bench controls that no simulator supplied. The third ranks four protein-design campaigns first in software and then in yeast display, and the two rankings do not agree. The post says “Our next step is to weave these threads together,” so for now they are three separate pieces of evidence about where an agent’s work has to be checked.

Three projects, three kinds of evidence

The post presents the projects as one sequence, but they are separate studies with separate benches. The papers are PDFs hosted on Google’s gstatic.com and carry dates of October 12 (aptamers), October 14 (epitope excision) and October 16 (VERID) on their first pages. Table 1 sets out what each one did.

I searched the aptamer and epitope papers for the name VERID and found no match, so the robot gate and the biology results were not run together. The post says as much when it calls combining them its next step.

What VERID’s gate checks, and what it does not

The gate

VERID, short for VERified Infrastructure for Discovery Laboratories, keeps a live model of the workcell: where each piece of labware sits, how much liquid is in each well, which pipette tips are left. It extends the open-source PyLabRobot framework. The paper says “Every command block must pass a simulation gate before physical dispatch.” The simulator “tracks mass balance, tip state, deck geometry, and elapsed time,” and the paper lists what that catches: pipetting into a lidded plate, running out of tips or reagent, channel pitch that exceeds a container’s bounds, and aspirating air from an empty well or overflowing a destination well. The dispatch tool then “re-verifies the command block against the live digital twin state” before the robot moves.

The one simulator rejection the case studies describe is geometric. In Case Study 2 the agent tried to lower all eight channels of the Span-8 head into a reagent trough, and the simulator refused the move because “8 tips span at least 72.0 mm, exceeding the 71.2 mm trough length.” The agent switched to mixing with seven channels, a 63.0 mm span, and sampling in two passes of four.

What the data caught

The errors that skewed results in those case studies came from measurements. In Case Study 1 the first serial dilution cut concentration by only about 1.4 to 1.5 times per column for red, green and yellow dye and 1.6 to 1.8 times for blue, instead of 2.00, and the agent’s hypothesis was insufficient mixing. “The operator then reviewed synchronized video” of the wells, which showed dense dye stratified at the well floor and tips pressing flush against the bottom. After the agent raised the tips to a 1.5 mm clearance and changed its mixing, the average step factors came to 1.92 to 1.99. The post says “synchronized video and execution traces caught uneven dye stratification.” In the paper a person read the video and named the cause, and the paper’s own summary of the case study says “The human operator then identified the root cause.”

Case Study 2 shows what an accounting simulator cannot see. After the veto, the seven-channel plan ran, and the plate-reader results showed a 15 to 20% concentration discrepancy between adjacent dilution levels and a 28 to 47% absorbance drop on Channel 8 across all dyes. The agent designed a factorial diagnostic. Inverting which channel stayed dry moved the deficit from Channel 8 to Channel 1, and the cause turned out to be dye film on the inner walls of the seven mixing tips, which the paper puts at +4.9 µL (+40.6%) of carryover. The paper’s discussion explains the division of labor: because the twin already enforces volume and geometry rules, “the agent only needs to learn the unmodeled physical residuals.” The paper does not say whether an eight-channel mix would have behaved the same way, so I read this as the edge of what the gate models, not as a fault in it.

What the safety claims cover

The post says “Through VERID, agents never send unverified code directly to the liquid handling robots,” and the architecture backs that: an agent calls a verification tool, then a dispatch tool that re-checks the block, and supervised runs add “an operator approval gate locked to the verified script.” Two parts of the safety story are described rather than exercised. The paper puts chemical-compatibility rules in the gate, but every case study used “dye and deionized water mixtures,” and it reports no case in which such a rule fired. For biosecurity, the paper’s own wording is future tense: extending the gate “will further enable automated detection of de novo DNA and RNA synthesis protocols.”

The post’s line that “Every designed sequence goes through biosecurity screening” traces, in the papers, to one statement. The aptamer paper says its oligonucleotide pools were synthesized commercially by Integrated DNA Technologies and “underwent mandatory automated customer and sequence biosecurity screening.” That reads as screening by the synthesis vendor. I found no biosecurity statement in the epitope paper.

Case Study 3 and the fivefold claim

Case Study 3 is the source of the post’s claim that an agent, by identifying its own optical and fluidic artifacts across runs, was able “to improve accuracy more than fivefold.” A Challenger prepares four blinded binary dye mixtures, and a Solver agent gets one 96-well plate, a capped tip allowance and an overhead RGB camera. An independent plate reader scores the Solver’s predicted composition as a tempered KL divergence in millinats (mnats), where lower is better. Table 2 gives the scores the paper’s text reports.

The paper’s own sentence is that the 2B run reached 58 mnats, “a more than five-fold improvement over MC-2A.” By my arithmetic that is 307 divided by 58, or 5.3; against the 247 the 2A run started from it is 4.3. Three things keep me from reading it as one task getting five times better. The 2A run was a cold start and got worse as it went. The single-cycle solver scored 142 on that same challenge, better than the 2A multi-cycle run’s 307. And the paper says “the difficulty of identification varies significantly” by dye pair, while the pairs its text names for the two challenges differ (Red+Blue for 2A; Red+Green, Yellow+Green and Yellow+Blue for 2B), and it does not say the two used the same mixtures. What the three rows support is narrower: carrying learnings forward helped in Challenge 2B, and multiple cycles beat a single cycle in two of three pairs, each from one reported run.

In the aptamer loop, bench controls did the checking

The aptamer loop is simple. In each round the agents design about 1,000 DNA sequences, the pool is bought from a synthesis vendor, human operators run a magnetic-bead pulldown against the target, and sequencing reads out which sequences survived. A round takes about 10 days and about $3 a candidate, though the abstract says 1 to 2 weeks and $1 to $3. The post says “Agents iteratively design, synthesize, select, and sequence aptamers in a closed loop.” In the paper the agents design and analyze. The human operators held the assay conditions constant across all three rounds, and the agents’ suggestions for changing the protocol were logged as recommendations while the buffer stayed fixed. Table 3 shows the rounds.

The thrombin zero in Round 1 is a design flaw: the fixed PCR primers paired with the binding core, which the agents fixed with stem clamps in Round 2 after modeling the full construct. The myoglobin column is the instructive one. Between Rounds 1 and 2 the share of the pool enriched at least twofold, before any bead subtraction, rose from 6.5% to 12.4%, while the bead-corrected hit rate fell from 10.2% to 3.7%. The agents had kept the library about 58 to 61% G-quadruplexes in Rounds 1 and 2, and those sequences stick to beads and to other proteins: in the counter-screened benchmark 80.9% (72 of 89) of the binders were cross-reactive. The paper says “the analyst agent diagnosed this failure mode without human input, finding that 97% of cross-reactive binders were G-quadruplexes acting as non-specific polyanionic stickers.” In Round 3 the agents cut unclamped G-quadruplexes from 20.4% to 1.3% of the pool and moved 72.3% of it into other scaffolds, and the hit rate rose 9.1 times.

That diagnosis was possible because the assay carried its own controls. Every pool was sequenced next to triplicate target and bare-bead pulldowns, other proteins served as off-target references, and the paper reports that all random-sequence negative controls “remained at 0.0% across every round.” The software scorers the paper tested could not have done the same job. On the Round 3 myoglobin library (1,441 sequences, 292 of them hits) the paper scored twelve open-license predictor metrics from nine tools zero-shot. Boltz-2 interface-confidence scores landed at ROC-AUC 0.499 and 0.391, where 0.50 is chance, and only target-agnostic folding measures from ViennaRNA showed signal (0.729 and 0.763), which know nothing about the target protein. The discussion goes further: “raw Boltz-2 interface confidence scores anticorrelated with specific binding because the model rewarded compact G-quadruplex folds that adhered non-specifically to solid supports.” When the agents used Boltz-2 prospectively in Round 3, as calibration tiers, Tier A (iPTM above 0.7) produced 0 of 25 statistically significant hits. A score the agents could raise in software pointed the wrong way, and the correction was a round of bench data that cost about ten days and, by my arithmetic from $1 to $3 a candidate, $1,000 to $3,000 of synthesis and reagents.

The thrombin numbers need a qualifier the post omits. The 42.1% Round 3 rate includes designs built on known thrombin motifs; across strictly de novo candidates it is 37.1%. In single-clone titrations of eight full-length thrombin candidates, the abstract reports preliminary apparent affinities of 24.6 to 133 nM for the literature-derived motifs the agents engineered, against “low-micromolar binding” for strictly de novo scaffolds. The titrations and the pooled sequencing agreed with each other (Pearson r = 0.90), which is the paper’s evidence that the readout is real.

In the protein campaign, the software winner came last in the wet lab

The third paper addresses a compute problem. De novo binder design generates thousands of backbones and sequences, and the refolding step, where each candidate is folded with its target in AlphaFold 3, dominates the GPU bill. The target, FAP, is a 719-residue enzyme ectodomain that is costly to refold repeatedly, so the authors crop the target to the 85 to 130 residues around the designed interface. Across 15 benchmark targets and 3,666 designs the cropped folds reproduced full-target interfaces with a median Cα RMSD of 0.37 Å, at a measured cost in ranking: mean ROC-AUC fell from 0.777 to 0.729 on the 11 informative targets. The abstract’s 88% cut in GPU hours traces to a model, not a measured campaign. The figure caption labels it “Expected AF3 compute reduction under the token-bucket model”: an intact FAP complex falls into a 1,024-token bucket at 59.3 seconds per fold, a cropped one into the 256-token bucket at 6.9 seconds, which is 8.6 times faster per fold.

Table 4 reproduces the paper’s autonomy comparison on FAP and adds the wet-lab result. The paper titles its table “Autonomy-ladder outcomes at equal compute,” but the text defines equal as a cap: the budget was “capped at the 132.87 GPU-hours spent by the human baseline from Arm A.” Arm B, the hybrid in which a scientist fixes the strategy and the agent handles tactics, used 77.18 of those hours, or 58%, and Arm C stopped itself at 94%.

Read the first five columns and the post’s claim holds: the hybrid’s hit rate is 1.84 times the baseline’s, with about the same number of hits from 58% of the GPU time. Two footnotes qualify it. The baseline’s 5.06% is “rescored from its original backbone-only scoring, which reported 9.92%,” so the near doubling depends on a stricter side-chain metric applied to every arm, a reasonable choice that the paper discloses. And the B-variant, which changes two scoring settings and reaches 32.62% on the shared metric, is excluded from the autonomy comparison because it exists to separate scoring effects from agent effects.

The last column is the wet-lab result, and it does not follow the software ranking. The yeast-display library held 5,000 designs, 1,250 from each arm, plus random mutagenesis. After three rounds of cell sorting at 300, 150 and 75 nM target, deep sequencing found 571 enriched unique sequences, of which 60 matched a computational design exactly: 8 from Arm A, 6 from Arm B, 24 from Arm C and 22 from the B-variant. The arm with the highest in-silico hit rate among the three comparable arms contributed the fewest. The authors read “the 3- to 4-fold higher representation of Arm C relative to Arms A and B” as suggesting that “the in silico prediction metrics used by autonomous agents effectively accounted for expression.” That is a hypothesis. The paper’s Table 1 lists one campaign per arm, and the paper’s own caveat is that the autonomy comparison used “a single large enzyme target (FAP) and a single backbone class (three-helix bundles) driven by one LLM controller.” Four arms and one run each cannot separate that explanation from chance, so I would treat both rankings as unsettled.

The post’s nanomolar claim needs the same care. It says “the top designs yielded binders with single-digit nanomolar affinity, a level of tight, drug-like binding typically achieved only after extensive rounds of laboratory optimization.” The paper reports clonal binding assays on four enriched clones, with apparent dissociation constants from 3.0 to 9.1 nM measured in the yeast display format. It does not say whether those four are among the 60 exact designs or are mutagenized variants, and it lists “binding characterization of solubly expressed binders, evaluation of FAP domain specificity, and inhibition assays of FAP catalytic activity” as future work. The epitope paper does not call the binders drug-like.

What this means for anyone building agent harnesses

A gate and a control answer different questions. A simulation gate answers “can this run?” It is fast, deterministic and cheap, which is why VERID can make it mandatory. A control answers “is this true?” and costs a round of real measurements. The Google papers show both doing real work, and they show the second deciding the science.

Put negative controls inside every loop. The aptamer team’s random-sequence negatives stayed at 0.0% in every round, which showed the readout was not simply flagging everything, and its bare-bead and off-target pulldowns showed which enrichment was sticking rather than binding. An agent harness for code or infrastructure needs the equivalent: inputs that must fail, held out from the agent. Our SQLite plan-gate tutorial shows the same split in miniature. A plan gate and a step budget stopped slow queries, and one local model’s wrong answer still passed both, because neither layer checks the answer. Our OPA guardrails tutorial names two ways a policy silently passes something it should block: a shape you did not model, and a value that is missing or unknown. The tip-wall carryover is the first of those, an effect the twin did not include.

Treat any score the agent can raise as something it will raise. Raw pull-down went up while specific binding went down, and the paper notes that a supervised classifier trained on early rounds before bead subtraction “learned G-tract frequency as a positive feature and inverted its accuracy on subsequent pools.” The post’s own list of open problems includes “ensuring agents optimize for true target binding rather than unintended experimental shortcuts.”

Keep a record the agent does not write. VERID’s trace dossier, with state snapshots, execution logs and synchronized video, is what let a person see the stratified dye behind a failure the agent had put down to insufficient mixing. It is close to what our October 11 post on Satya Nadella’s insider-risk essay summarized as a record of what the model did that does not depend on the model’s own account.

And ask what “autonomous” covers. Human operators held the assay fixed, an approval gate sat in front of supervised robot runs, and in the protein campaign the best arm in software was the one where a scientist set the strategy. The papers say so. The post’s verbs are stronger.

What Google has not shown yet

The aptamer paper says its complete sequence-level tables “will be made available upon peer-reviewed publication,” which implies the paper has not been through review; the other two PDFs say nothing either way, and the VERID and epitope papers carry no code or data statement that I could find. The VERID case studies ran on “an internal preview checkpoint of Gemini 4 Argon,” and although the paper calls VERID “model-agnostic,” it reports no other model. The three projects have not been run as one pipeline, so there is no evidence yet on whether the gate, the bench controls and the design agents hold together when an agent steers a robot through a biological assay. The VERID paper’s own conclusion says as much: “The next step is demonstrating that these gains lead to significant biological breakthroughs on grand challenge research questions.”

Sources and method

The sources are the Google Research post and the three PDFs it links: VERID, aptamers and epitope excision. I extracted the PDFs’ text with pdftotext and checked every quotation in this piece against the saved text of the post or the PDF it comes from. I read the text, tables and captions, not the figure images, and I ran no laboratory or design experiments. Figures I derived myself are my arithmetic: the hit-rate ratios, the per-arm enrichment shares (each count divided by the 1,250 designs that arm contributed), the 307-to-58 and 247-to-58 ratios, the 9.1-fold and 8.6-fold values and the $1,000 to $3,000 per round. The script below recomputes them from counts printed in the papers.

# check_numbers.py
# Recompute the derived figures from the counts printed in the three Google papers
arms = {  # arm: (in-silico hits, designs, GPU-hours, exact designs enriched in the yeast library of 1,250 per arm)
    "A  scientist alone": (1672, 33024, 132.87, 8),
    "B  human + agent": (1790, 19184, 77.18, 6),
    "C  agent alone": (1803, 29176, 124.43, 24),
    "B-variant": (7106, 21784, 90.94, 22),
}
CAP = 132.87
print("arm                 hit rate  hits/GPU-h  GPU-h vs cap  enriched exact per 1,250")
for name, (hits, designs, gpu, enriched) in arms.items():
    print(f"{name:19} {100 * hits / designs:6.2f}%  {hits / gpu:9.1f}  {100 * gpu / CAP:10.1f}%  {100 * enriched / 1250:21.2f}%")
print()
print("hit rate, B over A:", round(9.33 / 5.06, 2), "x; GPU-hours saved:", round(100 * (1 - 77.18 / CAP), 1), "%")
print("exact designs enriched, all arms:", sum(a[3] for a in arms.values()), "of", 4 * 1250,
      "=", round(100 * 60 / 5000, 1), "%")
print()
print("Case Study 3 ratios (millinats):")
print("  307 -> 58:", round(307 / 58, 2), "x   247 -> 58:", round(247 / 58, 2), "x   142 vs 307 (single vs multi, 2A):", round(307 / 142, 2), "x worse")
print()
print("per-fold AF3 runtime, token-bucket model: 59.3 s -> 6.9 s =", round(59.3 / 6.9, 2), "x,", round(100 * (1 - 6.9 / 59.3), 1), "% less")
print("myoglobin hit rate 3.7% -> 33.5%:", round((184 / 549) / (9 / 244), 2), "x")
print("aptamer cost per round at $1 to $3 per candidate and about 1,000 candidates: $1,000 to $3,000")

Its output on Python 3.13.14:

arm                 hit rate  hits/GPU-h  GPU-h vs cap  enriched exact per 1,250
A  scientist alone    5.06%       12.6       100.0%                   0.64%
B  human + agent      9.33%       23.2        58.1%                   0.48%
C  agent alone        6.18%       14.5        93.6%                   1.92%
B-variant            32.62%       78.1        68.4%                   1.76%

hit rate, B over A: 1.84 x; GPU-hours saved: 41.9 %
exact designs enriched, all arms: 60 of 5000 = 1.2 %

Case Study 3 ratios (millinats):
  307 -> 58: 5.29 x   247 -> 58: 4.26 x   142 vs 307 (single vs multi, 2A): 2.16 x worse

per-fold AF3 runtime, token-bucket model: 59.3 s -> 6.9 s = 8.59 x, 88.4 % less
myoglobin hit rate 3.7% -> 33.5%: 9.09 x
aptamer cost per round at $1 to $3 per candidate and about 1,000 candidates: $1,000 to $3,000