Google’s Agile Deliberation Turns Image-Classifier Training Into a Conversation, but the Blog Outruns Its Paper: Nine Non-Expert Participants, Two Significant Survey Items, and No Labeling-Consistency Measure

On November 4, Google Research published Agile Deliberation: How AI helps humans define the gray areas, by Otilia Stretcu and Leijie Wang. It describes a workflow for building an image classifier for a fuzzy category such as healthy food or clickbait images that promise unrealistic rewards: the system proposes sub-categories, shows the user borderline images, and rewrites the classifier’s prompt from the user’s feedback. The post says the team “evaluated Agile Deliberation across 18 live, 1.5-hour sessions with human participants” and found that it “drastically reduced user fatigue and anxiety” and produced models “that outperform standard baselines.”

The study behind the post is a CVPR 2026 paper with a supplement, and the authors released code on GitHub. We read all four, retyped the paper’s results tables into a short script, and checked each of the blog’s headline claims against what the paper reports. The paper is the more careful document: it says outright that the study is small and that its main performance comparisons are between different people. The blog uses stronger verbs than nine participants can carry, and three of its claims have no counterpart in the paper or the supplement.

What we found in short

• The study is nine people. The 18 sessions are two 90-minute sessions per person, one per system, on different concepts. The blog never gives the number of participants, and the supplement lists the study’s scale as its first limitation.
• The F1 comparisons are between different people. Each person tried each system on a different concept, so classifier performance is compared “only between participants.” With four or five people per group, our simulations show a 3-point gap appearing by chance in roughly one study in five to one in three.
• Three blog claims have no counterpart in the paper. The participants were non-experts, not the “expert participants” the blog cites; there is no measure of labeling consistency; and there is no image-level analysis behind gains that were “most dramatic on the most challenging subjective images.”
• Two of six survey items clear p < .05. Effort required and feeling insecure, stressed or annoyed do. Frustration and mental demand do not, although the blog’s body text says participants felt “significantly less stressed, less frustrated, and less mentally drained.”
• The F1 gains are precision bought with recall. In the Agile Deliberation groups precision rose 22 and 11 points while recall fell 17 and 15. Scored with an F2 that weights recall more heavily, the Pay-to-Play gain disappears.
• The code release is a demo, not a replication kit. It walks through a healthy-food example over the LAION-400M image set and needs a Gemini API key. The study used WebLI and a proprietary 100-million-image set, and no study data is published.

What Agile Deliberation does

The framework has two stages. In concept scoping the system decomposes the starting concept into a hierarchy of sub-concepts, plus explicit exclusions, that the user accepts or rejects. In concept iteration it retrieves a batch of images near the boundary of the current definition, asks the user to label them, and rewrites the definition with a language model. The written definition is the classifier: it becomes the prompt for a vision-language model. The paper uses Gemini 2.5 Pro for decomposition and Gemini 2.5 Flash for everything else, and says the models “have not been further trained or fine-tuned in this work.”

The idea is reasonable and easy to state. People rarely start with a complete definition of healthy food, and a fruit smoothie, or fried fish served with vegetables, forces a decision they had not made yet. The question is how much of the blog’s evidence the study can bear, so the next sections take its design, its performance numbers and its survey in turn.

The study behind the claims

Nine people, two concepts, two systems

The paper’s experiment section is plainer than the blog about the design. The authors “recruited nine participants without prior professional content moderation experience through an open call within our organization,” and paid each of them $100. “Each participant completed two sessions on different concepts, one using Agile Deliberation and another Manual Deliberation, totaling 18 sessions.” In each session the participant spent about 45 minutes writing a concept definition, then labeled 200 held-out images, which is how F1 was computed against that person’s own labels, then filled in a short survey and took part in a 15-minute interview. The supplement puts the total at “nine participants across 18 sessions (approximately 27 hours of study time)” and says this “may limit the statistical power and generalizability of our findings.”

Two concepts and two systems for each person makes a small crossover design with a catch. A person cannot define the same concept twice, so the authors gave each person a different concept for each system. The paper spells out the consequence: the design “allows comparing subjective experience within participants, but classifier performance only between participants, since each system was trained on a different concept.” It adds that “such between-participant comparisons are only meaningful with enough participants per concept.”

The paper does not say how many people were in each group. Each person appears once in an Agile Deliberation group and once in a Manual group for the other concept, so the nine split into a group of a people who used Agile Deliberation on Pay-to-Play (and Manual Deliberation on Healthy food) and a group of 9 minus a who did the reverse. If the split was as even as nine allows, every cell of the results table averages four or five people. The authors explain the size themselves: “The substantial time required for these studies limited the total number of sessions.” Small human studies behind large claims have come up on this site before, for example in Microsoft’s StudentSim, and we audited another Google Research post against its papers in the wet-lab agents piece.

Blog statements against the paper

The table sets eight statements from the blog post against what the paper and its supplement contain. Quotation marks mark exact text. The blog writes Pay-to-Play where the paper writes paid to play.

The experts who were not in the study

The experts in the paper are the five professional content moderators interviewed while the framework was designed and the 20 expert-authored definitions the authors coded. The supplement says the moderators defined a concept and searched for borderline images “in a think-aloud manner” and that “Existing tools provided only limited support.” The sentence closest to the blog’s “blind spots” passage sits in the supplement’s analysis of the 20 definitions: borderline cases “made gaps in the definition immediately visible and helped practitioners clarify ambiguous language, resolve inconsistencies, and sharpen the intended scope.” That describes how experts work without the system. The evaluation that tested the system used non-experts.

Consistency and the hardest images

The blog says participants “demonstrated significantly higher labeling consistency over time.” We searched the paper and the supplement for any consistency measurement, such as agreement between rounds, repeated labels or a stability statistic, and found none. Consistency appears as a design intent: the structured definition “encourages consistency in how users apply their definitions across rounds.” The only round-by-round evidence is Figure 3, which plots the classifier’s F1 for each round. The paper notes that “the number of rounds was limited by session duration” and says the fluctuations “partly reflect the system’s real-time design.” No test accompanies the trend, and a rising F1 is not labeling consistency.

The claim about gains “most dramatic on the most challenging subjective images” has the same gap. The paper reports one F1 per person over a 200-image test set and does not break results out by image difficulty. The nearest statement, quoted in the table, concerns people whose own understanding differs from the model’s defaults, and its evidence is that the Manual Deliberation group on Healthy food began with a zero-shot F1 of 0.82.

The manual baseline

The blog describes manual deliberation as spreadsheets and guesswork. The paper ran a stronger baseline: participants used an image search engine to look for borderline images in the target dataset, could consult Modeling Collaborator’s generated prompts, and tried an average of 7.3 search queries. That is a fairer comparison than the blog suggests, and it makes the manual group’s smaller gains more informative. The two groups still hold different people.

What the F1 numbers can carry

Gains measured on different people

Table 1 of the paper reports mean F1 for each concept and each group of people, with standard deviations in parentheses. System means Agile Deliberation for the people who used it and Manual Deliberation for the others, and gain is that system’s F1 minus the same group’s zero-shot F1, in points.

Our script reproduces all four printed gains and the two headline averages, 10.5 points over zero-shot and 7.0 over Modeling Collaborator. The paper gives 7 for the second in Section 4.1 and 7.5 in its abstract, and rounding in the table can account for the half point.

Paid to play  Agile group   F1 0.48 -> 0.59 = +11, printed +11 (matches)
Paid to play  Manual group  F1 0.43 -> 0.51 = +8, printed +8 (matches)
Healthy food  Agile group   F1 0.48 -> 0.58 = +10, printed +10 (matches)
Healthy food  Manual group  F1 0.82 -> 0.79 = -3, printed -3 (matches)
Agile over zero-shot, mean of two concepts: 10.5 points (paper: 10.5)
Agile over Modeling Collaborator, mean of two concepts: 7.0 points (paper: 7 in Section 4.1, 7.5 in the abstract)
Paid to play: Agile gain minus Manual gain = +3 points (different participants in each group)
Healthy food: Agile gain minus Manual gain = +13 points (different participants in each group)

Look at the Healthy food rows. The same concept produced a zero-shot F1 of 0.48 for the people who used Agile Deliberation and 0.82 for the people who used Manual Deliberation, because each person judged the images by their own standard. The paper says as much: “Since participants varied in the complexity of their understanding for the same concept, the classification performances are not comparable across these two groups.” For the Manual group the zero-shot classifier already agreed with the participants, and the paper concedes that “participants using Manual Deliberation likely had an easy concept understanding, leaving limited room for further deliberation.” The 13-point advantage for Agile Deliberation on Healthy food therefore measures the difference in headroom between two groups as much as the difference between two tools. The Pay-to-Play comparison, 11 points against 8, is the cleaner one, and it is a 3-point difference between two groups of four or five people.

What four or five people can show

Table 1 gives standard deviations of 0.05 to 0.10 for the systems’ F1 scores. At a standard deviation of 0.06 and four or five people, the 95 percent interval around one group’s mean extends about 0.07 to 0.10 on each side, which is more than the 3-point gap in the abstract. The Pay-to-Play final F1 is the closest thing to a like-for-like comparison, because the two groups started from similar zero-shot scores (0.48 and 0.43). A Welch test on the printed means and standard deviations gives p of 0.066 for a 5-against-4 split and 0.077 for 4-against-5.

We then simulated studies of the same size. The paper does not publish per-person values or the spread of each person’s gain, so we assumed the two groups’ F1 values have the same standard deviation, 0.05 to 0.10 (the printed range), and asked two questions. How often does a gap of 3 points or more appear when the true gap is zero? And how often would a test call a true 3-point gap significant?

95 percent half-width of one group mean at SD 0.06: n=4 0.095, n=5 0.074
Paid to play final F1, Agile 0.59 (SD .06) vs Manual 0.51 (SD .05), n=5 vs 4: gap 0.08, Welch p = 0.066
Paid to play final F1, Agile 0.59 (SD .06) vs Manual 0.51 (SD .05), n=4 vs 5: gap 0.08, Welch p = 0.077
share of simulated studies (10,000 each), SD of the groups' F1 values assumed equal:
SD 0.050, n=5 vs 4: gap of 3 points or more with no true gap 18.8%; p < .05 with a true 3-point gap 10.1%
SD 0.050, n=4 vs 5: gap of 3 points or more with no true gap 18.6%; p < .05 with a true 3-point gap 10.8%
SD 0.075, n=5 vs 4: gap of 3 points or more with no true gap 27.2%; p < .05 with a true 3-point gap  6.3%
SD 0.075, n=4 vs 5: gap of 3 points or more with no true gap 27.3%; p < .05 with a true 3-point gap  6.7%
SD 0.100, n=5 vs 4: gap of 3 points or more with no true gap 32.7%; p < .05 with a true 3-point gap  4.7%
SD 0.100, n=4 vs 5: gap of 3 points or more with no true gap 32.5%; p < .05 with a true 3-point gap  5.2%

Under those assumptions, a study of this size shows a gap of 3 points or more in 19 to 33 percent of runs with no real difference, and detects a real 3-point difference in 5 to 11 percent of runs. Nothing here says the true gap is zero. It says these numbers cannot tell a 3-point advantage from none, so the abstract’s “more than 3% higher than manual deliberation” is a direction, not a result.

Precision bought with recall

The paper reports that the gains “were mainly driven by improved precision with only minor reductions in recall.” In the Agile Deliberation groups, precision rose from 0.35 to 0.57 on Pay-to-Play and from 0.33 to 0.44 on Healthy food, while recall fell from 0.80 to 0.63 and from 0.99 to 0.84. A recall of 0.63 means the classifier misses about 37 percent of the images the participant labeled in scope, against 20 percent for the zero-shot classifier. F1 weights precision and recall equally. A moderation team that fears missed content more than false alarms might use F2, which weights recall twice as heavily. Computed from the group means, F2 goes from 0.636 to 0.617 on Pay-to-Play and from 0.707 to 0.711 on Healthy food: no real gain either way.

concept       classifier     P     R     F1    F2
Paid to play  zero-shot   0.35  0.80  0.487  0.636
Paid to play  Agile       0.57  0.63  0.599  0.617
Healthy food  zero-shot   0.33  0.99  0.495  0.707
Healthy food  Agile       0.44  0.84  0.578  0.711

We are not claiming F2 is the right yardstick. The point is that the headline gain is a trade, and the paper does not show what the trade costs in the moderation setting that motivates the work.

The transfer test in the supplement

The supplement applies the final definitions to two smaller models, Qwen3-VL-8B and Qwen3-VL-30B-A3B-Instruct, that took no part in the sessions. The authors read the result as showing that definitions from both conditions transfer and that “Agile Deliberation produces definitions that are more sensitive to model capability,” with “the strongest gains appearing on more capable models.” The table recomputes the gains from the supplement’s Table 3, with the Gemini row from the paper’s Table 1.

model              concept        Agile gain  Manual gain  Agile minus Manual  Agile F1 minus Manual F1
Gemini 2.5 Flash   Paid to play         +11           +8                  +3                        +8
Gemini 2.5 Flash   Healthy food         +10           -3                 +13                       -21
Qwen3-VL-8B        Paid to play          +6          +15                  -9                        +1
Qwen3-VL-8B        Healthy food          +5           +0                  +5                       -29
Qwen3-VL-30B-A3B   Paid to play         +10          +12                  -2                        +9
Qwen3-VL-30B-A3B   Healthy food          +5           -5                 +10                       -26

On Pay-to-Play, the one concept where the two groups started from similar ground, the Manual group’s gain is smaller than the Agile group’s on Gemini (8 against 11) but larger on both Qwen models (15 against 6 and 12 against 10). Measured in absolute F1, Agile Deliberation is ahead by 1 point on Qwen3-VL-8B and 9 on the 30B model. With four or five people per group neither direction is settled, and the Healthy food rows again compare groups with different headroom, since the Manual group’s zero-shot F1 was 0.81 to 0.83. The transfer table is better read as a warning than a result: the advantage over Manual Deliberation is not stable across the models tested.

What the survey can carry

Two stars out of six

Each participant rated both systems after using them, so Table 2 holds nine ratings per system. The paper marks items at p < .05 but names neither the test nor the rating scale.

A Welch test on the printed means and standard deviations reproduces the paper’s pattern. Because every participant rated both systems, a paired test is the natural choice, and its answer depends on how closely one person’s two ratings move together, which the paper does not report. The script shows the paired p-values at assumed correlations of 0, 0.5 and 0.8.

item                                Agile Manual  Welch p  paired p at rho 0 / 0.5 / 0.8  star
Effort required                      3.11   4.67    0.023   0.029  0.010  0.003        *
Success articulating the concept     5.56   5.11    0.460   0.468  0.343  0.209        -
Frustration                          1.78   2.33    0.255   0.271  0.137  0.036        -
Insecure, stressed, annoyed          1.67   3.00    0.027   0.035  0.011  0.003        *
Ease of a comprehensive definition   4.11   4.00    0.814   0.815  0.773  0.729        -
Mental demand                        3.22   4.56    0.105   0.117  0.053  0.020        -

The two marked items stay below .05 under every variant. Frustration and mental demand clear .05 only if a person’s ratings of the two systems track each other very closely (correlation 0.8), and the paper’s own marks imply they did not clear it. The blog’s figure caption is accurate: it says the reductions in effort and stress were statistically significant and that mental demand was lower. The next paragraph then moves to “significantly less stressed, less frustrated, and less mentally drained,” and the paper’s own summary says participants “reported significantly lower workload and frustration.” Both go further than Table 2.

Fatigue and anxiety

None of the six items names fatigue or anxiety. The blog’s fatigue claim rests on interview material: manual participants tried an average of 7.3 search queries, often on a narrow kind of ambiguity, and one said “I think it’s hard for me to think of queries that will contradict my definitions because sometimes I don’t know what is the borderline for my definitions.” The anxiety claim rests on a sentence saying that some participants worried their edits “might overfit, making things worse in other images.” The blog turns that into participants who “reported experiencing persistent anxiety.”

All nine participants preferred Agile Deliberation, which is a strong result for a workflow people found easier. It is also a preference stated by nine people recruited within the authors’ organization and paid $100. The paper gives no wording for the preference question, the supplement contains no questionnaire, and we found no statement about what participants were told about which system was the authors’ own. A stated preference for a new tool built by the study’s authors usually needs an independent sample before it carries much weight.

The code release

The blog says the team “open-sourced the underlying code and algorithms on GitHub,” and the README calls the folder “a companion codebase and dataset.” At the commit we read (20b7d38, March 25, 2026) the folder holds a README, a Demo.ipynb notebook, build_index.py, setup.sh, requirements.txt, a .gitignore, a 148-byte indices_train.json, and a library folder of 16 Python files plus a nearest-neighbor subfolder of code. There is no study data in it: no participant definitions, labels, session records or test sets.

The demo is a healthy-food scenario whose first task text reads “Imagine you are a moderator for a Reddit community about healthy food.” It retrieves images from LAION-400M through a CLIP ViT-B/32 index that setup.sh builds from the first two embedding shards by default (about 260 MB), and it asks for a Gemini API key. The paper says the study retrieved from “a proprietary domain-specific 100M image dataset for paid to play and the WebLI dataset.” The released code shows the method but cannot regenerate the study’s retrieval pool or its numbers. We did not run it, because it needs a Gemini key and the index. The blog’s acknowledgments add that the work was “made possible through a close partnership between Google Research and Google Ads.”

We did find one small mismatch between the paper and the code. Algorithm 1 in the supplement clusters the ambiguity descriptions with DBSCAN using a minimum cluster size of 5 and a radius that grows from 0.2 to 0.8 in steps of 0.01. In reflection.py the loop starts at 0.2 in steps of 0.01 and a comment says “Set a maximum eps value to 0.8 to avoid overclustering,” but the default max_eps is 1 and the only call we found, in diverse_sample.py, passes max_eps=1. The cluster size matches: deliberators.py sets active_learning_batch to 5. We could not test what the difference does.

What to take from it if you build classifiers

Google’s blog post is the document most practitioners will read, and its verbs are the ones they will repeat. The paper supports a narrower and still useful message. The workflow takes less effort and feels less stressful than doing the same job by hand, at least for the nine people studied, and the idea of surfacing borderline images for the user to rule on is a sound one.

What the evidence supports

• Lower effort and lower stress for Agile Deliberation than for the manual workflow, each marked p < .05 with nine ratings per system.
• Higher F1 than a zero-shot classifier in the two Agile Deliberation groups (11 and 10 points), obtained by trading recall for precision.
• A plausible mechanism: people explore more kinds of ambiguity when the system chooses the images for them, as the interviews describe.

What would settle the rest

• A larger crossover study with a pre-registered test, in which several people define the same concept with each system, and publication of per-person paired differences.
• Recall at a fixed precision, or a precision-recall curve, for moderation use, not F1 alone.
• A shared labeled test set for each concept in addition to each person’s own labels, so the groups can be compared directly.
• Ablations of image retrieval against prompt optimization. The supplement says the evaluation “does not isolate the contributions of individual components.”
• Released study data, or a replication on the public LAION-400M index that the demo already uses.

How we checked this

We read the blog post (text and figure captions), the CVPR paper, the supplement and the code at the pinned commit. The tables in the script were typed from the rendered PDF pages, and the four printed gains in Table 1 reproduce exactly, which checks the transcription. The p-value routine is a short incomplete-beta implementation that agrees with SciPy 1.18.1 to within 1e-12 on a grid of 56 inputs, and its Welch result for the effort item (0.023) matches SciPy’s. The script uses only the Python standard library and was run on Python 3.13.14.

python agile_check.py

# agile_check.py
"""Recompute the numbers behind Google's Agile Deliberation blog post from the printed tables.

Table 1 and Table 2 come from the CVPR 2026 paper, Table 3 from its supplement.
Every Table 1 and Table 3 cell is (F1, precision, recall) averaged over participants.
Standard library only."""
import math
import random

T1 = {  # Table 1: classifiers built on Gemini 2.5 Flash
    "Paid to play": {
        "Agile group": {"zero": (0.48, 0.35, 0.80), "collab": (0.53, 0.47, 0.63), "system": (0.59, 0.57, 0.63)},
        "Manual group": {"zero": (0.43, 0.29, 0.84), "collab": (0.47, 0.40, 0.58), "system": (0.51, 0.42, 0.68)},
    },
    "Healthy food": {
        "Agile group": {"zero": (0.48, 0.33, 0.99), "collab": (0.50, 0.36, 0.91), "system": (0.58, 0.44, 0.84)},
        "Manual group": {"zero": (0.82, 0.73, 0.92), "collab": (0.78, 0.77, 0.79), "system": (0.79, 0.80, 0.79)},
    },
}
PRINTED = {("Paid to play", "Agile group"): 11, ("Paid to play", "Manual group"): 8,
           ("Healthy food", "Agile group"): 10, ("Healthy food", "Manual group"): -3}
T2 = [  # item, Agile mean, Agile SD, Manual mean, Manual SD, starred (p < .05) in the paper
    ("Effort required", 3.11, 1.62, 4.67, 0.71, True),
    ("Success articulating the concept", 5.56, 0.88, 5.11, 1.54, False),
    ("Frustration", 1.78, 1.09, 2.33, 0.87, False),
    ("Insecure, stressed, annoyed", 1.67, 0.71, 3.00, 1.41, True),
    ("Ease of a comprehensive definition", 4.11, 1.27, 4.00, 0.50, False),
    ("Mental demand", 3.22, 2.05, 4.56, 1.01, False),
]
T3 = {  # supplement Table 3: (Agile zero-shot, Agile system), (Manual zero-shot, Manual system) F1
    "Gemini 2.5 Flash": {"Paid to play": ((0.48, 0.59), (0.43, 0.51)), "Healthy food": ((0.48, 0.58), (0.82, 0.79))},
    "Qwen3-VL-8B": {"Paid to play": ((0.46, 0.52), (0.36, 0.51)), "Healthy food": ((0.47, 0.52), (0.81, 0.81))},
    "Qwen3-VL-30B-A3B": {"Paid to play": ((0.49, 0.59), (0.38, 0.50)), "Healthy food": ((0.47, 0.52), (0.83, 0.78))},
}


def pts(x):
    return round(100 * x)


def fbeta(p, r, beta):
    return (1 + beta * beta) * p * r / (beta * beta * p + r)


def betacf(a, b, x):  # continued fraction for the incomplete beta function (modified Lentz)
    tiny = 1e-300
    qab, qap, qam = a + b, a + 1.0, a - 1.0
    c, d = 1.0, 1.0 - qab * x / qap
    d = 1.0 / (d if abs(d) > tiny else tiny)
    h = d
    for m in range(1, 300):
        m2 = 2 * m
        for aa in (m * (b - m) * x / ((qam + m2) * (a + m2)), -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2))):
            d = 1.0 + aa * d
            d = 1.0 / (d if abs(d) > tiny else tiny)
            c = 1.0 + aa / c
            c = c if abs(c) > tiny else tiny
            h *= d * c
        if abs(d * c - 1.0) < 3e-12:
            break
    return h


def betai(a, b, x):  # regularized incomplete beta function
    if x <= 0.0 or x >= 1.0:
        return 0.0 if x <= 0.0 else 1.0
    front = math.exp(math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b) + a * math.log(x) + b * math.log1p(-x))
    if x < (a + 1.0) / (a + b + 2.0):
        return front * betacf(a, b, x) / a
    return 1.0 - front * betacf(b, a, 1.0 - x) / b


def p_two_sided(t, df):  # two-sided p-value of Student's t
    return betai(df / 2.0, 0.5, df / (df + t * t))


def welch(m1, v1, n1, m2, v2, n2):  # means, variances, sizes -> (difference, t, df)
    se2 = v1 / n1 + v2 / n2
    df = se2 ** 2 / ((v1 / n1) ** 2 / (n1 - 1) + (v2 / n2) ** 2 / (n2 - 1))
    return m1 - m2, (m1 - m2) / math.sqrt(se2), df


def t_crit(df):  # two-sided 5 percent critical value by bisection
    lo, hi = 0.0, 50.0
    for _ in range(80):
        mid = (lo + hi) / 2
        lo, hi = (mid, hi) if p_two_sided(mid, df) > 0.05 else (lo, mid)
    return lo


print("=== 1. Table 1: gain over each group's own zero-shot classifier, in F1 points")
for concept, groups in T1.items():
    for group, rows in groups.items():
        d = pts(rows["system"][0] - rows["zero"][0])
        flag = "matches" if d == PRINTED[(concept, group)] else "DIFFERS"
        print(f"{concept:13s} {group:13s} F1 {rows['zero'][0]:.2f} -> {rows['system'][0]:.2f} = {d:+d}, printed {PRINTED[(concept, group)]:+d} ({flag})")
over_zero = [pts(T1[c]["Agile group"]["system"][0] - T1[c]["Agile group"]["zero"][0]) for c in T1]
over_collab = [pts(T1[c]["Agile group"]["system"][0] - T1[c]["Agile group"]["collab"][0]) for c in T1]
print(f"Agile over zero-shot, mean of two concepts: {sum(over_zero) / 2} points (paper: 10.5)")
print(f"Agile over Modeling Collaborator, mean of two concepts: {sum(over_collab) / 2} points (paper: 7 in Section 4.1, 7.5 in the abstract)")
for c in T1:
    a = pts(T1[c]["Agile group"]["system"][0] - T1[c]["Agile group"]["zero"][0])
    m = pts(T1[c]["Manual group"]["system"][0] - T1[c]["Manual group"]["zero"][0])
    print(f"{c}: Agile gain minus Manual gain = {a - m:+d} points (different participants in each group)")

print()
print("=== 2. Precision for recall: F1 against a recall-weighted F2, from the Agile-group means")
print("concept       classifier     P     R     F1    F2")
for c in T1:
    for label, key in (("zero-shot", "zero"), ("Agile", "system")):
        f1, p, r = T1[c]["Agile group"][key]
        print(f"{c:13s} {label:10s} {p:5.2f} {r:5.2f} {fbeta(p, r, 1):6.3f} {fbeta(p, r, 2):6.3f}")

print()
print("=== 3. Table 2: which ratings clear p < .05 (9 participants, 9 ratings per system)")
print("item                                Agile Manual  Welch p  paired p at rho 0 / 0.5 / 0.8  star")
for item, ma, sa, mm, sm, star in T2:
    _, t, df = welch(ma, sa * sa, 9, mm, sm * sm, 9)
    paired = []
    for rho in (0.0, 0.5, 0.8):
        sd = math.sqrt(sa * sa + sm * sm - 2 * rho * sa * sm)
        paired.append(p_two_sided((ma - mm) / (sd / 3.0), 8))
    print(f"{item:35s} {ma:5.2f} {mm:6.2f}  {p_two_sided(t, df):7.3f}  {paired[0]:6.3f} {paired[1]:6.3f} {paired[2]:6.3f}        {'*' if star else '-'}")

print()
print("=== 4. What groups of 4 and 5 participants can show")
print(f"95 percent half-width of one group mean at SD 0.06: n=4 {t_crit(3) * 0.06 / 2:.3f}, n=5 {t_crit(4) * 0.06 / math.sqrt(5):.3f}")
for na, nm in ((5, 4), (4, 5)):
    d, t, df = welch(0.59, 0.06 ** 2, na, 0.51, 0.05 ** 2, nm)
    print(f"Paid to play final F1, Agile 0.59 (SD .06) vs Manual 0.51 (SD .05), n={na} vs {nm}: gap {d:.2f}, Welch p = {p_two_sided(t, df):.3f}")
rng = random.Random(1)
TRIALS = 10000
print("share of simulated studies (10,000 each), SD of the groups' F1 values assumed equal:")
for sd in (0.05, 0.075, 0.10):
    for na, nm in ((5, 4), (4, 5)):
        chance = power = 0
        for _ in range(TRIALS):
            xs = [rng.gauss(0.0, sd) for _ in range(na)]
            ys = [rng.gauss(0.0, sd) for _ in range(nm)]
            if sum(xs) / na - sum(ys) / nm >= 0.03:
                chance += 1
            xs = [rng.gauss(0.03, sd) for _ in range(na)]
            ys = [rng.gauss(0.0, sd) for _ in range(nm)]
            ma, mm = sum(xs) / na, sum(ys) / nm
            va = sum((x - ma) ** 2 for x in xs) / (na - 1)
            vm = sum((y - mm) ** 2 for y in ys) / (nm - 1)
            d, t, df = welch(ma, va, na, mm, vm, nm)
            if d > 0 and p_two_sided(t, df) < 0.05:
                power += 1
        print(f"SD {sd:.3f}, n={na} vs {nm}: gap of 3 points or more with no true gap {chance / TRIALS:5.1%}; p < .05 with a true 3-point gap {power / TRIALS:5.1%}")

print()
print("=== 5. Supplement Table 3: the same definitions applied by other models (F1 points)")
print("model              concept        Agile gain  Manual gain  Agile minus Manual  Agile F1 minus Manual F1")
for model, concepts in T3.items():
    for concept, ((az, asys), (mz, msys)) in concepts.items():
        ga, gm = pts(asys - az), pts(msys - mz)
        print(f"{model:18s} {concept:13s} {ga:+10d} {gm:+12d} {ga - gm:+19d} {pts(asys - msys):+25d}")

The simulation seed is fixed, so the percentages repeat. The per-person values, the group sizes and the test behind the stars are the data that would turn these approximations into answers, and neither the paper nor the repository publishes them. We have not contacted the authors.

Sources

• Google Research blog post, November 4, 2026
• Agile Deliberation: Concept Deliberation for Subjective Visual Classification, CVPR 2026 open-access PDF
• CVPR 2026 supplement
• arXiv 2512.10821, submitted December 11, 2025 and last revised April 4, 2026
• agile_deliberation folder in google-research/google-research at commit 20b7d38, and its Demo.ipynb