Ai2’s FairStress Turns Fairness Testing Into a Pressure Test, but the Released Files Do Not Match the Report

Many fairness benchmarks ask one question: did the model pick the same candidate when only the names changed? On October 20, Ai2 published FairStress, which adds a second one. Once a model has made the right call, how easily does one added sentence talk it out of that call, and does the answer depend on which group the sentence favors? The release comes with a blog post, a tech report titled “Asymmetrically Unfair Decision Making and Reasoning in LLMs” (by Pardis Sadat Zahraei, Faeze Brahman, Dilek Hakkani-Tür, Gokhan Tur and Maarten Sap), a dataset under CC BY 4.0 and code under the MIT License. The blog post says the benchmark is available for download and is also being built into Ai2’s open evaluation workbench, olmo-eval.

The report’s headline, in its own words: “With explicit identity, accuracy differs by only 2.9 points between groups, yet one added sentence makes 15 models abandon correct answers favoring the majority 9.3 points more often, and flip ties toward the minority 23.9 points more often.” In plainer terms, when the right answer favors the majority-coded candidate, a sentence arguing for the minority-coded one pries the model loose 9.3 points more often than the mirror-image sentence does when the right answer favors the minority-coded candidate.

We checked what can be checked from the public artifacts. The headline means recompute from the per-model numbers in the code repository. The released files are a different story. On October 20, within a day of the upload, the Ai2 dataset repo failed to load with the Hugging Face datasets library, the released Core split did not match the report’s description of Core, the full split had 53,921 fewer rows than the report counts, and the scoring script used different metric names and the opposite sign for the report’s Robustness Gap. We ran no language model, and we found no model responses in the repository or the dataset, so nothing here tests the report’s behavioral findings. It tests whether the released files let someone else test them.

What FairStress measures

FairStress has 300 two-candidate decisions in 29 domains such as hiring, lending, healthcare, housing and criminal justice. Scenarios come in two forms: a ground-truth form in which one profile is objectively stronger, and an ambiguous form in which the two are equal. Six narrative and personal-statement scenarios have only the ambiguous form, so 294 have both. Each scenario is rendered with 33 demographic pairings (26 minority versus majority and 7 minority versus minority), four degrees of identity signal, both candidate orders and, optionally, one adversarial prefix: a sentence placed after the two profiles that argues for one candidate.

The four degrees run from D0, no identity, through D1, a demographically coded name, and D2, the name plus one lifestyle sentence, to D3, the name plus an explicit label. There are 78 prefixes in 15 families. Nine families are directional (authority, social consensus, representation, historical guilt, synthetic evidence, emotional appeal, legal liability, synthetic statistics and disadvantage claims), each with a mirror twin that argues for the other candidate. The rest are neutral or control sentences such as evaluation awareness and a placebo.

Four metrics, two of them under pressure

The first two rows are output-level checks. The last two are the new part, and both are conditional: the report’s Table 1 notes that “Robustness Gap and Tie-Break Shift condition on the model’s answer without the prefix.” On clear-cut cases the model must first be right in both identity-swapped versions, so the Robustness Gap measures abandoning a correct answer, not a baseline mistake. For every metric, values above the fair value (above 50 percent for the lean) mean overcorrecting toward the minority-coded candidate, and values below it mean stereotypical behavior. The blog post calls the pattern “anti-stereotypical over-correction,” and it states the limits of the definition plainly: “FairStress does not attempt to capture every conception of fairness.” The report’s ethics statement adds that under equity-based views “part of the lean we measure could be intended rather than unfair.”

Who wrote the scenarios, and how they were checked

Scenarios and prefixes are model-written. The report says “We used gpt-4o-mini to generate the scenarios under a fixed prompt,” a gpt-4o judge validated them, and “all cases were further validated by humans.” The 78 prefixes were also generated with gpt-4o-mini. Three annotators then judged 100 clear-cut and 100 tied pairs with names and tags removed. The majority picked the intended stronger profile on 94 percent of the clear-cut pairs and called a tie on 90 percent of the tied pairs. Agreement was far higher on the first than the second: Fleiss’ kappa was 0.938 on clear-cut pairs and 0.504 on ties, and only 81 percent of tied pairs got a unanimous tie verdict. The report’s own summary is that “agreement is near-perfect on which profile is stronger and moderate on whether a pair is tied.”

That matters because the largest headline number, the 23.9-point Tie-Break Shift, rests on the tied items. The mirrored design helps, since, in the report’s words, “we can still measure how equally the same behavior happens in both directions,” but the tie labels are the softest part of the dataset. For how kappa works and how to check an automated judge against human labels, see our tutorial on calibrating an LLM judge with Cohen’s kappa.

The headline numbers recompute

The repository ships the data behind the report’s Figure 2: per-model values for all four metrics at each degree, for the 15 models run on the full corpus. Averaging them reproduces the report and the blog post.

Our recomputation from the per-model rows gives 6.91 and 9.29 for the Robustness Gap at D2 and D3 and 23.93 for the Tie-Break Shift at D3, so the blog post’s “6.9 percentage points with implied identity and 9.3 points with explicit labels” holds. Four details in the per-model numbers are worth knowing before anyone quotes the 9.3.

• The spread is wide. At D3 the Robustness Gap runs from +1.1 (Gemma-4-31B) to +15.7 (the DPO checkpoint of Tulu-3-8B) and is positive for all 15 models. At D2 one model, Tulu-3.1, is negative (-0.7), so 14 of 15 are positive there, and the report flags it and Qwen2.5-7B for low movement (any-flip rates of 4.31 and 4.45 percent), so their Robustness Gaps rest on fewer flips. Pooled over all four degrees, the report’s Table 18 rows average about +4.6 (our arithmetic), so 9.3 is the explicit-label case.
• The roster leans on Ai2’s own models. Ten of the 15 are checkpoints from the OLMo and Tulu lines (SFT, DPO and final versions of OLMo-3-7B-Instruct, OLMo-3.1-32B-Instruct and Tulu-3-8B, plus Tulu-3.1-8B), so they are related to one another. The other five are Qwen2.5-7B, Llama-3.1-8B, Gemma-3-27B, Gemma-4-31B and GPT-4o-mini. At D3 the Ai2 checkpoints average +10.25 on the Robustness Gap and the other five average +7.38, while the Tie-Break Shift is nearly identical at 23.82 and 24.14.
• The controls are not zero. On the seven minority-versus-minority contrasts, which the report uses as a control, the D3 values are -0.6, 47.2, -2.3 and -5.1 for the four metrics. The neutral and control prefixes also move ties: “An eval-awareness prefix shifts a tied decision toward the minority candidate by +5.3 points at D3 and even a placebo sentence shifts it by +4.2.” Directional prefixes push the shift to between +19 and +34, so most of the effect is directional, but a tie moves a little even when the added sentence argues for nobody.
• The strongest prefix makes no group claim. At D3 the largest Robustness Gaps come from the synthetic-evidence family (+16.0) and the historical-guilt family (+15.5), while authority, social consensus and disadvantage claims land between +4.8 and +5.8. The report argues this is not a plausibility artifact: “synthetic evidence (F6), which makes no claim about any group, has the largest gap.”

What a zero at D0 means

At D0 the prompt carries no identity text, so the contrast label is only metadata. In the released Core split, 12,426 D0 rows hold only 7,033 distinct prompts, and one prompt appears 13 times under 13 different contrast labels. At D1, D2 and D3 every row was distinct in both samples we read. The report notes that at D0 “no model shows a gap,” which is what a balanced labeling of identical prompts would produce. It works as a negative control on the pipeline. It is not evidence about how a model treats people when identity is hidden.

The frontier-model result is about ties

Six frontier models, each in answer and reasoning mode, were run on the Core subset (GPT-4o-mini was also run on the full corpus). The report’s Table 31 gives three numbers for those 12 runs: an Accuracy Gap between -1.2 and +1.5, a Tie-Break Lean between 60.0 and 65.3 percent, and a Tie-Break Shift between +8.8 and +42.4. The table has no Robustness Gap for them, the main text reports only the lean and the shift, and the report says that “Per-family and per-degree cells on FairStress-Core are too small for the frontier models to be compared family by family.” So the blog post’s statement that frontier models behave similarly rests on ties. Whether they also abandon correct answers asymmetrically is not in the table. The nearly fivefold spread in the shift (8.8 to 42.4) also means that “positive for all 12 runs” is the robust statement, not any single size. Two of the 12 runs, both Claude Opus 5, also crossed the report’s own refusal flag of 5 percent (8.49 percent in answer mode and 5.21 percent in reasoning mode). Refusals are kept as a separate outcome, and a pair enters the flip calculation only when both responses resolve to a candidate.

What the fixes do, and what they cost

The report tests nine post-hoc debiasing methods on three open models (OLMo-3-7B, Gemma-3-12B and Qwen2.5-7B) using Core, with settings tuned on a separate 1,199-item calibration set. The blog post summarizes: “Most methods did not reliably reduce the asymmetry, and some produced more balanced choices at a substantial cost to accuracy.” The table behind that sentence has a few specifics worth keeping:

• Counterfactual chain-of-thought, which asks the model to reconsider the case with identities swapped, brings the Robustness Gap to between +0.3 and +2.0 points while clear-cut accuracy stays at 0.97 to 0.99. The report also lists a capability cost (MMLU down 8.2 points on OLMo-3), and the lean on ties stays at 54 to 57 percent.
• One activation-steering method (CAA) cuts the Robustness Gap but drops Qwen2.5-7B’s clear-cut accuracy from 0.98 to 0.47, and Gemma-3-12B’s from 0.98 to 0.48.
• Self-critique made the Robustness Gap larger on OLMo-3-7B (+7.4 to +9.1) and Qwen2.5-7B (+4.3 to +6.7) and smaller on Gemma-3-12B (+4.4 to +1.9).
• The Tie-Break Shift intervals are wide. For counterfactual chain-of-thought on OLMo-3-7B the shift reads +1.3 with a 95 percent interval of -5.4 to 8.0, against +7.2 (1.0 to 13.5) with no intervention. The intervals come from resampling the 300 scenarios 2,000 times.

The report also tests five 3B models pretrained with different amounts of constitution-guided text. In the strongest recipe the D3 Accuracy Gap falls from +14.6 to +4.5 and the mean Robustness Gap from +14.7 to +1.7, but those models pick the first-listed candidate on 98 to 99.9 percent of ties, so the authors call it promising rather than a solution.

What the 13 million items are, and are not

The blog post says the variations expand “FairStress’ 300 scenarios into more than 13 million test cases.” The report counts 13,032,104 items and is explicit about what that means: “The 13,032,104 items come from 300 underlying scenarios, so individual renderings are not independent.” It shows the effect for OLMo-3-7B-Instruct at D3: the 95 percent interval for the Accuracy Gap is [3.00, 3.88] points under an item-level bootstrap and [2.47, 4.49] once resampling is done by scenario. The released split has 12,978,183 rows, about 43,000 per scenario.

Most of the corpus is not the pressure condition. The report says “FairStress is mostly control, placebo, and neutral sentences.” We read the metadata columns of all 46 released parquet files (no prompt text was downloaded) and counted:

The structure is regular. The family totals divide evenly by the 278,248 no-prefix rows, which implies that each scaffold gets exactly 18 placebo, 10 evaluation-awareness, 5 reactance, 4 irrelevant-control, 3 stakes-framing and 2 relevant-control renderings (42 in all, as the report describes) plus 3.6 directional ones on average. The rows that carry the Robustness Gap pressure, clear-cut cases with a directional prefix arguing for the weaker candidate, number 250,304, or 1.9 percent of the split, and 70,086 of them are at D3. Tied cases with a directional prefix, the Tie-Break Shift cell, number 513,117 (4.0 percent). Separately, 3,656,414 rows (28.2 percent) sit at D0, where the prompt has no identity text. In one shard of 282,135 rows, which holds a slice of all 300 scenarios, the D0 rows numbered 79,333 and held 44,501 distinct prompts, so the corpus-wide repeat rate should be at least that high. Anyone planning to run the full corpus should de-duplicate prompts before paying for model calls. The 13 million is a bill, not a sample size.

Where the released files and the report part ways

These checks ran on October 20, around 20:00 UTC, within a day of the dataset’s upload to the Ai2 namespace. The explanation may be benign, such as a regenerated corpus or an earlier draft of Core, and we have not contacted the authors. The row-count check below takes a few seconds to run. The other figures come from reading the metadata columns of the same files and the repository’s scoring script.

The Ai2 dataset does not load

The dataset card for allenai/fairstress declares a core split at core_data/*.parquet, but the Core file sits at data/core-00000-of-00001.parquet. With datasets 5.1.0, load_dataset_builder("allenai/fairstress") raised a ValueError:

ValueError: Couldn't infer the same data file format for all splits. Got {'full': ('parquet', {}), 'core': (None, {})}

The Hugging Face datasets-server API reported viewer: false for that repo. The README’s own usage snippet loads PardisSzah/fairstress, a copy under what appears to be the lead author’s own account, and that one resolves (46 files for the full split, one for Core). Both namespaces hold the same row counts:

import duckdb  # pip install duckdb (tested with 1.5.6)

con = duckdb.connect()
con.execute("INSTALL httpfs")
con.execute("LOAD httpfs")
checks = [
    ("allenai", "data/full-*.parquet"),
    ("allenai", "data/core-*.parquet"),
    ("PardisSzah", "data/*.parquet"),
    ("PardisSzah", "core_data/*.parquet"),
]
for repo, pattern in checks:
    sql = f"SELECT count(*) FROM read_parquet('hf://datasets/{repo}/fairstress/{pattern}')"
    n = con.execute(sql).fetchone()[0]
    print(f"{repo:11} {pattern:22} {n:>11,}")

allenai     data/full-*.parquet     12,978,183
allenai     data/core-*.parquet         48,005
PardisSzah  data/*.parquet          12,978,183
PardisSzah  core_data/*.parquet         48,005

DuckDB answers the count from little more than the parquet footers, so the check downloads almost nothing.

Row counts and degree cells

The dataset card says the full split “is the exact data behind the paper’s results.” The report’s Tables 7 and 11 give 13,032,104 items for the full corpus and 50,012 for Core. The released files give 12,978,183 and 48,005. The net 0.41 percent shortfall hides larger offsetting differences by degree, which we computed by grouping the released rows by form and degree:

D2, the only degree that is rendered in a subset of domains, is 12.8 percent smaller than the report’s Table 11, while D0, D1 and D3 are about 2.2 percent larger. The released split is not a sample of the report’s corpus. It looks like a different rendering run.

The Core split

The five coverage counts in the report’s Table 10 (29 domains, 8 contrast categories, 232 domain-by-category cells, 33 contrast pairs, 294 scenarios with both forms) all hold on the released Core. The balance and composition figures do not. The report explains why Core was rebalanced: “A random 50k draw would therefore spend most of its budget on the null condition, so FairStress-Core is rebalanced to 45% directional pressure.” In the released Core only 3.9 percent of rows are directional, 1,867 rows against about 22,500 that a 45 percent share would give, and half the rows (50.1 percent) carry no prefix at all. Measured only against rows that have a prefix, the released directional share is 7.8 percent. The clear-cut rows with a directional prefix arguing for the weaker candidate, the cell behind the Robustness Gap, number 430 across all four degrees and 131 at D3. This matters because Core is what the report uses for the frontier models and the debiasing study, and it is the cheaper split an outside team is most likely to run.

The scoring script

The repository README says scoring/ holds “The four metrics, refusal rates and confidence intervals.” The script, scoring/score.py, describes itself as “Score FIRM benchmark responses,” reads firm_benchmark.jsonl, and computes compute_dag, compute_gsr, compute_fas, compute_iss and compute_dcg. Reading the code:

• The Accuracy Gap (compute_dag, accuracy on minority-favoring minus majority-favoring items) and the Tie-Break Lean (compute_gsr, the mean of the minority-pick rate in each slot) match the report’s formulas.
• The pressure metrics differ. compute_fas, a “Firmness Asymmetry Score,” defines a flip as the no-prefix accuracy for the matching scenario minus the accuracy under the prefix. We did not find the report’s conditioning on a correct answer in both mirrored forms. compute_iss likewise reports the change in minority-selection rate rather than the report’s conditional probability.
• The sign is reversed. The script’s printed flip difference is the flip rate toward the majority minus the flip rate toward the minority, while the report’s Robustness Gap is the flip rate toward the minority minus the flip rate toward the majority.
• The intervals resample items. The only place the script mentions scenarios is the key that matches a prefix row to its no-prefix baseline, so there is no scenario-level resampling like the one behind the report’s Table 4, and the report itself shows item-level intervals are the narrower ones.

We fed compute_fas a synthetic model that is always right without a prefix, flips 10 percent of the time when the weaker candidate is majority-coded and 20 percent when it is minority-coded. The report’s Robustness Gap for that model is about +10 points (+9.9 in the toy data). The script prints a flip difference of -0.099 and a Firmness Asymmetry Score of 0.50, and its own description says a score above 1 means the model defends majority candidates more than minority candidates. The magnitude agrees and the sign does not, so someone who runs the script on an overcorrecting model will see negative numbers where the report prints positive ones. Because the repository ships no model responses (its figures/data folder holds the aggregated numbers behind the figures), the report’s tables cannot be regenerated from public files alone.

None of this shows the report’s numbers are wrong. The headline means recompute from the repository’s own per-model file, and the authors list their limits openly: scenarios that are “synthetic and short,” one-sentence prefixes in a single position, and a training-data audit that “links data to behavior by correlation, not intervention.” It does limit what an outside team can reproduce today.

What teams should take from it

Test whether a decision holds, not only what it is. A deployment that screens applicants, claims or tenants can run a small version of this: take 50 to 100 decisions with a clear answer, render each with swapped, coded names, and add one persuasive sentence for the weaker candidate in both directions. Compare how often the model gives up the right answer for each group. Keep a placebo sentence and a minority-versus-minority pair as controls, because the report’s placebo moved ties by +4.2 points at D3 and the minority-versus-minority pairs stayed near zero.

Count scenarios, not rows. The report’s own scenario-level resampling widened its intervals. Any in-house version should resample by scenario, and it should say how many scenarios stand behind each number.

Decide the standard before reading the sign. FairStress measures equality: similar people should be equally likely to be chosen and equally hard to talk out of a well-supported decision. The report says an equity view could treat part of the lean as intended. A team subject to hiring or lending rules needs its own answer, and the benchmark cannot supply one.

Treat the tooling as an input to verify. Check row counts, composition and sign conventions against the paper before comparing a model with the published tables. Each of the file checks above took minutes and turned up a difference.

Weigh the tie labels. If an in-house test borrows ambiguous cases, validate them with several annotators and report agreement. FairStress’s kappa of 0.504 on ties is a reminder that “equally qualified” is a judgment call.

The bottom line

FairStress puts a useful idea on the table: a model that picks the right candidate for every group can still be easier to sway for one of them. The report’s headline means recompute, its limits are stated in its own text, and its design includes controls a pressure test needs (minority-versus-minority pairs, placebo prefixes, scenario-level intervals). What an outside reader can run today is another matter. The Ai2 dataset needs a config fix, the released Core is not the rebalanced Core the report describes, and the scoring script uses different metric names and the opposite sign. Until those are reconciled, treat the published tables as the authors’ results on their own files, and treat the public files as a starting point for a pressure test of your own. Our earlier look at Singapore’s AI risk guidelines covers a related question, who decides which AI uses get the heaviest independent review.