Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses
Ai2’s XYBench figures reproduce, but the model that judges the answers also drafted every rubric, 77 percent of the “real-world” questions are generated, and the human baseline is graded on rubrics...
Ai2’s Hugging Face post on XYBench (November 13) says that five models, asked a question built on a mistaken approach, tend to help with the mistaken approach instead of the goal behind it, and that human answers do better. Its example is a cupcake: someone asks how to thicken frosting that keeps sliding off, when the real problem is that the cupcakes are still too warm. The paper behind it, arXiv 2609.06842, won the Best Resource Paper award at EMNLP 2026, which the conference’s awards page confirms.
Table Of Content
- What I checked, and what I could not
- The post’s numbers reproduce
- Five things the post does not say
- 1. Three quarters of the “real-world questions” are generated
- 2. The model that scores the answers also drafted the rubrics
- 3. The metric with the lowest judge agreement carries a main result
- 4. The human baseline is scored on rubrics taken from its own answers
- 5. The multiple-choice contrast is not like for like
- The paper and the repository disagree with themselves in small ways
- How precise are the model rankings?
- What survives the audit
- Questions I would put to the authors
- Reproduce the checks
- Step 1: what the script does
- Step 2: the output
- Step 3: confirm the repository facts yourself
- Sources
I audited the post against the paper, the public code and the repository’s commit history. Every figure I could check in the post reproduces. What does not hold up as well is the framing around those numbers: three quarters of the “real-world questions” are written by the same model that scores the answers, the human baseline is scored against rubrics read off its own answers, the metric with the lowest judge agreement carries one of the post’s main results, and the paper and the repository disagree with themselves in several small places. I could not read the dataset itself, which sits behind Hugging Face’s click-through gate, so everything below comes from the post, the paper, the code, the data documentation and the commit log.
What I checked, and what I could not
I read the blog post, the full HTML of the paper including appendices (judge validation in Appendix E, multiple choice in F.2, the oracle experiment in Table 11), the allenai/XYBench repository at main (merge commit 29b58f1, November 15, 05:50 UTC) with its README, docs/DATA.md, the generation, judging, scoring and multiple-choice scripts and the last ten commits, and the Hugging Face API record for the dataset. That record shows the dataset as gated: auto with the license cc-by-nc-sa-3.0, while the code is Apache-2.0. Reading it requires logging in to Hugging Face and accepting the authors’ terms, which I did not do, and rerunning the benchmark needs paid API keys for the judge and the five models. So this is an audit of the published tables, text and code, not a rerun.
The one thing I could do offline is recompute from the paper’s printed Table 2. The script at the end of this article does that with the table typed in; every comparison below that says “my arithmetic” comes from it.
The post’s numbers reproduce
Credit first. The figures the post quotes match the paper, and the paper’s appendices are unusually candid about how the data and the judge were validated. The checks:
| Claim in the post | Source | Result |
|---|---|---|
| 8,115 questions from four sources: 1,557 Stack Overflow, 184 Stack Exchange, 102 manually curated, 6,272 WikiHow | Paper, Sections 3.2 to 3.5 | Sums to 8,115 |
| On Stack Overflow, stated-request solutions appear in 75 to 86 percent of responses, underlying-goal solutions in 59 to 71 percent | Table 2 | Matches |
| On Stack Overflow, models focus on the underlying goal in 35 to 41 percent of responses and on the stated request in 47 to 57 percent | Table 2 | Matches |
| Oracle experiment, Sonnet 4.6 on Stack Overflow: goal solution 59 to 67 percent, misconception identification 47 to 34 percent | Tables 2 and 11 | Matches (0.67 at +14 percent, 0.34 at -28 percent) |
| A two-option choice picks the goal-focused answer in 71 to 84 percent of Stack Overflow cases | Section 4 | Matches the paper’s text |
| Judge agreement with two reviewers: 97.5 to 100 percent on solution presence, 84.6 to 93.3 on misconception identification, 74.2 to 79.8 on emphasis | Appendix E | Matches the pairwise figures |
| Best Resource Paper at EMNLP 2026 | Conference awards page | Confirmed |
| Every model is more likely to include the stated-request solution than the intended one, in all four sources | Table 2 | True in 20 of 20 model-by-source cells (my arithmetic) |
The direction of the headline finding is therefore solid. The questions are about size, comparison and framing.
Five things the post does not say
1. Three quarters of the “real-world questions” are generated
The post’s opening section says XYBench draws on “8,115 real-world questions spanning programming and everyday activities such as cooking and home repair”. Further down it says the benchmark was built from four sources, one of them “6,272 questions generated from WikiHow guides”. That is 77.3 percent of the total. The real or hand-curated part, 1,741 Stack Overflow and Stack Exchange questions plus 102 curated ones, is 22.7 percent (my arithmetic). The cupcake query that opens the post belongs to the generated group: the paper’s Section 3.4 uses it as the worked example of the WikiHow pipeline.
The paper is open about how that group was made. In its words, “We use Claude-Sonnet-4.5, for data generation as well as filtering and rewriting stages.” Two authors then annotated 400 of the generated instances: per-dimension accuracy of the pipeline ran from 74.9 to 97.0 percent, and 68.5 percent of instances were fully correct across all fields. For the Stack Overflow-derived data the equivalent check covered 100 instances and found 84.0 percent fully correct. The post mentions validation only for the judge. The limitations section adds that the authors “lack human references for the synthetically constructed queries based on WikiHow”, so the largest subset has no human row to compare against at all.
2. The model that scores the answers also drafted the rubrics
For Stack Overflow, Stack Exchange and the curated subset, the stated and intended requests and the three rubrics (what a literal answer, a misconception explanation and a better alternative should contain) were extracted by an LLM. The paper says: “Both passes use Claude-Sonnet-4.5, a strong reasoning LLM.” For WikiHow the same model wrote the queries and rubrics. The post says what happens next: “In XYBench, Claude Sonnet 4.5 serves as the automated judge.” The repository fixes the judge as anthropic/claude-sonnet-4-5-20250929, with a comment that the call settings used for the paper were “no extended thinking, temperature 0.7”.
The authors know about this kind of overlap. When they filtered Stack Overflow queries, they used GPT-5 “to prevent bias from a single model family influencing both data construction and filtering”. The scoring step has no such separation, and one of the five models under test, Claude Sonnet 4.6, comes from the judge’s own family. The paper’s robustness check tested two OpenAI models, gpt-4.1 and gpt-5-mini, as alternate judges on 250 query-model pairs (150 WikiHow, 100 Stack Overflow). Agreement with the Claude judge was Cohen’s kappa 0.31 to 0.76, and with the human annotations 0.38 to 0.87. The conclusion drawn is that the evaluation is “fairly robust to the choice of judge, with Claude-Sonnet-4.5 agreeing the best with human annotations.” That is a sample-level agreement check. It does not rescore Table 2 with another judge, and it does not use GPT-5, the filter model, as a judge.
One pattern in Table 2 is worth a look, with the caveat that it proves nothing alone. Sonnet 4.6’s rank among the five models on misconception identification is last on Stack Overflow (0.47) and Stack Exchange (0.46), tied for first on the curated subset (0.52) and first on WikiHow (0.68, against 0.52 to 0.65 for the others). The WikiHow subset uses a different misconception prompt and label set (mistake_practice instead of request_mismatch in the data documentation), so the swing may come from the task, not the family. The paper gives no per-model judge error, so a family effect can be neither ruled in nor ruled out from the published material.
3. The metric with the lowest judge agreement carries a main result
The post reports the judge’s pairwise agreement with two reviewers on emphasis: 74.2 to 79.8 percent. Appendix E also reports Krippendorff’s alpha, which corrects for chance: 0.587 to 0.600 for emphasis and 0.634 to 0.866 for misconception identification. The same appendix says the two annotators agree with each other perfectly on emphasis (“solution presence and emphasis have perfect agreement between the annotators”), and then calls the judge-to-human agreement “commensurate” with the human-to-human figure. One hundred percent against 74 to 80 percent is not commensurate. By my arithmetic the judge’s emphasis label differs from the annotators’ shared label on 20.2 to 25.8 percent of responses.
The validation set was 500 responses, and the post does not say how they split: 100 from Stack Overflow and 400 from WikiHow. None came from Stack Exchange or the curated subset, which are two of the three subsets with human references. The post’s emphasis result, models focusing on the underlying goal in 35 to 41 percent of Stack Overflow responses, spans only 6 points across the five models, against a judge disagreement rate of 20 to 26 percent. Judge errors need not fall evenly on all five models, which is exactly why a per-model error breakdown would settle the matter. The paper does not give one.
4. The human baseline is scored on rubrics taken from its own answers
For Stack Overflow and Stack Exchange, the paper keeps the top three highest-scored answers plus the accepted answer as “reference” human responses. It then has the LLM “identify and extract the S-request and the I-request based on how it has been answered in reference human answers for that query”, and “re-examine the human reference answers corresponding to the query to identify elements that align with the corresponding three response dimensions”. The curated subset goes through “the same two-stage extraction pipeline”, followed by an author’s edits. So the definition of a cooperative answer was read off the human answers, and the human answers are then graded against it. Models are graded against someone else’s reading of what a good answer contains. The post’s sentence, “Unlike human reference answers, all five models were less likely to steer users away from that approach”, describes a comparison with that built-in advantage and does not say so.
The ceiling is not 100 percent either. On the curated subset, where each human answer is the source of its own rubric, the judge gives it 0.95 on intended-solution presence, 0.82 on intended-goal emphasis and 0.79 on misconception identification. How the human rows were produced is hard to confirm from outside. The paper says that “Human reference answers are evaluated identically.” The data documentation says the human files carry precomputed judge outputs and metrics and “are not re-evaluated”, and that for Stack Overflow and Stack Exchange the misconception flag “comes from the annotation of whether the answer identifies the misconception”, while for the curated set it comes from the judge’s tags. That reads like two different paths. Both statements can be true, and the gated data keeps me from checking.
The composite metrics add a wrinkle the post skips. Table 2’s “fully cooperative” score needs the misconception identified, both solutions present, and balanced or intended-goal emphasis. On Stack Overflow all five models score 0.20 to 0.28 against 0.18 for all human answers, and two of them (GPT-5 at 0.28, DeepSeek-V4 Pro at 0.24) are above the 0.23 of the accepted answers. On Stack Exchange GPT-5 (0.23) is above the 0.20 of all human answers. The large human advantage is on “fully pragmatic”: 0.47 against at most 0.31 on Stack Overflow, 0.57 against 0.23 on Stack Exchange and 0.68 against 0.25 on the curated subset (my arithmetic). Whether models “fall far short” of humans depends on which composite you pick.
5. The multiple-choice contrast is not like for like
The post sets 71 to 84 percent against 35 to 41 percent and concludes: “The results expose a gap between recognizing useful advice and producing it, which a multiple-choice test alone would miss.” Three things make that comparison loose. First, a two-option choice has a 50 percent chance rate, so 71 to 84 percent is 21 to 34 points above chance. Second, the options were not natural answers: Appendix F.2 says they “are generated using Claude-Sonnet-4.5 by providing as input the explicit response dimensions”, rewritten to similar length. Third, the pragmatic option is not purely pragmatic. The paper’s main text calls it “a response focused on misconception identification and I-solution”, but the repository’s multiple-choice script describes it as a hybrid_response that “answers the stated request, identifies the misconception, and addresses the intended request”. An option that contains the literal answer plus a misconception explanation plus a better alternative is a near-superset of the other option, so a preference for it is not a surprising outcome.
On the generation side, the 35 to 41 percent counts only answers whose emphasis is primarily the underlying goal. A hybrid-style answer would likely be labelled balanced or goal-focused, and that share on Stack Overflow is 44 to 53 percent for the five models (emphasis on the intended solution plus balanced, my arithmetic from Table 2). Put next to each other, 71 to 84 percent chosen from two options and 44 to 53 percent generated is still a gap, but a smaller and differently shaped one than the post’s pairing suggests.
A related detail: the table the appendix gives for the oracle experiment, where models are told the stated and intended requests, covers four models, and they are not the five in the headline. Table 11 has Claude Sonnet 4.6, GPT-4.1, GPT-5 and Gemini 3.1 Pro. Qwen3.5-9B and DeepSeek-V4 Pro are absent, and GPT-4.1 is an addition.
The paper and the repository disagree with themselves in small ways
None of these changes the story on its own. Together they say the numbers in prose were not all regenerated from the final tables, and that the released data kept moving after the paper.
| Item | What the text says | What the table or file says |
|---|---|---|
| Stated-solution presence range | Abstract and introduction: 0.75 to 0.92. Section 4: 0.57 to 0.91 | Table 2 minimum 0.57 (Sonnet 4.6, WikiHow), maximum 0.91 (GPT-5, Stack Exchange) |
| Misconception gap to humans | Section 4: models lag 25 to 40 points on Stack Overflow, Stack Exchange and the curated set | 19 points (GPT-5, Stack Overflow) to 42 (Sonnet 4.6, Stack Exchange) against the all-answers row (my arithmetic) |
| Intended-solution presence gap on Stack Overflow and Stack Exchange | Section 4: 3 to 17 points, except GPT-5 on Stack Overflow | GPT-5 on Stack Overflow is 4 points above humans; the others run from 1 to 25 points (my arithmetic) |
| Number of queries | README overview: 8,115 | README data table: 1,557 + 184 + 102 + 6,268 = 8,111 |
| Data license | Paper, Appendix G: “XYBench will be released under CC by 4.0.” | Hugging Face record: cc-by-nc-sa-3.0; the README states a license for the code only (Apache 2.0) |
| “Human (highest score)” row, Stack Overflow | Paper: highest-scored answers are marginally more fully pragmatic than accepted ones (0.47 against 0.45 on Stack Overflow, 0.56 against 0.54 on Stack Exchange) | The Stack Overflow row is identical to “Human (all)” in all eight columns; the repository’s code for it was changed on November 15 |
The last row deserves a closer description. On November 15 at 05:27 UTC, two days after the blog post, commit 808f636 arrived with the message “fix the code for highest score computation. updating data doc for fixed SE file.” In compute_results.py it changed the condition that puts an answer into the “highest score” bucket from is_highest_score or is_high_score to is_highest_score alone, and in docs/DATA.md it removed a note that is_highest_score is null for Stack Exchange. The old condition is consistent with a Stack Overflow “highest” row that equals the “all” row, and, if the old note was accurate, with a Stack Exchange row that could only come from the is_high_score fallback. I cannot see the corrected values without the data. What I can say is that the repository’s current code no longer computes the row the paper prints, and that the paper’s sentence about highest-scored versus accepted answers rests on it.
The timeline matters too. The repository was created on August 31, the paper is dated September 6, a commit titled “dataset updated” landed on November 4, the post ran on November 13, and the Stack Exchange file was fixed on November 15. A reader who downloads the data today may not get the files behind Table 2. The 6,268 against 6,272 WikiHow count may be a trace of that, though the repository does not say.
How precise are the model rankings?
The paper reports significance for one analysis, the phi coefficients in Table 10, and notes there that the curated subset, at 102 queries, is small enough that two models fall short of significance. It gives no intervals, paired tests or repeated runs for Table 2. A search of compute_results.py, evaluate_responses.py, utils.py and run_mcq.py for bootstrap, stderr, confidence, variance, repeat, majority and vote found nothing. The post describes one response per question, the repository generates at temperature 1.0 by default, and the judge runs at temperature 0.7 with no repeated passes or voting that I could find.
The sampling-only noise is easy to bound. The table below uses the binomial half-width of a proportion near 0.4, treats queries as independent, and ignores the judge’s own error. A paired comparison of two models on the same queries can be tighter than two independent intervals, so read these as rough scales, not tests.
| Subset | Queries | 95% half-width at p = 0.4 | Spread of the five models’ intended-goal emphasis |
|---|---|---|---|
| Stack Overflow | 1,557 | 2.4 points | 6 points |
| Stack Exchange | 184 | 7.1 points | 9 points |
| Manually curated | 102 | 9.5 points | 7 points |
| WikiHow | 6,272 | 1.2 points | 3 points |
On Stack Overflow and WikiHow the differences between models are about two and a half times the sampling noise, though still smaller than the judge’s disagreement rate on emphasis. On Stack Exchange and the curated subset they are about the size of the noise, so most pairwise rankings there are not resolved. The differences between models and humans are much larger and survive this check; they carry the other caveats above.
What survives the audit
The finding that models mention the literal solution more often than the underlying one holds in every cell of Table 2, and on Stack Overflow the gap (75 to 86 against 59 to 71 percent) is many times the sampling noise. The oracle experiment is, to my reading, the most interesting result in the paper, and it is more nuanced than the post’s summary. On WikiHow, where no human reference exists, telling the model the goal lifts intended-goal emphasis sharply (Sonnet 4.6 from 0.32 to 0.77, GPT-5 from 0.30 to 0.60), so models can act on a stated goal. On Stack Overflow the same hint lowers misconception identification for three of the four models tested (Sonnet 4.6, GPT-4.1 and GPT-5), and the paper’s explanation, memorization of Stack Overflow, is a hypothesis it does not test.
Two scope notes for anyone citing the result. The evaluated models are Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro; the navigation of OpenAI’s ChatGPT Wallet announcement, read on November 15, lists GPT-6.1 Sol, GPT-6 Astra, GPT-5.6 and GPT-5.5 under “Latest Advancements”, and none of those was tested. And the Hub record lists a non-commercial license (cc-by-nc-sa-3.0) where the paper promises CC BY 4.0, so check the terms before using the data in a commercial evaluation pipeline.
Questions I would put to the authors
- Can Table 2 be rescored with a judge outside the Claude family, for example GPT-5, and the model ranking compared?
- What is the judge’s emphasis and misconception error per evaluated model, and on Stack Exchange and the curated subset, which the 500-response validation did not cover?
- Can the human rows be regenerated with the same approach-unit judge used for models, with rubrics built from held-out answers, so the baseline is not graded on its own rubric?
- What are the paired confidence intervals for the model-to-model and model-to-human differences?
- Which files produced the printed Table 2, how many WikiHow queries do they hold (6,272 or 6,268), and what does the “highest score” row look like after the November 15 fix?
- Which license applies to the data, CC BY 4.0 as the paper says or the CC BY-NC-SA 3.0 on the Hub record?
Reproduce the checks
Step 1: what the script does
The script below holds the eight metric columns of the paper’s Table 2 for each model and human row, typed from the HTML version of arXiv 2609.06842v1. It uses only the Python standard library. It recomputes the query counts, the cell-by-cell comparison, the ranges, the human gaps, the composite scores, Sonnet 4.6’s rank, the multiple-choice contrast, the sampling half-widths and the judge disagreement, and it checks whether the Stack Overflow “highest” row equals the “all” row. Save it as xy_check.py and run python xy_check.py with Python 3.13 or later.
"""xy_check.py: recompute what can be recomputed from XYBench's printed Table 2 (arXiv 2609.06842v1).
Values are typed from Table 2 of the paper, order per row:
S-solution present, I-solution present, emphasis S, emphasis balanced, emphasis I,
misconception identified, fully pragmatic, fully cooperative.
Standard library only (Python 3.13)."""
from math import sqrt
COLS = ["s_pres", "i_pres", "emph_s", "emph_b", "emph_i", "miscon", "fully_prag", "fully_coop"]
T2 = {
"Stack Overflow": {
"Sonnet 4.6": [.82, .59, .57, .08, .36, .47, .25, .20],
"GPT-5": [.83, .71, .47, .12, .41, .60, .31, .28],
"Gemini 3.1 Pro": [.79, .62, .51, .08, .40, .54, .29, .22],
"Qwen3.5-9B": [.75, .64, .53, .06, .41, .60, .29, .21],
"DeepSeek-V4 Pro": [.86, .66, .57, .09, .35, .55, .25, .24],
"Human (all)": [.58, .67, .39, .06, .55, .79, .47, .18],
"Human (accepted)": [.66, .67, .40, .08, .52, .81, .45, .23],
"Human (highest)": [.58, .67, .39, .06, .55, .79, .47, .18],
},
"Stack Exchange": {
"Sonnet 4.6": [.75, .46, .69, .04, .27, .46, .15, .12],
"GPT-5": [.91, .64, .64, .10, .26, .63, .22, .23],
"Gemini 3.1 Pro": [.80, .49, .62, .06, .32, .53, .23, .18],
"Qwen3.5-9B": [.81, .62, .66, .06, .29, .59, .23, .17],
"DeepSeek-V4 Pro": [.89, .54, .69, .08, .23, .50, .19, .19],
"Human (all)": [.53, .71, .37, .03, .60, .88, .57, .20],
"Human (accepted)": [.67, .77, .39, .04, .57, .90, .54, .26],
"Human (highest)": [.53, .71, .37, .03, .60, .88, .56, .20],
},
"Manually curated": {
"Sonnet 4.6": [.82, .58, .62, .05, .33, .52, .25, .21],
"GPT-5": [.86, .63, .62, .09, .29, .45, .17, .18],
"Gemini 3.1 Pro": [.83, .63, .56, .10, .34, .47, .21, .25],
"Qwen3.5-9B": [.82, .48, .64, .03, .33, .52, .20, .17],
"DeepSeek-V4 Pro": [.82, .57, .61, .03, .36, .45, .24, .19],
"Human": [.46, .95, .07, .11, .82, .79, .68, .33],
},
"WikiHow": {
"Sonnet 4.6": [.57, .33, .66, .02, .32, .68, .15, .08],
"GPT-5": [.82, .55, .67, .02, .30, .52, .16, .13],
"Gemini 3.1 Pro": [.71, .41, .68, .02, .30, .65, .18, .12],
"Qwen3.5-9B": [.61, .37, .70, .01, .29, .61, .15, .08],
"DeepSeek-V4 Pro": [.74, .47, .69, .02, .30, .57, .17, .12],
},
}
MODELS = ["Sonnet 4.6", "GPT-5", "Gemini 3.1 Pro", "Qwen3.5-9B", "DeepSeek-V4 Pro"]
N = {"Stack Overflow": 1557, "Stack Exchange": 184, "Manually curated": 102, "WikiHow": 6272}
def col(src, who, name):
return T2[src][who][COLS.index(name)]
print("== 1. Query counts")
print("paper: 1557 + 184 + 102 + 6272 =", 1557 + 184 + 102 + 6272)
print("repo README table: 1557 + 184 + 102 + 6268 =", 1557 + 184 + 102 + 6268)
print("WikiHow share of 8,115: %.1f%%" % (100 * 6272 / 8115))
print("real or hand-curated: %d, %.1f%% of 8,115" % (1741 + 102, 100 * (1741 + 102) / 8115))
print("filter kept: SO %.1f%% (1557/1780), SE %.1f%% (184/214)" % (100 * 1557 / 1780, 100 * 184 / 214))
print("\n== 2. Stated-request solution present more often than intended-request solution?")
cells = [(s, m) for s in T2 for m in MODELS]
yes = sum(col(s, m, "s_pres") > col(s, m, "i_pres") for s, m in cells)
print("%d of %d model-by-source cells" % (yes, len(cells)))
print("\n== 3. Ranges over all 20 cells (abstract says S 0.75-0.92, I 0.33-0.71; Section 4 says S 0.57-0.91)")
for name in ("s_pres", "i_pres"):
vals = [(col(s, m, name), s, m) for s, m in cells]
lo, hi = min(vals), max(vals)
print("%s min %.2f (%s, %s) max %.2f (%s, %s)" % (name, lo[0], lo[1], lo[2], hi[0], hi[1], hi[2]))
print("\n== 4. Gap to Human (all) in points: misconception identified (paper text: 25-40) and I-solution present on SO and SE (text: 3-17)")
gaps = []
for s in ("Stack Overflow", "Stack Exchange"):
for m in MODELS:
gaps.append((round(100 * (col(s, "Human (all)", "miscon") - col(s, m, "miscon"))), s, m))
for m in MODELS:
gaps.append((round(100 * (col("Manually curated", "Human", "miscon") - col("Manually curated", m, "miscon"))), "Manually curated", m))
print("misconception gap min %d (%s, %s), max %d (%s, %s)" % (min(gaps)[0], min(gaps)[1], min(gaps)[2], max(gaps)[0], max(gaps)[1], max(gaps)[2]))
ig = [(round(100 * (col(s, "Human (all)", "i_pres") - col(s, m, "i_pres"))), s, m) for s in ("Stack Overflow", "Stack Exchange") for m in MODELS]
print("I-solution gap min %d (%s, %s), max %d (%s, %s)" % (min(ig)[0], min(ig)[1], min(ig)[2], max(ig)[0], max(ig)[1], max(ig)[2]))
print("\n== 5. Fully cooperative (needs S and I solution, balanced or I emphasis, misconception identified)")
for s in ("Stack Overflow", "Stack Exchange"):
h_all, h_acc = col(s, "Human (all)", "fully_coop"), col(s, "Human (accepted)", "fully_coop")
ms = {m: col(s, m, "fully_coop") for m in MODELS}
print("%s: humans all %.2f accepted %.2f | models %s" % (s, h_all, h_acc, ", ".join("%s %.2f" % kv for kv in ms.items())))
print(" models at or above Human (all): %s" % [m for m, v in ms.items() if v >= h_all])
for s in ("Stack Overflow", "Stack Exchange", "Manually curated"):
h = col(s, "Human (all)" if s != "Manually curated" else "Human", "fully_prag")
print("%s fully pragmatic: human %.2f, best model %.2f" % (s, h, max(col(s, m, "fully_prag") for m in MODELS)))
print("\n== 6. Sonnet 4.6 rank (1 = best of the five models) on misconception identified")
for s in T2:
order = sorted(MODELS, key=lambda m: -col(s, m, "miscon"))
print("%-17s rank %d (Sonnet %.2f; best %s %.2f)" % (s, order.index("Sonnet 4.6") + 1, col(s, "Sonnet 4.6", "miscon"), order[0], col(s, order[0], "miscon")))
print("\n== 7. Blog contrast: multiple choice vs generated emphasis, Stack Overflow")
print("two-option choice picks the pragmatic answer 71-84% of the time; chance is 50%")
for m in MODELS:
i, b = col("Stack Overflow", m, "emph_i"), col("Stack Overflow", m, "emph_b")
print(" %-16s I-focused %.2f, I-focused or balanced %.2f" % (m, i, i + b))
print("\n== 8. Sampling noise only: 95% half-width of a proportion (one response per query, judge error ignored)")
for s, n in N.items():
hw = 1.96 * sqrt(0.4 * 0.6 / n)
spread_i = max(col(s, m, "emph_i") for m in MODELS) - min(col(s, m, "emph_i") for m in MODELS)
print("%-17s n=%-5d half-width at p=0.4: %.1f points | spread of the five models' I-emphasis: %.0f points" % (s, n, 100 * hw, 100 * spread_i))
print("\n== 9. Judge versus the two annotators (paper appendix, 500 responses: 100 SO + 400 WikiHow)")
for name, lo, hi in (("solution presence", .975, 1.0), ("misconception", .846, .933), ("emphasis", .742, .798)):
print("%-18s agreement %.1f-%.1f%% -> judge disagrees on %.1f-%.1f%% of responses" % (name, 100 * lo, 100 * hi, 100 * (1 - hi), 100 * (1 - lo)))
print("\n== 10. Is the SO 'Human (highest)' row identical to 'Human (all)'?")
print("Stack Overflow:", T2["Stack Overflow"]["Human (highest)"] == T2["Stack Overflow"]["Human (all)"],
"| Stack Exchange:", T2["Stack Exchange"]["Human (highest)"] == T2["Stack Exchange"]["Human (all)"])
Step 2: the output
This is what it prints, which is where every “my arithmetic” figure above comes from.
== 1. Query counts
paper: 1557 + 184 + 102 + 6272 = 8115
repo README table: 1557 + 184 + 102 + 6268 = 8111
WikiHow share of 8,115: 77.3%
real or hand-curated: 1843, 22.7% of 8,115
filter kept: SO 87.5% (1557/1780), SE 86.0% (184/214)
== 2. Stated-request solution present more often than intended-request solution?
20 of 20 model-by-source cells
== 3. Ranges over all 20 cells (abstract says S 0.75-0.92, I 0.33-0.71; Section 4 says S 0.57-0.91)
s_pres min 0.57 (WikiHow, Sonnet 4.6) max 0.91 (Stack Exchange, GPT-5)
i_pres min 0.33 (WikiHow, Sonnet 4.6) max 0.71 (Stack Overflow, GPT-5)
== 4. Gap to Human (all) in points: misconception identified (paper text: 25-40) and I-solution present on SO and SE (text: 3-17)
misconception gap min 19 (Stack Overflow, GPT-5), max 42 (Stack Exchange, Sonnet 4.6)
I-solution gap min -4 (Stack Overflow, GPT-5), max 25 (Stack Exchange, Sonnet 4.6)
== 5. Fully cooperative (needs S and I solution, balanced or I emphasis, misconception identified)
Stack Overflow: humans all 0.18 accepted 0.23 | models Sonnet 4.6 0.20, GPT-5 0.28, Gemini 3.1 Pro 0.22, Qwen3.5-9B 0.21, DeepSeek-V4 Pro 0.24
models at or above Human (all): ['Sonnet 4.6', 'GPT-5', 'Gemini 3.1 Pro', 'Qwen3.5-9B', 'DeepSeek-V4 Pro']
Stack Exchange: humans all 0.20 accepted 0.26 | models Sonnet 4.6 0.12, GPT-5 0.23, Gemini 3.1 Pro 0.18, Qwen3.5-9B 0.17, DeepSeek-V4 Pro 0.19
models at or above Human (all): ['GPT-5']
Stack Overflow fully pragmatic: human 0.47, best model 0.31
Stack Exchange fully pragmatic: human 0.57, best model 0.23
Manually curated fully pragmatic: human 0.68, best model 0.25
== 6. Sonnet 4.6 rank (1 = best of the five models) on misconception identified
Stack Overflow rank 5 (Sonnet 0.47; best GPT-5 0.60)
Stack Exchange rank 5 (Sonnet 0.46; best GPT-5 0.63)
Manually curated rank 1 (Sonnet 0.52; best Sonnet 4.6 0.52)
WikiHow rank 1 (Sonnet 0.68; best Sonnet 4.6 0.68)
== 7. Blog contrast: multiple choice vs generated emphasis, Stack Overflow
two-option choice picks the pragmatic answer 71-84% of the time; chance is 50%
Sonnet 4.6 I-focused 0.36, I-focused or balanced 0.44
GPT-5 I-focused 0.41, I-focused or balanced 0.53
Gemini 3.1 Pro I-focused 0.40, I-focused or balanced 0.48
Qwen3.5-9B I-focused 0.41, I-focused or balanced 0.47
DeepSeek-V4 Pro I-focused 0.35, I-focused or balanced 0.44
== 8. Sampling noise only: 95% half-width of a proportion (one response per query, judge error ignored)
Stack Overflow n=1557 half-width at p=0.4: 2.4 points | spread of the five models' I-emphasis: 6 points
Stack Exchange n=184 half-width at p=0.4: 7.1 points | spread of the five models' I-emphasis: 9 points
Manually curated n=102 half-width at p=0.4: 9.5 points | spread of the five models' I-emphasis: 7 points
WikiHow n=6272 half-width at p=0.4: 1.2 points | spread of the five models' I-emphasis: 3 points
== 9. Judge versus the two annotators (paper appendix, 500 responses: 100 SO + 400 WikiHow)
solution presence agreement 97.5-100.0% -> judge disagrees on 0.0-2.5% of responses
misconception agreement 84.6-93.3% -> judge disagrees on 6.7-15.4% of responses
emphasis agreement 74.2-79.8% -> judge disagrees on 20.2-25.8% of responses
== 10. Is the SO 'Human (highest)' row identical to 'Human (all)'?
Stack Overflow: True | Stack Exchange: False
Step 3: confirm the repository facts yourself
The commit and the data documentation are public. The GitHub commit page for 808f636 shows the one-line change to the “highest score” condition and the removed note in docs/DATA.md. The Hugging Face API URL https://huggingface.co/api/datasets/allenai/XYBench-dataset returns the gated and license fields. The awards page lists the Best Resource Paper entry with the author list.
Sources
- Ai2 on Hugging Face, XYBench: Helping with the problem behind the question (November 13, 2026)
- Yerukola et al., XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions? (arXiv 2609.06842v1, September 6, 2026; HTML version)
- allenai/XYBench repository, docs/DATA.md and commit 808f636
- allenai/XYBench-dataset on Hugging Face (gated)
- EMNLP 2026 awards
- OpenAI, Introducing ChatGPT Wallet (November 13, 2026), for the model list in its navigation







No Comment! Be the first one.