TRENDING
A black manual telephone switchboard with rows of orange jacks and red patch cords plugged into the lower row, above a rotary dial and a handset
November 16, 2026
How to Route Alerts in Alertmanager So Pages Reach the On-Call and Tickets Reach the Queue, and Test Grouping, Inhibition, and Silences Before an Outage Does
A brass nail with a threaded screw section lying on graph paper, a fastener that is both nail and screw
November 16, 2026
Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses
An amber warning beacon on a white surface, a flashing-light stand-in for the FBI’s FLASH alert
November 16, 2026
The FBI’s ARTEX Warning Says a Supply Chain Compromise “Enabled Access” to 2,250 Tenants Where Microsoft Counted Sign-In Attempts and Cites “Over 1,600” Consoles Where Censys Counted 1,465, and a Microsoft Post It Cites Mistypes One Indicator in Its Hunting Query
Two eggs and a colour-changing egg timer in a saucepan of water on an induction hob
November 16, 2026
How to Set gRPC Deadlines and Retries in Python, Pass Them Down a Call Chain, and Stop Retries From Charging a Card Twice
A whole lemon and a halved lemon side by side on a black background
November 16, 2026
IBM’s LLMON Turns Prompt-Injection Defense Into a Token Layout, but the “Nearly Zero” Chart Is One Benchmark, the Paper’s Own Red Team Lets 15% to 20% of Indirect Attacks Through, and the Shipped Template Lets a Document Type the Delimiters
16 Nov 2026
SXZ.io SXZ.io
  • Home
Search the Site
Popular Searches:
Technology Amazon AI
Recent Posts
A manual paper cutter with a wooden grid board, a clear plastic guard and a long cutting arm, a picture of text being trimmed to fit
How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
November 16, 2026
A brown llama and her gray cria walking across a salt flat beside a red lake in Bolivia
llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries
November 16, 2026
A stack of chocolate chip cookies on a white plate beside a cup of tea
NoisyTCP Can Crash a Linux Server Over the Network, but the Fix Has Reached Only Three of the Seven Kernel Lines Still Being Released, Red Hat and Ubuntu Have No Update, and the Researcher’s Own Timings Run From 5 Seconds to 14 Hours
November 16, 2026
SXZ.io SXZ.io
  • Home

Categories

Articles 302 Posts
News 307 Posts
Learning Hub 274 Posts
Home/Articles/Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses
Articles

Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses

Ai2’s XYBench figures reproduce, but the model that judges the answers also drafted every rubric, 77 percent of the “real-world” questions are generated, and the human baseline is graded on rubrics...

November 15, 2026 20 Min Read
17

Ai2’s Hugging Face post on XYBench (November 13) says that five models, asked a question built on a mistaken approach, tend to help with the mistaken approach instead of the goal behind it, and that human answers do better. Its example is a cupcake: someone asks how to thicken frosting that keeps sliding off, when the real problem is that the cupcakes are still too warm. The paper behind it, arXiv 2609.06842, won the Best Resource Paper award at EMNLP 2026, which the conference’s awards page confirms.

Table Of Content

  • What I checked, and what I could not
  • The post’s numbers reproduce
  • Five things the post does not say
  • 1. Three quarters of the “real-world questions” are generated
  • 2. The model that scores the answers also drafted the rubrics
  • 3. The metric with the lowest judge agreement carries a main result
  • 4. The human baseline is scored on rubrics taken from its own answers
  • 5. The multiple-choice contrast is not like for like
  • The paper and the repository disagree with themselves in small ways
  • How precise are the model rankings?
  • What survives the audit
  • Questions I would put to the authors
  • Reproduce the checks
  • Step 1: what the script does
  • Step 2: the output
  • Step 3: confirm the repository facts yourself
  • Sources

I audited the post against the paper, the public code and the repository’s commit history. Every figure I could check in the post reproduces. What does not hold up as well is the framing around those numbers: three quarters of the “real-world questions” are written by the same model that scores the answers, the human baseline is scored against rubrics read off its own answers, the metric with the lowest judge agreement carries one of the post’s main results, and the paper and the repository disagree with themselves in several small places. I could not read the dataset itself, which sits behind Hugging Face’s click-through gate, so everything below comes from the post, the paper, the code, the data documentation and the commit log.

What I checked, and what I could not

I read the blog post, the full HTML of the paper including appendices (judge validation in Appendix E, multiple choice in F.2, the oracle experiment in Table 11), the allenai/XYBench repository at main (merge commit 29b58f1, November 15, 05:50 UTC) with its README, docs/DATA.md, the generation, judging, scoring and multiple-choice scripts and the last ten commits, and the Hugging Face API record for the dataset. That record shows the dataset as gated: auto with the license cc-by-nc-sa-3.0, while the code is Apache-2.0. Reading it requires logging in to Hugging Face and accepting the authors’ terms, which I did not do, and rerunning the benchmark needs paid API keys for the judge and the five models. So this is an audit of the published tables, text and code, not a rerun.

The one thing I could do offline is recompute from the paper’s printed Table 2. The script at the end of this article does that with the table typed in; every comparison below that says “my arithmetic” comes from it.

The post’s numbers reproduce

Credit first. The figures the post quotes match the paper, and the paper’s appendices are unusually candid about how the data and the judge were validated. The checks:

Claim in the post Source Result
8,115 questions from four sources: 1,557 Stack Overflow, 184 Stack Exchange, 102 manually curated, 6,272 WikiHow Paper, Sections 3.2 to 3.5 Sums to 8,115
On Stack Overflow, stated-request solutions appear in 75 to 86 percent of responses, underlying-goal solutions in 59 to 71 percent Table 2 Matches
On Stack Overflow, models focus on the underlying goal in 35 to 41 percent of responses and on the stated request in 47 to 57 percent Table 2 Matches
Oracle experiment, Sonnet 4.6 on Stack Overflow: goal solution 59 to 67 percent, misconception identification 47 to 34 percent Tables 2 and 11 Matches (0.67 at +14 percent, 0.34 at -28 percent)
A two-option choice picks the goal-focused answer in 71 to 84 percent of Stack Overflow cases Section 4 Matches the paper’s text
Judge agreement with two reviewers: 97.5 to 100 percent on solution presence, 84.6 to 93.3 on misconception identification, 74.2 to 79.8 on emphasis Appendix E Matches the pairwise figures
Best Resource Paper at EMNLP 2026 Conference awards page Confirmed
Every model is more likely to include the stated-request solution than the intended one, in all four sources Table 2 True in 20 of 20 model-by-source cells (my arithmetic)

The direction of the headline finding is therefore solid. The questions are about size, comparison and framing.

Five things the post does not say

1. Three quarters of the “real-world questions” are generated

The post’s opening section says XYBench draws on “8,115 real-world questions spanning programming and everyday activities such as cooking and home repair”. Further down it says the benchmark was built from four sources, one of them “6,272 questions generated from WikiHow guides”. That is 77.3 percent of the total. The real or hand-curated part, 1,741 Stack Overflow and Stack Exchange questions plus 102 curated ones, is 22.7 percent (my arithmetic). The cupcake query that opens the post belongs to the generated group: the paper’s Section 3.4 uses it as the worked example of the WikiHow pipeline.

The paper is open about how that group was made. In its words, “We use Claude-Sonnet-4.5, for data generation as well as filtering and rewriting stages.” Two authors then annotated 400 of the generated instances: per-dimension accuracy of the pipeline ran from 74.9 to 97.0 percent, and 68.5 percent of instances were fully correct across all fields. For the Stack Overflow-derived data the equivalent check covered 100 instances and found 84.0 percent fully correct. The post mentions validation only for the judge. The limitations section adds that the authors “lack human references for the synthetically constructed queries based on WikiHow”, so the largest subset has no human row to compare against at all.

2. The model that scores the answers also drafted the rubrics

For Stack Overflow, Stack Exchange and the curated subset, the stated and intended requests and the three rubrics (what a literal answer, a misconception explanation and a better alternative should contain) were extracted by an LLM. The paper says: “Both passes use Claude-Sonnet-4.5, a strong reasoning LLM.” For WikiHow the same model wrote the queries and rubrics. The post says what happens next: “In XYBench, Claude Sonnet 4.5 serves as the automated judge.” The repository fixes the judge as anthropic/claude-sonnet-4-5-20250929, with a comment that the call settings used for the paper were “no extended thinking, temperature 0.7”.

The authors know about this kind of overlap. When they filtered Stack Overflow queries, they used GPT-5 “to prevent bias from a single model family influencing both data construction and filtering”. The scoring step has no such separation, and one of the five models under test, Claude Sonnet 4.6, comes from the judge’s own family. The paper’s robustness check tested two OpenAI models, gpt-4.1 and gpt-5-mini, as alternate judges on 250 query-model pairs (150 WikiHow, 100 Stack Overflow). Agreement with the Claude judge was Cohen’s kappa 0.31 to 0.76, and with the human annotations 0.38 to 0.87. The conclusion drawn is that the evaluation is “fairly robust to the choice of judge, with Claude-Sonnet-4.5 agreeing the best with human annotations.” That is a sample-level agreement check. It does not rescore Table 2 with another judge, and it does not use GPT-5, the filter model, as a judge.

One pattern in Table 2 is worth a look, with the caveat that it proves nothing alone. Sonnet 4.6’s rank among the five models on misconception identification is last on Stack Overflow (0.47) and Stack Exchange (0.46), tied for first on the curated subset (0.52) and first on WikiHow (0.68, against 0.52 to 0.65 for the others). The WikiHow subset uses a different misconception prompt and label set (mistake_practice instead of request_mismatch in the data documentation), so the swing may come from the task, not the family. The paper gives no per-model judge error, so a family effect can be neither ruled in nor ruled out from the published material.

3. The metric with the lowest judge agreement carries a main result

The post reports the judge’s pairwise agreement with two reviewers on emphasis: 74.2 to 79.8 percent. Appendix E also reports Krippendorff’s alpha, which corrects for chance: 0.587 to 0.600 for emphasis and 0.634 to 0.866 for misconception identification. The same appendix says the two annotators agree with each other perfectly on emphasis (“solution presence and emphasis have perfect agreement between the annotators”), and then calls the judge-to-human agreement “commensurate” with the human-to-human figure. One hundred percent against 74 to 80 percent is not commensurate. By my arithmetic the judge’s emphasis label differs from the annotators’ shared label on 20.2 to 25.8 percent of responses.

The validation set was 500 responses, and the post does not say how they split: 100 from Stack Overflow and 400 from WikiHow. None came from Stack Exchange or the curated subset, which are two of the three subsets with human references. The post’s emphasis result, models focusing on the underlying goal in 35 to 41 percent of Stack Overflow responses, spans only 6 points across the five models, against a judge disagreement rate of 20 to 26 percent. Judge errors need not fall evenly on all five models, which is exactly why a per-model error breakdown would settle the matter. The paper does not give one.

4. The human baseline is scored on rubrics taken from its own answers

For Stack Overflow and Stack Exchange, the paper keeps the top three highest-scored answers plus the accepted answer as “reference” human responses. It then has the LLM “identify and extract the S-request and the I-request based on how it has been answered in reference human answers for that query”, and “re-examine the human reference answers corresponding to the query to identify elements that align with the corresponding three response dimensions”. The curated subset goes through “the same two-stage extraction pipeline”, followed by an author’s edits. So the definition of a cooperative answer was read off the human answers, and the human answers are then graded against it. Models are graded against someone else’s reading of what a good answer contains. The post’s sentence, “Unlike human reference answers, all five models were less likely to steer users away from that approach”, describes a comparison with that built-in advantage and does not say so.

The ceiling is not 100 percent either. On the curated subset, where each human answer is the source of its own rubric, the judge gives it 0.95 on intended-solution presence, 0.82 on intended-goal emphasis and 0.79 on misconception identification. How the human rows were produced is hard to confirm from outside. The paper says that “Human reference answers are evaluated identically.” The data documentation says the human files carry precomputed judge outputs and metrics and “are not re-evaluated”, and that for Stack Overflow and Stack Exchange the misconception flag “comes from the annotation of whether the answer identifies the misconception”, while for the curated set it comes from the judge’s tags. That reads like two different paths. Both statements can be true, and the gated data keeps me from checking.

The composite metrics add a wrinkle the post skips. Table 2’s “fully cooperative” score needs the misconception identified, both solutions present, and balanced or intended-goal emphasis. On Stack Overflow all five models score 0.20 to 0.28 against 0.18 for all human answers, and two of them (GPT-5 at 0.28, DeepSeek-V4 Pro at 0.24) are above the 0.23 of the accepted answers. On Stack Exchange GPT-5 (0.23) is above the 0.20 of all human answers. The large human advantage is on “fully pragmatic”: 0.47 against at most 0.31 on Stack Overflow, 0.57 against 0.23 on Stack Exchange and 0.68 against 0.25 on the curated subset (my arithmetic). Whether models “fall far short” of humans depends on which composite you pick.

5. The multiple-choice contrast is not like for like

The post sets 71 to 84 percent against 35 to 41 percent and concludes: “The results expose a gap between recognizing useful advice and producing it, which a multiple-choice test alone would miss.” Three things make that comparison loose. First, a two-option choice has a 50 percent chance rate, so 71 to 84 percent is 21 to 34 points above chance. Second, the options were not natural answers: Appendix F.2 says they “are generated using Claude-Sonnet-4.5 by providing as input the explicit response dimensions”, rewritten to similar length. Third, the pragmatic option is not purely pragmatic. The paper’s main text calls it “a response focused on misconception identification and I-solution”, but the repository’s multiple-choice script describes it as a hybrid_response that “answers the stated request, identifies the misconception, and addresses the intended request”. An option that contains the literal answer plus a misconception explanation plus a better alternative is a near-superset of the other option, so a preference for it is not a surprising outcome.

On the generation side, the 35 to 41 percent counts only answers whose emphasis is primarily the underlying goal. A hybrid-style answer would likely be labelled balanced or goal-focused, and that share on Stack Overflow is 44 to 53 percent for the five models (emphasis on the intended solution plus balanced, my arithmetic from Table 2). Put next to each other, 71 to 84 percent chosen from two options and 44 to 53 percent generated is still a gap, but a smaller and differently shaped one than the post’s pairing suggests.

A related detail: the table the appendix gives for the oracle experiment, where models are told the stated and intended requests, covers four models, and they are not the five in the headline. Table 11 has Claude Sonnet 4.6, GPT-4.1, GPT-5 and Gemini 3.1 Pro. Qwen3.5-9B and DeepSeek-V4 Pro are absent, and GPT-4.1 is an addition.

The paper and the repository disagree with themselves in small ways

None of these changes the story on its own. Together they say the numbers in prose were not all regenerated from the final tables, and that the released data kept moving after the paper.

Item What the text says What the table or file says
Stated-solution presence range Abstract and introduction: 0.75 to 0.92. Section 4: 0.57 to 0.91 Table 2 minimum 0.57 (Sonnet 4.6, WikiHow), maximum 0.91 (GPT-5, Stack Exchange)
Misconception gap to humans Section 4: models lag 25 to 40 points on Stack Overflow, Stack Exchange and the curated set 19 points (GPT-5, Stack Overflow) to 42 (Sonnet 4.6, Stack Exchange) against the all-answers row (my arithmetic)
Intended-solution presence gap on Stack Overflow and Stack Exchange Section 4: 3 to 17 points, except GPT-5 on Stack Overflow GPT-5 on Stack Overflow is 4 points above humans; the others run from 1 to 25 points (my arithmetic)
Number of queries README overview: 8,115 README data table: 1,557 + 184 + 102 + 6,268 = 8,111
Data license Paper, Appendix G: “XYBench will be released under CC by 4.0.” Hugging Face record: cc-by-nc-sa-3.0; the README states a license for the code only (Apache 2.0)
“Human (highest score)” row, Stack Overflow Paper: highest-scored answers are marginally more fully pragmatic than accepted ones (0.47 against 0.45 on Stack Overflow, 0.56 against 0.54 on Stack Exchange) The Stack Overflow row is identical to “Human (all)” in all eight columns; the repository’s code for it was changed on November 15

The last row deserves a closer description. On November 15 at 05:27 UTC, two days after the blog post, commit 808f636 arrived with the message “fix the code for highest score computation. updating data doc for fixed SE file.” In compute_results.py it changed the condition that puts an answer into the “highest score” bucket from is_highest_score or is_high_score to is_highest_score alone, and in docs/DATA.md it removed a note that is_highest_score is null for Stack Exchange. The old condition is consistent with a Stack Overflow “highest” row that equals the “all” row, and, if the old note was accurate, with a Stack Exchange row that could only come from the is_high_score fallback. I cannot see the corrected values without the data. What I can say is that the repository’s current code no longer computes the row the paper prints, and that the paper’s sentence about highest-scored versus accepted answers rests on it.

The timeline matters too. The repository was created on August 31, the paper is dated September 6, a commit titled “dataset updated” landed on November 4, the post ran on November 13, and the Stack Exchange file was fixed on November 15. A reader who downloads the data today may not get the files behind Table 2. The 6,268 against 6,272 WikiHow count may be a trace of that, though the repository does not say.

How precise are the model rankings?

The paper reports significance for one analysis, the phi coefficients in Table 10, and notes there that the curated subset, at 102 queries, is small enough that two models fall short of significance. It gives no intervals, paired tests or repeated runs for Table 2. A search of compute_results.py, evaluate_responses.py, utils.py and run_mcq.py for bootstrap, stderr, confidence, variance, repeat, majority and vote found nothing. The post describes one response per question, the repository generates at temperature 1.0 by default, and the judge runs at temperature 0.7 with no repeated passes or voting that I could find.

The sampling-only noise is easy to bound. The table below uses the binomial half-width of a proportion near 0.4, treats queries as independent, and ignores the judge’s own error. A paired comparison of two models on the same queries can be tighter than two independent intervals, so read these as rough scales, not tests.

Subset Queries 95% half-width at p = 0.4 Spread of the five models’ intended-goal emphasis
Stack Overflow 1,557 2.4 points 6 points
Stack Exchange 184 7.1 points 9 points
Manually curated 102 9.5 points 7 points
WikiHow 6,272 1.2 points 3 points

On Stack Overflow and WikiHow the differences between models are about two and a half times the sampling noise, though still smaller than the judge’s disagreement rate on emphasis. On Stack Exchange and the curated subset they are about the size of the noise, so most pairwise rankings there are not resolved. The differences between models and humans are much larger and survive this check; they carry the other caveats above.

What survives the audit

The finding that models mention the literal solution more often than the underlying one holds in every cell of Table 2, and on Stack Overflow the gap (75 to 86 against 59 to 71 percent) is many times the sampling noise. The oracle experiment is, to my reading, the most interesting result in the paper, and it is more nuanced than the post’s summary. On WikiHow, where no human reference exists, telling the model the goal lifts intended-goal emphasis sharply (Sonnet 4.6 from 0.32 to 0.77, GPT-5 from 0.30 to 0.60), so models can act on a stated goal. On Stack Overflow the same hint lowers misconception identification for three of the four models tested (Sonnet 4.6, GPT-4.1 and GPT-5), and the paper’s explanation, memorization of Stack Overflow, is a hypothesis it does not test.

Two scope notes for anyone citing the result. The evaluated models are Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro; the navigation of OpenAI’s ChatGPT Wallet announcement, read on November 15, lists GPT-6.1 Sol, GPT-6 Astra, GPT-5.6 and GPT-5.5 under “Latest Advancements”, and none of those was tested. And the Hub record lists a non-commercial license (cc-by-nc-sa-3.0) where the paper promises CC BY 4.0, so check the terms before using the data in a commercial evaluation pipeline.

Questions I would put to the authors

  1. Can Table 2 be rescored with a judge outside the Claude family, for example GPT-5, and the model ranking compared?
  2. What is the judge’s emphasis and misconception error per evaluated model, and on Stack Exchange and the curated subset, which the 500-response validation did not cover?
  3. Can the human rows be regenerated with the same approach-unit judge used for models, with rubrics built from held-out answers, so the baseline is not graded on its own rubric?
  4. What are the paired confidence intervals for the model-to-model and model-to-human differences?
  5. Which files produced the printed Table 2, how many WikiHow queries do they hold (6,272 or 6,268), and what does the “highest score” row look like after the November 15 fix?
  6. Which license applies to the data, CC BY 4.0 as the paper says or the CC BY-NC-SA 3.0 on the Hub record?

Reproduce the checks

Step 1: what the script does

The script below holds the eight metric columns of the paper’s Table 2 for each model and human row, typed from the HTML version of arXiv 2609.06842v1. It uses only the Python standard library. It recomputes the query counts, the cell-by-cell comparison, the ranges, the human gaps, the composite scores, Sonnet 4.6’s rank, the multiple-choice contrast, the sampling half-widths and the judge disagreement, and it checks whether the Stack Overflow “highest” row equals the “all” row. Save it as xy_check.py and run python xy_check.py with Python 3.13 or later.

"""xy_check.py: recompute what can be recomputed from XYBench's printed Table 2 (arXiv 2609.06842v1).
Values are typed from Table 2 of the paper, order per row:
S-solution present, I-solution present, emphasis S, emphasis balanced, emphasis I,
misconception identified, fully pragmatic, fully cooperative.
Standard library only (Python 3.13)."""
from math import sqrt

COLS = ["s_pres", "i_pres", "emph_s", "emph_b", "emph_i", "miscon", "fully_prag", "fully_coop"]
T2 = {
    "Stack Overflow": {
        "Sonnet 4.6": [.82, .59, .57, .08, .36, .47, .25, .20],
        "GPT-5": [.83, .71, .47, .12, .41, .60, .31, .28],
        "Gemini 3.1 Pro": [.79, .62, .51, .08, .40, .54, .29, .22],
        "Qwen3.5-9B": [.75, .64, .53, .06, .41, .60, .29, .21],
        "DeepSeek-V4 Pro": [.86, .66, .57, .09, .35, .55, .25, .24],
        "Human (all)": [.58, .67, .39, .06, .55, .79, .47, .18],
        "Human (accepted)": [.66, .67, .40, .08, .52, .81, .45, .23],
        "Human (highest)": [.58, .67, .39, .06, .55, .79, .47, .18],
    },
    "Stack Exchange": {
        "Sonnet 4.6": [.75, .46, .69, .04, .27, .46, .15, .12],
        "GPT-5": [.91, .64, .64, .10, .26, .63, .22, .23],
        "Gemini 3.1 Pro": [.80, .49, .62, .06, .32, .53, .23, .18],
        "Qwen3.5-9B": [.81, .62, .66, .06, .29, .59, .23, .17],
        "DeepSeek-V4 Pro": [.89, .54, .69, .08, .23, .50, .19, .19],
        "Human (all)": [.53, .71, .37, .03, .60, .88, .57, .20],
        "Human (accepted)": [.67, .77, .39, .04, .57, .90, .54, .26],
        "Human (highest)": [.53, .71, .37, .03, .60, .88, .56, .20],
    },
    "Manually curated": {
        "Sonnet 4.6": [.82, .58, .62, .05, .33, .52, .25, .21],
        "GPT-5": [.86, .63, .62, .09, .29, .45, .17, .18],
        "Gemini 3.1 Pro": [.83, .63, .56, .10, .34, .47, .21, .25],
        "Qwen3.5-9B": [.82, .48, .64, .03, .33, .52, .20, .17],
        "DeepSeek-V4 Pro": [.82, .57, .61, .03, .36, .45, .24, .19],
        "Human": [.46, .95, .07, .11, .82, .79, .68, .33],
    },
    "WikiHow": {
        "Sonnet 4.6": [.57, .33, .66, .02, .32, .68, .15, .08],
        "GPT-5": [.82, .55, .67, .02, .30, .52, .16, .13],
        "Gemini 3.1 Pro": [.71, .41, .68, .02, .30, .65, .18, .12],
        "Qwen3.5-9B": [.61, .37, .70, .01, .29, .61, .15, .08],
        "DeepSeek-V4 Pro": [.74, .47, .69, .02, .30, .57, .17, .12],
    },
}
MODELS = ["Sonnet 4.6", "GPT-5", "Gemini 3.1 Pro", "Qwen3.5-9B", "DeepSeek-V4 Pro"]
N = {"Stack Overflow": 1557, "Stack Exchange": 184, "Manually curated": 102, "WikiHow": 6272}


def col(src, who, name):
    return T2[src][who][COLS.index(name)]


print("== 1. Query counts")
print("paper: 1557 + 184 + 102 + 6272 =", 1557 + 184 + 102 + 6272)
print("repo README table: 1557 + 184 + 102 + 6268 =", 1557 + 184 + 102 + 6268)
print("WikiHow share of 8,115: %.1f%%" % (100 * 6272 / 8115))
print("real or hand-curated: %d, %.1f%% of 8,115" % (1741 + 102, 100 * (1741 + 102) / 8115))
print("filter kept: SO %.1f%% (1557/1780), SE %.1f%% (184/214)" % (100 * 1557 / 1780, 100 * 184 / 214))

print("\n== 2. Stated-request solution present more often than intended-request solution?")
cells = [(s, m) for s in T2 for m in MODELS]
yes = sum(col(s, m, "s_pres") > col(s, m, "i_pres") for s, m in cells)
print("%d of %d model-by-source cells" % (yes, len(cells)))

print("\n== 3. Ranges over all 20 cells (abstract says S 0.75-0.92, I 0.33-0.71; Section 4 says S 0.57-0.91)")
for name in ("s_pres", "i_pres"):
    vals = [(col(s, m, name), s, m) for s, m in cells]
    lo, hi = min(vals), max(vals)
    print("%s min %.2f (%s, %s)  max %.2f (%s, %s)" % (name, lo[0], lo[1], lo[2], hi[0], hi[1], hi[2]))

print("\n== 4. Gap to Human (all) in points: misconception identified (paper text: 25-40) and I-solution present on SO and SE (text: 3-17)")
gaps = []
for s in ("Stack Overflow", "Stack Exchange"):
    for m in MODELS:
        gaps.append((round(100 * (col(s, "Human (all)", "miscon") - col(s, m, "miscon"))), s, m))
for m in MODELS:
    gaps.append((round(100 * (col("Manually curated", "Human", "miscon") - col("Manually curated", m, "miscon"))), "Manually curated", m))
print("misconception gap min %d (%s, %s), max %d (%s, %s)" % (min(gaps)[0], min(gaps)[1], min(gaps)[2], max(gaps)[0], max(gaps)[1], max(gaps)[2]))
ig = [(round(100 * (col(s, "Human (all)", "i_pres") - col(s, m, "i_pres"))), s, m) for s in ("Stack Overflow", "Stack Exchange") for m in MODELS]
print("I-solution gap min %d (%s, %s), max %d (%s, %s)" % (min(ig)[0], min(ig)[1], min(ig)[2], max(ig)[0], max(ig)[1], max(ig)[2]))

print("\n== 5. Fully cooperative (needs S and I solution, balanced or I emphasis, misconception identified)")
for s in ("Stack Overflow", "Stack Exchange"):
    h_all, h_acc = col(s, "Human (all)", "fully_coop"), col(s, "Human (accepted)", "fully_coop")
    ms = {m: col(s, m, "fully_coop") for m in MODELS}
    print("%s: humans all %.2f accepted %.2f | models %s" % (s, h_all, h_acc, ", ".join("%s %.2f" % kv for kv in ms.items())))
    print("   models at or above Human (all): %s" % [m for m, v in ms.items() if v >= h_all])
for s in ("Stack Overflow", "Stack Exchange", "Manually curated"):
    h = col(s, "Human (all)" if s != "Manually curated" else "Human", "fully_prag")
    print("%s fully pragmatic: human %.2f, best model %.2f" % (s, h, max(col(s, m, "fully_prag") for m in MODELS)))

print("\n== 6. Sonnet 4.6 rank (1 = best of the five models) on misconception identified")
for s in T2:
    order = sorted(MODELS, key=lambda m: -col(s, m, "miscon"))
    print("%-17s rank %d  (Sonnet %.2f; best %s %.2f)" % (s, order.index("Sonnet 4.6") + 1, col(s, "Sonnet 4.6", "miscon"), order[0], col(s, order[0], "miscon")))

print("\n== 7. Blog contrast: multiple choice vs generated emphasis, Stack Overflow")
print("two-option choice picks the pragmatic answer 71-84% of the time; chance is 50%")
for m in MODELS:
    i, b = col("Stack Overflow", m, "emph_i"), col("Stack Overflow", m, "emph_b")
    print("  %-16s I-focused %.2f, I-focused or balanced %.2f" % (m, i, i + b))

print("\n== 8. Sampling noise only: 95% half-width of a proportion (one response per query, judge error ignored)")
for s, n in N.items():
    hw = 1.96 * sqrt(0.4 * 0.6 / n)
    spread_i = max(col(s, m, "emph_i") for m in MODELS) - min(col(s, m, "emph_i") for m in MODELS)
    print("%-17s n=%-5d half-width at p=0.4: %.1f points | spread of the five models' I-emphasis: %.0f points" % (s, n, 100 * hw, 100 * spread_i))

print("\n== 9. Judge versus the two annotators (paper appendix, 500 responses: 100 SO + 400 WikiHow)")
for name, lo, hi in (("solution presence", .975, 1.0), ("misconception", .846, .933), ("emphasis", .742, .798)):
    print("%-18s agreement %.1f-%.1f%% -> judge disagrees on %.1f-%.1f%% of responses" % (name, 100 * lo, 100 * hi, 100 * (1 - hi), 100 * (1 - lo)))

print("\n== 10. Is the SO 'Human (highest)' row identical to 'Human (all)'?")
print("Stack Overflow:", T2["Stack Overflow"]["Human (highest)"] == T2["Stack Overflow"]["Human (all)"],
      "| Stack Exchange:", T2["Stack Exchange"]["Human (highest)"] == T2["Stack Exchange"]["Human (all)"])

Step 2: the output

This is what it prints, which is where every “my arithmetic” figure above comes from.

== 1. Query counts
paper: 1557 + 184 + 102 + 6272 = 8115
repo README table: 1557 + 184 + 102 + 6268 = 8111
WikiHow share of 8,115: 77.3%
real or hand-curated: 1843, 22.7% of 8,115
filter kept: SO 87.5% (1557/1780), SE 86.0% (184/214)

== 2. Stated-request solution present more often than intended-request solution?
20 of 20 model-by-source cells

== 3. Ranges over all 20 cells (abstract says S 0.75-0.92, I 0.33-0.71; Section 4 says S 0.57-0.91)
s_pres min 0.57 (WikiHow, Sonnet 4.6)  max 0.91 (Stack Exchange, GPT-5)
i_pres min 0.33 (WikiHow, Sonnet 4.6)  max 0.71 (Stack Overflow, GPT-5)

== 4. Gap to Human (all) in points: misconception identified (paper text: 25-40) and I-solution present on SO and SE (text: 3-17)
misconception gap min 19 (Stack Overflow, GPT-5), max 42 (Stack Exchange, Sonnet 4.6)
I-solution gap min -4 (Stack Overflow, GPT-5), max 25 (Stack Exchange, Sonnet 4.6)

== 5. Fully cooperative (needs S and I solution, balanced or I emphasis, misconception identified)
Stack Overflow: humans all 0.18 accepted 0.23 | models Sonnet 4.6 0.20, GPT-5 0.28, Gemini 3.1 Pro 0.22, Qwen3.5-9B 0.21, DeepSeek-V4 Pro 0.24
   models at or above Human (all): ['Sonnet 4.6', 'GPT-5', 'Gemini 3.1 Pro', 'Qwen3.5-9B', 'DeepSeek-V4 Pro']
Stack Exchange: humans all 0.20 accepted 0.26 | models Sonnet 4.6 0.12, GPT-5 0.23, Gemini 3.1 Pro 0.18, Qwen3.5-9B 0.17, DeepSeek-V4 Pro 0.19
   models at or above Human (all): ['GPT-5']
Stack Overflow fully pragmatic: human 0.47, best model 0.31
Stack Exchange fully pragmatic: human 0.57, best model 0.23
Manually curated fully pragmatic: human 0.68, best model 0.25

== 6. Sonnet 4.6 rank (1 = best of the five models) on misconception identified
Stack Overflow    rank 5  (Sonnet 0.47; best GPT-5 0.60)
Stack Exchange    rank 5  (Sonnet 0.46; best GPT-5 0.63)
Manually curated  rank 1  (Sonnet 0.52; best Sonnet 4.6 0.52)
WikiHow           rank 1  (Sonnet 0.68; best Sonnet 4.6 0.68)

== 7. Blog contrast: multiple choice vs generated emphasis, Stack Overflow
two-option choice picks the pragmatic answer 71-84% of the time; chance is 50%
  Sonnet 4.6       I-focused 0.36, I-focused or balanced 0.44
  GPT-5            I-focused 0.41, I-focused or balanced 0.53
  Gemini 3.1 Pro   I-focused 0.40, I-focused or balanced 0.48
  Qwen3.5-9B       I-focused 0.41, I-focused or balanced 0.47
  DeepSeek-V4 Pro  I-focused 0.35, I-focused or balanced 0.44

== 8. Sampling noise only: 95% half-width of a proportion (one response per query, judge error ignored)
Stack Overflow    n=1557  half-width at p=0.4: 2.4 points | spread of the five models' I-emphasis: 6 points
Stack Exchange    n=184   half-width at p=0.4: 7.1 points | spread of the five models' I-emphasis: 9 points
Manually curated  n=102   half-width at p=0.4: 9.5 points | spread of the five models' I-emphasis: 7 points
WikiHow           n=6272  half-width at p=0.4: 1.2 points | spread of the five models' I-emphasis: 3 points

== 9. Judge versus the two annotators (paper appendix, 500 responses: 100 SO + 400 WikiHow)
solution presence  agreement 97.5-100.0% -> judge disagrees on 0.0-2.5% of responses
misconception      agreement 84.6-93.3% -> judge disagrees on 6.7-15.4% of responses
emphasis           agreement 74.2-79.8% -> judge disagrees on 20.2-25.8% of responses

== 10. Is the SO 'Human (highest)' row identical to 'Human (all)'?
Stack Overflow: True | Stack Exchange: False

Step 3: confirm the repository facts yourself

The commit and the data documentation are public. The GitHub commit page for 808f636 shows the one-line change to the “highest score” condition and the removed note in docs/DATA.md. The Hugging Face API URL https://huggingface.co/api/datasets/allenai/XYBench-dataset returns the gated and license fields. The awards page lists the Best Resource Paper entry with the author list.

Sources

  • Ai2 on Hugging Face, XYBench: Helping with the problem behind the question (November 13, 2026)
  • Yerukola et al., XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions? (arXiv 2609.06842v1, September 6, 2026; HTML version)
  • allenai/XYBench repository, docs/DATA.md and commit 808f636
  • allenai/XYBench-dataset on Hugging Face (gated)
  • EMNLP 2026 awards
  • OpenAI, Introducing ChatGPT Wallet (November 13, 2026), for the model list in its navigation

Tags:

AI BenchmarksAi2Dataset TransparencyLLM EvaluationLLM-as-a-JudgeXYBench

Share

An amber warning beacon on a white surface, a flashing-light stand-in for the FBI’s FLASH alert
Previous Post

The FBI’s ARTEX Warning Says a Supply Chain Compromise “Enabled Access” to 2,250 Tenants Where Microsoft Counted Sign-In Attempts and Cites “Over 1,600” Consoles Where Censys Counted 1,465, and a Microsoft Post It Cites Mistypes One Indicator in Its Hunting Query

A black manual telephone switchboard with rows of orange jacks and red patch cords plugged into the lower row, above a rotary dial and a handset
Next Post

How to Route Alerts in Alertmanager So Pages Reach the On-Call and Tickets Reach the Queue, and Test Grouping, Inhibition, and Silences Before an Outage Does

No Comment! Be the first one.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Latest
16 Nov
How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
16 Nov
llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries
Trending
November 16, 2026
How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
November 16, 2026
llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries
November 16, 2026
NoisyTCP Can Crash a Linux Server Over the Network, but the Fix Has Reached Only Three of the Seven Kernel Lines Still Being Released, Red Hat and Ubuntu Have No Update, and the Researcher’s Own Timings Run From 5 Seconds to 14 Hours
November 15, 2026
How to Route Alerts in Alertmanager So Pages Reach the On-Call and Tickets Reach the Queue, and Test Grouping, Inhibition, and Silences Before an Outage Does
November 15, 2026
Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses
November 15, 2026
The FBI’s ARTEX Warning Says a Supply Chain Compromise “Enabled Access” to 2,250 Tenants Where Microsoft Counted Sign-In Attempts and Cites “Over 1,600” Consoles Where Censys Counted 1,465, and a Microsoft Post It Cites Mistypes One Indicator in Its Hunting Query

Related Posts

Blue-lit server racks in a modern data center, illustrating the compute infrastructure behind the AI boom.
Articles

The AI Boom Is Spending Real Money Before Proving Real Returns

June 7, 2026
Technician working with a laptop beside server racks, representing enterprise AI retrieval infrastructure
Articles

Google’s Agentic RAG Push Makes Enterprise AI Less of a One-Shot Guess

June 7, 2026
A person with a laptop and smartphone, representing digital attention and AI-assisted work
Articles

AI Chatbots Are Making Attention a Design Problem

June 7, 2026
A customer-support representative wearing a headset against a dark studio background.
Articles

The Meta AI Support Hack Was a Plain Old Authorization Failure

June 7, 2026
SXZ.io SXZ.io
  • [email protected]

Categories

Articles
Learning Hub
News

All Rights Reserved by SXZ.io ©2026