Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses
Ai2’s XYBench figures reproduce, but the model that judges the...






