IEEE Spectrum’s AI-Slop Package Turns Research Integrity Into a Detection Race, but Its Headline Numbers Measure Vocabulary, a Group-Level Ratio, and 50 Flagged Reviews
IEEE Spectrum’s AI-slop package leans on statistics that trace to a vocabulary score, a group-level ratio and 50 flagged reviews, and its fake-paper test shows how differently reference checkers...
IEEE Spectrum’s November issue carries a package about AI slop in science. An editor’s note by Harry Goldstein, titled “Guarding the Gates of Knowledge” (the page’s metadata title is “AI Slop Threatens Trust in Science”), went online on November 1 at 14:00 UTC. It introduces a feature by Benjamin Skuse, who tried to get two AI-made papers past IEEE Access (published October 27, updated October 30), and a Q&A with a Wikipedia volunteer (October 21). The note sums up the feature’s lesson as “an arms race between increasingly plausible slop and the systems aimed at detecting it.”
Table Of Content
- What I found
- What the package claims, and what each claim rests on
- The 57 percent is a vocabulary score on four publishers
- What the PNAS abstract says
- A ten-minute illustration of what a word score sees
- The “over 12 percent” is 50 reviews flagged by a classifier trained on synthetic text
- Where the number comes from
- What the authors say limits it, and what supports it
- Where “completely” comes from
- The 1-in-40 figure is a ratio of two different measurements
- The ICLR acceptances were workshop acceptances, and the papers were withdrawn
- The experiment: three reference-check counts and no denominator
- What the feature reports
- An independent benchmark: five tools on the same 104 references
- A reference check you can run in a minute
- Run 1: the article the feature links
- Run 2: a control, the reference list of a Nature paper
- What a clean reference list does and does not show
- The Wikipedia numbers rest on a commercial detector
- What holds up, and five questions for the next AI-slop statistic
- What I could not check
- Sources
The feature backs that framing with several statistics and one experiment. I traced each statistic to the study behind it, checked the conference-acceptance claims against the papers they come from, and ran a small DOI checker on two real reference lists to see what a “reference check” can and cannot tell you. The short version: the figures match their sources, but each one measures something narrower than the sentence that carries it, and the experiment’s three reference-check counts cannot be compared with one another.
What I found
- The 57 percent is a share of articles from four publishers (Elsevier, Frontiers, MDPI and PLoS) that scored high on a list of 228 words. It is not “all published academic articles” and it is not a count of slop. PNAS blocked my script, so I worked from the abstract and could not check the threshold.
- The “over 12 percent” of Nature Communications reviews is 50 of 406 sampled reviews, flagged by a classifier trained on DeepSeek-written reviews, in an arXiv preprint that describes itself as an independent study for a University of Pennsylvania course. The preprint calls its result “suggestive evidence” and never says “completely.”
- The 1-in-40 figure is a ratio of a group-level estimate to a 0.43 percent disclosure rate for the first quarter of 2025, taken from two different samples. It is not a count of papers.
- The AI-written papers “accepted” by the International Conference on Learning Representations were workshop acceptances (the workshop took 70 percent of submissions), and they were withdrawn.
- In the experiment, three checkers reported 7, 12 and 24 problem references with no shared denominator. An independent benchmark that ran five tools on the same 104 references, 33 of them problematic, got flag counts from 38 to 77.
- My own checker, run on the article the feature links as its fabricated-reference example, flags 5 of 17 references: three DOIs that do not exist and two that belong to unrelated papers. Run on the reference list of a Nature paper, it flags 2 of 37 DOIs, and both are benign.
What the package claims, and what each claim rests on
The table pairs each claim, in the feature’s words, with the source it links and what that source actually measured.
| Spectrum says | Source | What the source measured |
|---|---|---|
| “an estimated 57 percent of all published academic articles exhibited detectable evidence of large language model (LLM) influence,” “a massive jump from 12 percent in 2023” | Siler, PNAS, May 29, 2026 | Full texts of 7.3 million journal articles from four publishers (Elsevier, Frontiers, MDPI, PLoS), 2020 to 2025, scored on 228 words that rose sharply after 2022. “Evidence of LLM influence” spans subtle polishing to mostly or entirely generated text. |
| “over 12 percent of Nature Communications reviews were found to be completely AI-generated in 2025” | Shen and Wang, arXiv 2602.00319v2, March 23, 2026 | 50 of 406 sampled reviews (12.32 percent) classified as AI-generated by a Longformer model trained on 2021 reviews against DeepSeek-written ones. |
| “for every 40 papers showing statistical evidence of AI usage in 2025, only one formally disclosed it” | He and Bu, PNAS, February 24, 2026 | A ratio of a group-level AI-content estimate (5.2 million abstracts) to a disclosure rate of 0.43 percent in the first quarter of 2025 (164,579 full-text papers). |
| AI Scientist and Autoscience papers “accepted through double-blind peer review in 2025 by the International Conference on Learning Representations” | Lu et al., Nature, March 25, 2026; Autoscience Institute | One of three AI Scientist submissions scored above the threshold at the ICLR 2025 ICBINB workshop (70 percent acceptance rate) and was withdrawn by design. Autoscience’s papers were at workshops (Tiny Papers track) and were withdrawn too. |
| FARS “created 166 papers in 417 hours” | Tang et al., arXiv 2606.31651v2, July 13, 2026 | Matches the paper. The paper also reports about $186,000 and 21.6 billion model tokens, and integrity flags on 39 of the 140 papers that were reviewed. |
| IEEE Access gets “about 60,000 submissions per year, accepting around 20 percent, or about 12,000 papers” | Managing editor Jenny Mahoney, as reported in the feature; Crossref | Crossref lists 13,943 journal articles with a 2025 date and 13,423 for 2024, so 12,000 is in range but about 16 percent below the 2025 count (my query). |
| Peer review was “processing well over 3 million papers in 2022” | NSF NCSES publications-output report (nsb202333) | “Global publication output reached 3.3 million articles in 2022,” a count of published articles, so rejected submissions come on top. The claim holds. |
The 57 percent is a vocabulary score on four publishers
What the PNAS abstract says
Kyle Siler’s paper, “The diffusion of large language models in published academic articles,” appeared in PNAS on May 29, 2026. Its abstract says it analyzes “full texts of 7.3 million journal articles” from “four major publishers (Elsevier, Frontiers, MDPI, and PLoS).” It built a list of “228 focal words exhibiting sharp post-2022 frequency increases consistent with LLM output” and scored each article on its rate of using them. The result: “By 2025, an estimated 57% of published articles exhibited evidence of LLM influence, up from 12% in 2023.” The same abstract adds that influenced articles show “substantial heterogeneity, ranging from subtle linguistic influence to articles mostly or entirely LLM-generated.”
Spectrum’s sentence says “all published academic articles.” The abstract’s population is journal articles from four publishers; IEEE, Springer Nature, Wiley and the rest of the literature sit outside it. That does not make 57 percent wrong, and the feature itself adds a fair caveat: “Of course, evidence of AI use doesn’t necessarily mean a paper is slop.” It does mean the number describes a vocabulary signature in one slice of the literature, produced by a score that, as the abstract describes it, is a rate of focal-word use rather than a finding about who wrote what.
PNAS answered both my script and a headless browser with a Cloudflare challenge, so I worked from the abstract as deposited with Crossref and indexed by PubMed. I cannot tell you where the paper sets the threshold for “evidence,” how it treats articles near that threshold, or how it handles the fact that the words were chosen because they rose after 2022.
A ten-minute illustration of what a word score sees
To see what the underlying signal looks like, I counted how many PubMed abstracts contain a handful of the words usually named in this literature, using Europe PMC’s public search API (one query per word per year, restricted to PubMed records with an abstract, between 1.33 and 1.50 million abstracts a year from 2021 to 2025). This is not Siler’s corpus or method, which uses full text from four publishers; it only shows the kind of signal a focal-word score is built on. Rates are per 100,000 abstracts.
| Word | 2021 | 2022 | 2023 | 2024 | 2025 | 2025 vs 2021 | 2025 vs 2024 |
|---|---|---|---|---|---|---|---|
| delves | 5.0 | 7.4 | 72.6 | 356.8 | 175.7 | 35.0 times | 0.49 |
| showcasing | 13.8 | 16.4 | 49.5 | 226.0 | 173.0 | 12.5 times | 0.77 |
| intricate | 134.4 | 131.9 | 315.4 | 981.2 | 718.9 | 5.3 times | 0.73 |
| underscores | 102.8 | 102.0 | 279.7 | 1,421.9 | 2,110.8 | 20.5 times | 1.48 |
| notably | 1,138.4 | 1,194.1 | 1,584.6 | 3,432.4 | 4,442.6 | 3.9 times | 1.29 |
| commendable | 5.1 | 5.8 | 14.5 | 49.0 | 25.9 | 5.1 times | 0.53 |
- Flat, then a break. Between 2021 and 2022 the rates barely move (underscores 102.8 to 102.0, intricate 134.4 to 131.9); the break comes in 2023 and 2024.
- The baseline is not zero. In 2021, “notably” was already in about 1,100 of every 100,000 abstracts and “underscores” in about 100. A word-based score therefore measures an excess over an expected rate, and the threshold chosen for “evidence” drives the headline percentage (my reasoning, not a claim about Siler’s threshold).
- The signature drifts. From 2024 to 2025, “delves” fell to 49 percent of its 2024 rate and “commendable” to 53 percent, while “underscores” and “notably” kept climbing. Authors, models and prompts change, so a fixed word list scores a style that is itself moving.
- Vocabulary is not authorship. A researcher polishing a paragraph with a chatbot and a fully generated manuscript can raise the same counts. He and Bu say their own method cannot tell “linguistic polishing” from “substantive text generation.”
One caveat on the table: Europe PMC matched exact word forms (it counts “delves” and “delving” separately), so each row undercounts its word family.
The “over 12 percent” is 50 reviews flagged by a classifier trained on synthetic text
Where the number comes from
Spectrum links “Detecting AI-Generated Content in Academic Peer Reviews” by Siyuan Shen and Kai Wang, posted on arXiv on January 30, 2026 (version 2 on March 23). The method is straightforward. Fine-tune a Longformer model (with LoRA) on a balanced set of human reviews from 2021 and synthetic reviews written through the DeepSeek Reasoner API, then freeze it and run it over later years of reviews from ICLR and Nature Communications. The paper carries a disclaimer that it reports “the outcome of an independent study of the AMCS 5999 course in the Department of Mathematics at the University of Pennsylvania.”
The table gives the counts from the paper’s Tables 1 and 2.
| Year | Nature Communications: flagged / reviews | ICLR: flagged / reviews |
|---|---|---|
| 2022 | 0 / 348 (0.00%) | 2 / 1,937 (0.10%) |
| 2023 | 1 / 287 (0.35%) | 0 / 1,887 (0.00%) |
| 2024 | 8 / 460 (1.74%) | 133 / 1,818 (7.32%) |
| 2025 | 50 / 406 (12.32%) | 392 / 1,961 (19.99%) |
The 2025 Nature Communications figure is 50 flagged reviews out of 406. My arithmetic puts the 95 percent Wilson interval for that sample at 9.5 to 15.9 percent, before any question about how representative the sample is. The 2024 figure is 8 of 460, so the flagged rate rose about sevenfold in a year, which is the paper’s main finding and is large against that uncertainty.
What the authors say limits it, and what supports it
- The authors say their binary framing “certainly over-simplifies the underlying phenomenon,” and that the percentages “should be interpreted as estimates of reviews that exhibit strong AI-generated characteristics with minimal human editing, and they likely underestimate the prevalence of AI-assisted review writing.”
- All synthetic training reviews came from one model with one prompt, so “it remains unclear how well the detector generalizes to reviews written or edited using other models, such as ChatGPT or Gemini.” They also flag “the potential risk of overfitting arising from the use of synthetic training data that can be more easily separable than real-world AI-assisted reviews.”
- The abstract’s own verdict is “suggestive evidence.”
- In its favor: in 2022 and 2023, when the paper reads AI-assisted reviewing as still uncommon, the detector flagged almost nothing: 2 of 3,824 ICLR reviews and 1 of 635 Nature Communications reviews. That is the closest thing the paper offers to a false-positive check on human text the model never saw, and it is reassuring, though it mostly tests human-written text rather than the hybrid human-plus-AI text the authors call challenging.
Where “completely” comes from
Neither the abstract nor the results use the word “completely.” The nearest published wording is Nature’s. Its November 27, 2025 report is headlined “Major AI conference flooded with peer reviews written fully by AI,” and its standfirst says “21% of manuscript reviews for an international AI conference were found to be generated by artificial intelligence.” Shen and Wang describe that count as Pangram Labs’ finding of around 21 percent “fully AI-generated” ICLR reviews. It covers ICLR’s following review cycle with a different detector, so it supports the order of magnitude but says nothing direct about Nature Communications; Spectrum’s sentence attaches the stronger wording to the Nature Communications number. (I could read only the opening of the Nature report.)
The 1-in-40 figure is a ratio of two different measurements
The 1-in-40 figure comes from He and Bu, whose paper covers 5,114 journals and more than 5.2 million papers. Their headline measure, the “AI content proportion,” comes from a maximum-likelihood method that models the distribution of adjectives in abstracts as a mixture of human and AI text. The paper is explicit that this is a group-level estimate “rather than the directly measured percentage of AI-generated text in individual papers,” and its limitations section says the method “provides probabilistic estimates at the group level rather than definitive judgments on individual manuscripts.”
The disclosure side is a separate measurement. The authors could download full text for 164,579 papers. Among the 75,172 published since 2023, “only 76 papers (~0.1%) disclosed using AI tools for writing assistance in the methods and/or acknowledgment sections,” a rate that rose “from 0.01% in early 2023 to 0.43% in the first quarter of 2025.” Dividing the group-level estimate for that quarter by 0.43 percent gives the paper’s “approximately 40:1.” That implies an AI content proportion near 17 percent (my arithmetic, 40 times 0.43), an estimate for a group of abstracts rather than a count of papers.
Three details differ from the feature’s sentence. The period is the first quarter of 2025, not “2025.” The numerator and denominator come from different samples: 5.2 million abstracts against 164,579 downloadable PDFs, a subset in which, the authors note, “papers from high-OA publishers were more accessible for retrieval.” And only statements in the methods or acknowledgments sections count as disclosure.
The next sentence in the feature is the one I would flag: “these figures likely grossly underestimate LLM involvement today.” The disclosure ratio shows that disclosure lags detection; it does not show that the detection figures are too low. The sources themselves point two ways. Shen and Wang expect their review figure to understate AI-assisted writing because it catches only strongly machine-written text. He and Bu warn that their method “may still yield false positives, particularly when distinguishing between AI-generated text and the formulaic writing styles in certain technical disciplines or the linguistic patterns typical of non-native English speakers.” Both can be true at once, which is another reason to read each number as a statement about one method.
The ICLR acceptances were workshop acceptances, and the papers were withdrawn
The feature says an AI Scientist paper and papers from the Autoscience Institute “were accepted through double-blind peer review in 2025 by the International Conference on Learning Representations.” The primary sources are more specific.
- The Nature paper says the manuscript “passed the first round of peer review for a workshop of a top-tier machine learning conference” and adds “The workshop had an acceptance rate of 70%.” It notes that “workshops have much higher acceptance rates than main conferences,” that “Only one of three submissions was accepted,” and that the reviewers scored it 6, 7 and 6 (average 6.33). The organizers said the paper “would have been accepted in all likelihood were it not withdrawn according to our pre-established protocol,” because the team had decided in advance that “all AI-generated submissions would be withdrawn after peer review, regardless of outcome.” The paper puts the ICLR 2025 main conference’s acceptance rate at 32 percent, against the workshop’s 70.
- Autoscience’s own announcement says its system, Carl, produced papers “accepted through double-blind peer reviews at a workshop in the International Conference on Learning Representations (ICLR),” on the Tiny Papers track, and that “Autoscience has since withdrawn its accepted papers from the ICLR workshops.” It also says its team “manually edited the citations and LaTeX formatting in the papers.”
So the claim that AI-generated papers have cleared workshop-level peer review is supported. “By the International Conference on Learning Representations” suggests the main conference, and it omits that the papers were pulled.
The experiment: three reference-check counts and no denominator
What the feature reports
Skuse prompted free ChatGPT and Gemini 3 Flash, step by step, to write a paper on optical solitons (his own Ph.D. topic), under rules that barred editing the output and paid tools. IEEE Access’s managing editor, Jenny Mahoney, ran the result through checkers. The feature reports three different counts of bad references. Gemini, asked afterward, said seven “modern LLM-related research references have been fabricated to support the paper’s specific narrative.” Mahoney “picked up 12” with the STM Integrity Hub, and “another checker from MolecularConnections could not validate 24 references.” Neither paper was accepted.
The feature does not say how many references the paper contained, and the three statuses differ: “fabricated” (for LLM-related references only), “picked up,” and “could not validate.” Without the reference count and an answer key, three numbers cannot say how many references were fake or which tool was closest.
An independent benchmark: five tools on the same 104 references
A July 2026 position paper by Fidan Badalova and Philipp Mayr (arXiv 2607.22693) does have an answer key. It ran five reference-checking tools on “a manually verified test set of 104 references from three scholarly documents, including 33 problematic references.” The paper’s Table 4 gives the caught, wrongly flagged and missed counts; the flagged column is my sum of caught and wrongly flagged.
| Tool | Flagged | Problematic caught (of 33) | Real references wrongly flagged (of 71) | Problematic missed |
|---|---|---|---|---|
| CheckIfExist | 65 | 31 | 34 | 2 |
| HalluCiteChecker | 38 | 18 | 20 | 15 |
| Hallucinator | 57 | 29 | 28 | 4 |
| HalRef | 77 | 24 | 53 | 9 |
| RefChecker | 68 | 32 | 36 | 1 |
Same input, five answers: from 38 to 77 references flagged, of which 18 to 32 were truly problematic, while 20 to 53 of the 71 verified references were flagged by mistake. The authors conclude the tools are useful for screening but “the output of these tools should not be treated as a final decision,” and they call for “more transparent and multi-source detection systems.” It is a six-page position paper on three documents, so treat the rates as an illustration rather than a ranking of the tools.
A reference check you can run in a minute
To see what the simplest possible check does, I wrote one that asks three questions about every reference that carries a DOI. Does the DOI exist at all (doi.org’s handle API, so any registration agency counts)? Who registered it (Crossref, or DataCite, which holds arXiv)? And does the title registered under it share at least half its words with the citation text? It reads the reference list the publisher deposited with Crossref, uses only public endpoints, and prints just the flagged references.
# refcheck.py: check the DOIs in a paper's reference list (Python 3.10+, requests)
# usage: python refcheck.py 10.1007/s40199-025-00580-z
import re
import sys
import time
import requests
HEADERS = {"User-Agent": "reference check sketch/1.0 (+https://sxz.io)"}
STOP = {"the", "and", "for", "with", "from", "that", "this", "are", "was", "has", "have", "its", "into", "their"}
def tokens(text):
words = re.sub(r"[^a-z0-9 ]+", " ", text.lower()).split()
return [w for w in words if len(w) >= 3 and w not in STOP]
def overlap(title, citation):
"""Share of the registered title's words that appear in the citation text."""
wanted = tokens(title)
have = set(tokens(citation))
return sum(w in have for w in wanted) / len(wanted) if wanted else 0.0
def ascii_safe(text):
"""A Windows console on code page 1252 cannot print every character Crossref returns."""
return text.encode("ascii", "replace").decode("ascii")
def get(url):
for attempt in range(3):
response = requests.get(url, headers=HEADERS, timeout=60)
if response.status_code != 429:
return response
time.sleep(3 * (attempt + 1))
return response
def check_doi(doi, citation):
"""Return (status, registration agency, registered title)."""
handle = get("https://doi.org/api/handles/" + doi).json()
if handle.get("responseCode") != 1:
return "doi_not_found", None, None
agency = get("https://doi.org/doiRA/" + doi).json()[0].get("RA")
title = None
if agency == "Crossref":
reply = get("https://api.crossref.org/works/" + doi)
if reply.status_code == 200:
title = (reply.json()["message"].get("title") or [""])[0]
elif agency == "DataCite":
reply = get("https://api.datacite.org/dois/" + doi)
if reply.status_code == 200:
title = (reply.json()["data"]["attributes"].get("titles") or [{}])[0].get("title")
if not title:
return "exists_no_title", agency, None
return ("ok" if overlap(title, citation) >= 0.5 else "doi_other_work"), agency, title
def main(work_doi):
message = get("https://api.crossref.org/works/" + work_doi).json()["message"]
counts, agencies, flagged = {}, {}, []
for ref in message.get("reference", []):
citation = " ".join(str(v) for k, v in ref.items() if k not in ("DOI", "key", "doi-asserted-by"))
doi = ref.get("DOI")
if not doi:
status, agency, title = "no_doi", None, None
else:
status, agency, title = check_doi(doi, citation)
agencies[agency] = agencies.get(agency, 0) + 1
time.sleep(0.2)
counts[status] = counts.get(status, 0) + 1
if status not in ("ok", "no_doi"):
flagged.append((ref.get("key"), status, doi, citation[:70], title))
print(ascii_safe(message["title"][0][:80]))
print("references:", sum(counts.values()), "| status:", dict(sorted(counts.items())))
print("DOI registration agencies:", dict(sorted((str(k), v) for k, v in agencies.items())))
for key, status, doi, citation, title in flagged:
print(f" {key:<10} {status:<14} {doi}")
print(f" cited as: {ascii_safe(citation)}")
if title:
print(f" registered title: {ascii_safe(title[:70])}")
if __name__ == "__main__":
main(sys.argv[1])
One detail in the code is there because of a failure I hit myself: ascii_safe replaces characters that a Windows console on code page 1252 cannot print. My first version crashed on a Unicode hyphen (U+2010) in one registered title when I re-ran the published code the way a reader would, so that character now prints as a question mark.
Run 1: the article the feature links
The feature’s sentence about a fabricated reference “supposedly written by Oransky himself” links to a November 2025 article in DARU Journal of Pharmaceutical Sciences about post-publication peer review platforms. I ran the checker on the 17 references Springer deposited with Crossref (the Springer page returned only a stub to my script, so I did not compare the deposit with the page).
An expert criticism on post-publication peer review platforms: the case of pubpe
references: 17 | status: {'doi_not_found': 3, 'doi_other_work': 2, 'no_doi': 6, 'ok': 6}
DOI registration agencies: {'Crossref': 8, 'None': 3}
580_CR3 doi_not_found 10.1038/d41586-019-02054-z
cited as: 7 571 I Oransky 2019 Oransky I. A new kind of watchdog is shaking up r
580_CR4 doi_not_found 10.1007/s11948-016-9831-2
cited as: 5 1511 23 JA Teixeira da Silva 2017 Teixeira da Silva JA. The weaponiz
580_CR6 doi_not_found 10.1001/jama.1990.03440100128022
cited as: 10 1438 263 DF Horrobin 1990 Horrobin DF. The philosophical basis of p
580_CR8 doi_other_work 10.1002/jor.24888
cited as: 1 1 39 JPA Ioannidis 2021 Ioannidis JPA. A new era of hyper-skepticism
registered title: The efficacy of core decompression for steroid?associated osteonecrosi
580_CR10 doi_other_work 10.1111/febs.16646
cited as: 3 571 290 VAUX DL 2023 VAUX DL. Scientific misconduct: the elephant in
registered title: In conversation with Peter Macheroux
Reference 3 matches the feature’s description: “Oransky I. A new kind of watchdog is shaking up research. Nature. 2019;571:7.” Its DOI, 10.1038/d41586-019-02054-z, returns 404 at doi.org and at nature.com, and Crossref and OpenAlex title searches find no record under that title. The table lists all five flagged references.
| Ref | Cited as | Printed DOI | What the DOI system says | Title search (Crossref and OpenAlex) |
|---|---|---|---|---|
| 3 | Oransky, “A new kind of watchdog is shaking up research,” Nature 2019 | 10.1038/d41586-019-02054-z | Does not exist | No record under that title |
| 4 | Teixeira da Silva, “The weaponization of publishing: a case study of a predatory vigilante (PubPeer),” Sci Eng Ethics 2017 | 10.1007/s11948-016-9831-2 | Does not exist | No record under that title |
| 6 | Horrobin, “The philosophical basis of peer review and the suppression of innovation,” JAMA 1990 | 10.1001/jama.1990.03440100128022 | Does not exist | Real paper; Crossref lists it as 10.1001/jama.1990.03440100162024 (volume 263, issue 10, page 1438) |
| 8 | Ioannidis, “A new era of hyper-skepticism in science,” J Orthop Res 2021 | 10.1002/jor.24888 | Exists, but is registered to a rabbit study of core decompression for osteonecrosis (Maruyama et al., J Orthop Res 39(7):1441-1451) | No record under the cited title |
| 10 | Vaux, “Scientific misconduct: the elephant in the room,” FEBS J 2023 | 10.1111/febs.16646 | Exists, but is registered to “In conversation with Peter Macheroux” | No record under the cited title |
Three cautions apply. A bad reference is not proof of a language model: a faulty reference-manager export, a copy-paste slip or text generation can each produce one, and a DOI check cannot tell them apart. The DOI is also only one field; the other twelve references either pass (six) or carry no DOI (six: a STAT News article, four COPE web pages and a 1959 book). And I have not contacted the authors or the journal.
Run 2: a control, the reference list of a Nature paper
A checker that flagged everything would also “find” the DARU problems, so I ran the same script on the AI Scientist paper in Nature, whose 64 references are typical of an AI-research paper: arXiv preprints, conference papers, web pages and books.
Towards end-to-end automation of AI research
references: 64 | status: {'doi_other_work': 2, 'no_doi': 27, 'ok': 35}
DOI registration agencies: {'Crossref': 30, 'DataCite': 7}
10265_CR19 doi_other_work 10.2139/ssrn.4526071
cited as: Girotra, K., Meincke, L., Terwiesch, C. & Ulrich, K. T. Ideas Are Dime
registered title: <p>AI and Its Impact on Creativity and Diversity: An Empirical S
10265_CR20 doi_other_work 10.1007/978-981-99-3814-8_11
cited as: Lehman, J. et al. in Handbook of Evolutionary Machine Learning (eds Ba
registered title: Evolution Through Large Models
Of the 64 references, 27 (42 percent) carry no DOI and cannot be checked this way: GitHub repositories, system cards, conference proceedings. Of the 37 with DOIs, 35 pass and 2 are flagged, and both flags are benign. One is an SSRN record whose registered title and author list differ from the citation (the registered authors include all four cited ones, so it looks like a retitled paper). The other is a book chapter cited without its chapter title, so the title’s words are not in the citation text. Seven of the 37 DOIs are registered with DataCite rather than Crossref; they are arXiv preprints, which a checker that asks only Crossref would report as missing.
What a clean reference list does and does not show
Mahoney told Skuse that the FARS-written paper had “no fake references,” though “our team would have rejected this paper because it did have a huge percentage of references to preprints.” That is the kind of list on which a checker built on one registry struggles, and one paper is not a rate. FARS’s own paper reports what its volunteer reviewers found across 140 papers: its integrity audit “flagged problems in 47 of 282 reviews (16.7%), affecting 39 of 140 reviewed papers (27.9%),” and for “hallucinated citations (13/16)” reviewers escalated 13 of 16 mentions to genuine integrity violations. A clean list in the one paper Skuse generated is compatible with hallucinated references elsewhere in the same system’s output.
The second limit is the one the Wikipedia volunteer names. Ellia Lebleu says that inventing a citation outright is “much rarer today because of retrieval-augmented generation,” and that “More often, it finds a real source but gets its content completely wrong. AI citations that don’t actually support what is being written are extremely commonplace.” A check that confirms a source exists, however carefully, cannot see that failure. This is where the arms-race framing is most accurate: the existence checks that caught Skuse’s free-tool paper test the layer that retrieval-based systems are best at passing, and the next layers (does the source say this, did the experiment run) need reading, code and logs, which is what FARS’s reviewers were asked to do.
The Wikipedia numbers rest on a commercial detector
The Wikipedia half of the package rests on a different kind of number. Lebleu says she worked with the AI-detection company Pangram: “In late 2025, it flagged perhaps 20 percent of new articles and drafts as AI-generated. In early 2026, that went down to 15 or 16 percent.” She tested it on pre-ChatGPT Wikipedia text “and found no false positives,” and says “we don’t yet know whether that’s because our recent policies have had an effect or because models are getting more powerful and Pangram hasn’t had time to adapt.” That is a careful statement from a participant, and it is a single-source figure from a commercial detector that I could not re-run.
What holds up, and five questions for the next AI-slop statistic
The direction of travel is consistent across independent methods: abstract vocabulary shifted sharply after 2022 (in my PubMed counts as well as in Siler’s paper), a classifier and a commercial detector both find a rising share of reviews flagged as AI-generated, disclosure is rare, and reference checks catch careless fakes. Nothing here shows the numbers are wrong. What it shows is that a headline statistic compresses a measurement into a claim about the world, and the compression is where the errors creep in. Before repeating the next one, ask:
- What was counted? Four publishers or all of science; 406 sampled reviews or every review.
- What counts as a hit? A word-rate threshold, a classifier label or a disclosure statement, and “evidence of influence” versus “written by AI.”
- Is it per paper or per group? A group-level estimate cannot be turned into “40 papers.”
- Was the detector tested on text unlike its training data? Synthetic text from one model is easier to separate than real, human-edited text.
- What is the denominator? “Picked up 12” and “could not validate 24” mean little without the reference count and an answer key.
Related reading on sxz.io: how a submission cap tries to control the flood in arXiv’s two-a-month rule, and why “verified” becomes a statement-reading problem in the claimed refutations of the Exponential Time Hypothesis.
What I could not check
- Siler’s full text (PNAS bot check): the threshold for “evidence,” the uncertainty, and the per-publisher rates.
- The rest of Nature’s ICLR report (only the opening was readable to my script) and the Pangram analysis behind its 21 percent.
- Skuse’s two papers and each checker’s per-reference output. They are not published, so the 7, 12 and 24 cannot be re-derived.
- The Lancet audit of fabricated citations, which Badalova and Mayr summarize as one fabricated reference per 2,828 biomedical papers in 2023 and one per 277 in early 2026. I did not read it.
- Whether the DARU reference problems come from a language model, and whether Springer’s page matches the Crossref deposit.
Sources
- IEEE Spectrum: editor’s note, November 1, 2026; Skuse feature, October 27; Lebleu Q&A, October 21.
- Siler, PNAS 2026, doi 10.1073/pnas.2605754123 (abstract via PubMed).
- He and Bu, PNAS 2026, full text on PubMed Central.
- Shen and Wang, arXiv 2602.00319 (HTML version 2).
- Naddaf, Nature news, November 27, 2025.
- Lu et al., Nature 2026, Towards end-to-end automation of AI research; Autoscience Institute, Meet Carl.
- Tang et al., FARS, arXiv 2606.31651.
- Badalova and Mayr, Detecting Hallucinated and Suspicious Citations, arXiv 2607.22693.
- NSF NCSES, Publications Output: U.S. Trends and International Comparisons.
- Reference-check inputs: Crossref record for the DARU article, the article, STM Integrity Hub.








No Comment! Be the first one.