How AI-generated reviews compare with expert human peer reviews, and what the comparison does and does not settle. The anchor is Gary King and Steven Worthington’s 2026 head-to-head experiment: AI reviews beat unusually good paid human reviews on every criterion they measured. 1
King and Worthington’s experiment
The corpus is 124 paper submissions and 134 human reviews from rapidPeer, an IQSS service that gives university faculty double-blind reviews of drafts before journal submission. Reviewers are paid, chosen by field experts from institutions other than their own, and instructed to read the whole paper and be comprehensive and constructive. Of 74 authors surveyed a week after delivery, 88% rated their review high or very high quality; the median author rated willingness to recommend the service 10 out of 10; former editors of major journals called the reviews among the best they had seen. In the authors’ words: “We know of no collection of human reviews written under better conditions.”
The AI side produced one review per submission per condition with Claude Opus 4.8 at maximum reasoning: a matched condition (anonymized manuscript plus the verbatim rapidPeer guidelines, no tools) and a tuned condition (a purpose-built agent prompt with web retrieval, a mean of 12.4 searches per review). Judges were GPT-5.5, chosen from a different model family so generator and judge share no lineage. Applicable judgment topics were locked per paper before judging, every comparison ran in both presentation orders, and a win was credited only when both orders agreed. Three former editors of leading journals judged 10 blinded pairs without knowing they were comparing AI against human reviews.
Results
- Overall, the AI review was preferred in 132 of 132 decided pairs by the AI judge and in 26 of 30 pairs (87%) by the human editors.
- By review area (AI judge): 100% for sources, weaknesses, and methods, 99% for figures and tables, 92% for theory and idea, framing, and communication.
- Human editors agreed with the AI judge’s verdict 87% of the time — more than they agreed with each other (80%).
Robustness
The paper reports ten categories of checks. The ones that matter for how much the headline can carry:
- Not length: on the 44 pairs whose AI-to-human word-count ratio fell in 0.7–1.4, the AI review was preferred in all 44.
- Not weak competition: the AI wins after post hoc selection of the top human reviews.
- Grounded judging: repeated with the full manuscript and instructions to check each review’s claims against it, the human review was never preferred in both presentation orders (133 of 134 matched pairs and 132 of 134 tuned pairs preferred AI).
- Not style: rewriting human reviews in AI style did not improve their judgments.
- Not training-data recency: the advantage does not differ between 2025–26 submissions and those from 2022 or earlier, and matched-condition AI reviews cited work published after the submission in only 2 of 124 cases.
- Accuracy: AI reviews cited no fabricated sources and 5.5% of their claims about the manuscript were judged incorrect, against 15.9% for the human reviews. In a planted-contradiction test the judge flagged all 258 false sentences and none of the 258 accurate ones.
- Cross-family replication: across 416 comparisons with two model families writing and judging, the AI review was preferred in 97.1% of cases under the Opus judge and 94.2% under the GPT judge.
- Tools matter for citations: matched-condition reviews without web access won the “sources” area only 62% of the time; with a browser the rate rose to 100%.
- AI reviews rated higher in every subgroup: manuscript novelty, research style (statistical, mathematical, qualitative), faculty rank, career length, and the tier of the journal the paper eventually reached.
What it does not settle
- The results speak to constructive advice for improving papers, not to how editors should make publication decisions — the paper is explicit that it did not measure the second question.
- Whether the reviewed papers actually improved after AI review, and the collective consequences of switching to AI reviewing, were not measured.
- Homogenization risk: AI helps individuals but converges on conventional ideas (Hao et al. 2026; Wenger and Kenett 2026), and reviews may push individual papers and the literature toward homogeneity. The heterogeneity of human reviewers who disagree with each other is partly load-bearing.
- Full replacement undermines itself in the limit: if the journal’s only review is an AI review authors can run themselves before submitting, it tells the editor little the author has not already addressed. At ICML 2026, 22.5% of reviewers assigned to a no-AI policy reported using it anyway.
Two evaluations of one artifact
King and Worthington measure how good AI reviews are on average. The same week’s Saez-Zucman episode shows what happens when the evaluation design is left to the evaluator: the anonymous audit, Cowen’s promotion of it, and Furman’s counter-evaluation are three AI-mediated judgments of the same material that split over grading rules and denominators rather than over the underlying facts — Furman’s re-checkers found the audit’s “individual facts mostly held up” while its “counting and grading did not,” and Furman closed with “two wrongs make a, well, nothing.” AI critique is good enough to be evidence; the instrument’s grading rules still decide the headline. 2 3
Connections
- saez-zucman-audit — the same week’s episode where AI evaluations of one research program split over grading rules; King and Worthington supply the evidence that the tool class is strong, the counter-evaluation the reminder that design decides the headline.
- adversarial-collaboration — locked topics, blind pairs, presentation-order symmetry, and a cross-family judge are the experimental-design version of adversarial collaboration’s neutral arbiter.
- llm-replication-economics — measured review quality is the enabling input for corpus-scale machine audit and replication work.
Sources
- Gary King, Steven Worthington 2026 — AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices
- Tyler Cowen 2026 — AI dominates human peer reviews
- 2026 — Furman thread: an AI evaluation of the Saez-Zucman audit