Harvard study: Claude's peer reviews beat paid human experts, editors pick the AI review 87% of the time

Peer review is one of the slowest and most complained about parts of science. A new working paper from Harvard argues that, at least for one important job, the machines have already won. Gary King and Steven Worthington of Harvard's Institute for Quantitative Social Science (IQSS) let Claude Opus 4.8 review the same 124 manuscripts that paid human experts had already reviewed, then asked both an AI judge and three former journal editors which review was better. The AI review won by every measure the authors could think of.

The paper is called "AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices". It is posted as a PDF on Gary King's Harvard site, with a version dated 22 October and an update dated 25 October 2026. It has not itself been through formal peer review yet, which is a little ironic, and it is worth reading with that in mind. But the design is careful, and the numbers are hard to ignore.

Not a test against weak reviewers

Most earlier studies compared AI with ordinary journal reviews, which vary a lot in quality. King and Worthington went for the strongest human competition they could find. Their reviews come from rapidPeer, a service built at Harvard IQSS and later expanded to other universities. Faculty send in drafts before submitting them to a journal and get a double blind review from an expert at another institution.

These reviewers are paid, and paid well, according to the paper. They are asked to read the whole paper, to write at least one and a half to two pages, and to be constructive instead of hunting for one fatal flaw so they can stop reading. They also know that a good job means more paid work later. The results show: 88 percent of the 74 authors who answered a survey a week after delivery rated their review as high or very high quality, and the median author scored their willingness to recommend the service at 10 out of 10. Former editors of major journals told the researchers that these reviews were among the best they had seen.

The corpus covers every eligible submission from 2019 to 2026: 124 papers and 134 human reviews, because ten papers got two independent reviews. The papers range from statistical analyses of data to formal mathematical models and purely qualitative studies. Many of the reviews were written before ChatGPT existed, so the authors are confident they were written by people.

How the experiment worked

For each paper, Claude Opus 4.8 wrote one review at its maximum reasoning setting, under two conditions. In the "matched" condition it got the anonymized manuscript with figures and tables plus the exact rapidPeer reviewer guidelines, and no tools. That is roughly what the human reviewers got. In the "tuned" condition it got a purpose built agent prompt and web retrieval. Every tuned review used the web, with 12.4 searches per review on average. The tuned prompt was written before any judging and never adjusted to the judges' scores.

The judging was done in two ways. The main judge was OpenAI's GPT-5.5, chosen so that the reviewer and the judge come from different model families. First the judge decided which review topics even applied to each paper, so a paper without figures was not scored on figures. Then it compared the two reviews in three ways: an overall ranking, a head to head ranking in seven areas taken from the rapidPeer guidelines (theory and idea, methods, weaknesses, framing, communication, figures and tables, and sources), and a separate 0 to 10 rating per area. Each comparison ran in both orders, and a win only counted if the same review was preferred both times.

The second judgment came from people. Three former editors of leading journals in similar fields each looked at ten randomly chosen pairs, side by side, without being told that one review in every pair came from an AI. They only learned that afterwards.

The results

The headline numbers are blunt. The GPT-5.5 judge preferred the AI review in 132 of 132 decided pairs. The three editors preferred it in 26 of their 30 judgments, which is 87 percent. By area, the AI won 100 percent of decided pairs on sources, weaknesses and methods, 99 percent on figures and tables, and 92 percent on theory and idea, framing and communication.

One detail I find more telling than the win rate: the human editors agreed with each other only 80 percent of the time, but they agreed with the AI judge 87 percent of the time on average. In other words, the AI judge was not an outlier with strange taste. It lined up with expert human judgment at least as well as the experts lined up with each other.

On the 0 to 10 scale, human reviews averaged about 6.0, and the AI reviews scored around 2.6 points higher. That gap was about the same in the ten papers the editors saw and in the other 114.

The robustness checks

A result this one sided invites suspicion, and the authors spend a good part of the paper on it. They report ten groups of robustness checks. The ones that matter most to me:

• Length. Maybe the AI just writes more. On 44 pairs where the AI and human reviews had a similar word count, the AI still won all 44.
• Style. Maybe an AI judge simply likes AI sounding text. The authors rewrote the human reviews in the AI's style. That did not help the humans at all.
• Accuracy. A review can be fluent and wrong. The researchers checked claims against the manuscripts. AI reviews cited no fabricated sources, and 5.5 percent of their claims about the manuscript were judged incorrect, compared with 15.9 percent for the human reviews. That is about a third of the human error rate.
• Can the judge spot errors at all? They planted one accurate and one subtly false sentence in versions of every review. The judge flagged all 258 false sentences and none of the 258 accurate ones.
• Hindsight. The AI's training data might include work published after a paper was submitted. The advantage was the same for 2025 to 2026 submissions as for papers from 2022 or earlier, and AI reviews without web access cited later work in only 2 of 124 cases.
• Other models. A replication with a new prompt, a new rubric, reviews from both Claude Opus and GPT, and judges from both families ran 416 comparisons on 104 human reviews. The AI review won 97.1 percent of the time with the Opus judge and 94.2 percent with the GPT judge.
• Subgroups. AI reviews scored higher in every group the authors could build: by novelty, by research style, by the author's rank and career length, and by the quality of the journal where the paper was finally published.

There is one interesting weak spot. Without web access, the matched AI reviews won only 62 percent of pairs in the sources area. Once the model could browse, as the human reviewers could, that rose to 100 percent. For citations and related work, the tools matter as much as the model.

What the authors suggest journals do

The paper is careful to separate two jobs of peer review. One is helping authors improve a paper. The other is helping editors decide what to publish. The evidence only speaks to the first. Still, King and Worthington sketch five policy options for journals, from mild to radical.

First, journals could ask authors to run repeated AI reviews themselves before submitting, a kind of self service revise and resubmit. Second, journals could publish an official AI review prompt that encodes their own standards. Third, they could require that an AI review made with that prompt is submitted with the paper, together with a letter from the author explaining where they disagree with it. Fourth, some editors might drop human reviews entirely. Fifth, and in the authors' view more realistic, editors could skip routine human reviews and decide based on the author's dossier, a second AI review from a prompt the editor keeps private, and a human reviewer only for hard cases.

The authors note that this shift may already be happening without anyone deciding it. At ICML 2026, which randomly assigned reviewers to different AI policies, 22.5 percent of the surveyed reviewers who were told not to use AI said they used it anyway.

The risks the authors themselves raise

This is not a cheerleading paper. The authors spend real space on the downsides. Their biggest worry is sameness. When many researchers brainstorm with AI, even with different models, they tend to arrive at similar, conventional ideas. AI reviews could push papers and whole fields in the same direction. The messy disagreement between human reviewers is annoying, but it also protects unusual ideas.

There is also a gaming problem. If a journal publishes its AI prompt and authors revise until that prompt is happy, the review tells the editor little that is new, and authors may start optimizing for what the model rewards rather than for good science. The authors also mention that AI can become "absurdly pedantic" as a paper improves, with their example being a complaint that the plus sign was never defined. Their answer is to keep a human editor in charge of the final decision, use hidden prompts and manipulation checks, and keep studying the cases where authors and editors disagree with the AI.

Why it matters

Peer review runs on unpaid or poorly paid goodwill, and it is slow. Months of waiting for a review that turns out to be shallow is a common experience in many fields. If an AI review is now more useful than the best paid human review, the cost and the waiting time for good feedback drop close to zero. That changes the incentives for every researcher with a draft on their laptop, whatever journals decide.

It also matters beyond academia. Reviewing a long, technical document, finding weak arguments, checking claims against the text and suggesting better figures is exactly the kind of work many companies do every day, from legal teams to engineering design reviews. This study is one of the clearest head to head tests so far that a frontier model can do that job better than strong human experts, at least when the task is giving constructive feedback.

My take

I find this study more convincing than most "AI beats humans" headlines, because the human side was strong and the authors tried hard to break their own result. Length, style, accuracy, hindsight and model choice were all tested, and the result held every time. The 87 percent from blind human editors is the number I would quote, because it does not depend on trusting another model as judge.

But I would keep three caveats in view. The editor test covered only ten pairs, judged by three people, so that 87 percent rests on 30 judgments. The study measures help for authors, not good publication decisions, and the authors say so clearly. And the most important risk is not a single bad review but a whole system that slowly converges on the same safe ideas.

My practical conclusion: if you write anything long and serious, a paper, a report or a technical spec, running it through a strong model as a critical reviewer before a human ever sees it now looks like basic hygiene. Give it clear review instructions and web access for the sources, then decide yourself which advice to take. The human role moves from finding the typos and gaps to judging what actually matters.

What happens next

The authors say the latest version of the paper and its supplementary appendix will be available at GaryKing.org/peerless, including the exact reviewer guidelines and prompts, so other researchers can rerun the test in their own fields. They explicitly call for repeats with each new generation of models. The more interesting follow ups will be in fields outside quantitative social science, and studies that track whether papers revised with AI feedback actually end up better and more original, not just more polished.

Sources: King and Worthington, "AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices", Harvard IQSS (PDF, 25 October 2026 version); HitlistAI summary (24 October 2026).