AI finds major errors in 6% of arXiv math papers

For years the honest answer to "how many published math results are actually wrong?" was a shrug. Nobody had the time to check thousands of papers line by line. On 15 October 2026 the team behind MathArena, a benchmark project created by the SRI Lab at ETH Zurich and INSAIT, published a large scale attempt to measure it with an AI agent. They pointed OpenAI's GPT-6.1 Sol at hundreds of mathematical papers on arXiv and asked it to do one very specific thing: find a counterexample that breaks one of the paper's main theorems.

The headline numbers are striking. Across 656 papers posted between January 2025 and July 2026, the model reports major mistakes in theorem statements in 6% of papers and minor mistakes in a further 30%. In the newest batch, 137 papers from August and September 2026, the share of papers with major mistakes fell to 1.5%. That is a drop of roughly 75% in two months, during exactly the period when AI acknowledgments started showing up in more than a quarter of new math papers.

The study was written by Jasper Dekoninck, Tim Gehrunger, Kári Rögnvaldsson, Chenhao Sun and Martin Vechev. It is careful about what it can and cannot claim, and that care is the most interesting part of the story. Here is what they did, what they found, and why it matters well beyond mathematics.

Why arXiv errors matter

arXiv is the main preprint server for mathematics, physics and computer science. Papers there are not peer reviewed before they appear. Many are later published in journals, but plenty of researchers read, cite and build on the arXiv version long before any referee has looked at it. If a central theorem in that version is false, follow up work that relies on it can inherit the error.

MathArena frames the problem directly: mathematicians spend a lot of time reading these papers, and "pollution in this source can therefore have significant consequences for the mathematical field." The team wanted answers to two questions. How many arXiv papers contain major mistakes that make their main results invalid? And has the recent wave of AI use in research made that better or worse?

A narrower task on purpose

The obvious approach would be to give an agent a paper and ask it to find every mistake. The authors explain why that does not work well yet. In their experience, current agents often overestimate how important the problems they find are, so a small typo can be labeled a major error. Checking long proofs requires tracking complicated dependencies across many pages. And judging how serious a gap in a proof really is can be extremely hard, because one broken lemma might be trivial to route around while another sinks the whole result.

So they chose a cleaner question. Instead of auditing every proof, the agent only tries to find counterexamples to the main theorems. A counterexample is concrete. Either it satisfies all the stated assumptions and violates the conclusion, or it does not. That makes the results easier to verify and to categorize, and it still catches the most consequential kind of error: the kind where anyone who uses the theorem later will also reach a false conclusion.

Every counterexample has to be placed in one of three buckets.

• Edge: the authors clearly meant the statement differently, but a typo or a missing, very common assumption makes it false as written.
• Minor: the statement is false in some cases, but it can be saved with small extra assumptions or corrections that do not change the main contribution.
• Major: the statement is fundamentally flawed and can only be salvaged by significantly limiting or changing its conclusion.

The examples in the post make the difference tangible. An edge case was a paper that accidentally dropped an absolute value in one equation while the correct formula appears elsewhere. A minor case was an optimization algorithm with an incorrect lower bound that can make it throw away the best solution and declare a worse one optimal. Major cases included a paper that claims to guarantee a feasible traffic schedule under certain conditions, where counterexamples with infeasible schedules exist, and a paper claiming that a GRPO based training algorithm improves monotonically under a mild condition, which fails because of a mismatch between the reward and the training objective.

The four stage pipeline

The team built a pipeline with four steps. Each one ran as a Codex agent using GPT-6.1 Sol at high reasoning effort.

1. Theorem extraction. From the raw TeX source of the first arXiv version, the model picks at most five main theorems that are most likely to be reused, and writes each one out with all of its assumptions and definitions so it can stand on its own. 2. Counterexample search. For each extracted theorem, the model looks for counterexamples and labels them edge, minor or major. 3. Verification. A separate pass rechecks every report and reconsiders the label. This stage changed the severity of only 2% of the mistakes. 4. Follow up versions. If the paper has later versions on arXiv, the model checks whether the mistake has been fixed. Where a later version exists, it reduces the severity of major errors in 50% of cases, and the frequency of any error drops by 28%.

The whole run cost about $2,500 in API credits. For a survey of nearly 800 papers, that is cheap. A human team doing the same audit would need far more time. The prompts for all four stages are published in full on the MathArena page, so other groups can rerun or criticize the method.

What the numbers say

For the older set, the model reports major errors in 6% of papers and in 2.3% of extracted theorems. According to the authors, feedback from several mathematicians suggests this is roughly what you would expect for material that has not been peer reviewed. The bigger surprise is the minor category. Another 30% of papers are reported to contain minor errors. Even with the deliberately narrow focus on counterexamples to main results, a large slice of arXiv math contains statements that are worth correcting.

Then comes the trend. In August and September 2026, the share of papers with major errors fell to 1.5%. MathArena also split papers by whether they include an AI use statement and found a similar decrease, from 5.4% to 1.8%. The authors read this as a sign that more AI use in research mathematics has reduced the error rate. Mathematicians they spoke to expect the same: at the very least, AI is now good at checking papers, which can lower mistakes while also increasing output.

The caveats are real

MathArena does not oversell the result, and neither should anyone sharing it. The team names several confounding factors. The mix of papers posted to arXiv could have shifted. And there is a more uncomfortable possibility: GPT-6.1 Sol might simply be worse at finding mistakes in papers that were written with its own help. If authors already ran a similar model over their drafts, the obvious errors that model can see would be gone, but errors it systematically misses would remain.

There is also the question of false positives. To test the pipeline, the authors first spot checked 20 errors by hand and found no mistakes in how they were categorized. Then they reran the search stage with a different frontier model, Claude Opus 5.5 at high effort, on all 75 theorems that GPT-6.1 Sol had flagged as having major errors, plus 75 randomly chosen theorems from each of the other three categories.

Opus 5.5 turned out to be more lenient overall, but it agreed on 78.7% of the cases Sol called major. The team manually investigated the six errors Opus downgraded from major to edge or none, and found that three of those six were in fact correctly labeled by Sol. When Sol was rerun on the 75 major errors, it kept 90% of them as major and downgraded the rest. Putting it together, MathArena estimates that a bit less than 90% of Sol's major findings are correct, which means roughly 10% false positives. That is good enough for a statistical picture of arXiv as a whole. It is not good enough to send automatic accusations to individual authors.

Why they will not email the authors

The team released all of its raw data with the post, but it says it will not contact authors whose papers likely contain mistakes. The first reason is that roughly one in ten major reports is probably wrong. The second reason is more philosophical: the authors believe that forcing other people to read and verify AI written text is counterproductive.

Their suggestion is that mathematicians run agents on their own past papers instead. They acknowledge that this is emotionally taxing. When a mistake turns up, a researcher has to accept that an autonomous system found it quickly in work that may have taken months or years. If the correction came from outside researchers instead, the authors write, that "might lead to understandable pushback."

Part of a bigger shift in how research is checked

The MathArena study did not appear in isolation. One day earlier, on 14 October 2026, Google Research made Deep Review public, an agentic system that checks technical documents for logical and mathematical problems. It grew out of Google's Paper Assistant Tool, which gave private feedback to authors of more than 16,000 opted in submissions at STOC, ICML, NeurIPS and ICLR. Google describes the core problem as a "validation bottleneck": AI helps produce papers and documents faster than humans can rigorously review them. Every Deep Review user gets two free document reviews per week, with more available through a Gemini API key.

On 15 October, the Simons Foundation announced a $15 million effort together with Anthropic and OpenAI to help mathematicians check AI assisted results, learn new tools and develop shared standards for AI use. The Verge reported that OpenAI is contributing $5 million of that.

Put those three announcements side by side and a pattern emerges. The first wave of AI in science was about generating things: drafts, proofs, code, experiments. The next wave is about checking them. If an agent can audit hundreds of papers for a few thousand dollars, verification stops being the scarce resource it has always been.

What this means for you

If you write technical work of any kind, the practical lesson is simple. Before you publish, run a strong model over your own main claims and ask it to break them, not to praise them. Treat what it finds as leads to investigate rather than verdicts, since even the best pipeline here is wrong about one time in ten.

If you read research, keep two ideas in mind at once. A 6% rate of major errors in unreviewed math papers is a reminder that "it is on arXiv" is not the same as "it is correct." And the sharp drop in the newest papers is promising, but it comes with a real asterisk until someone tests whether models are blind to errors in text they helped write.

The bottom line

MathArena used GPT-6.1 Sol to look for counterexamples to the main theorems of 793 arXiv math papers for about $2,500. It found major errors in 6% of papers posted before August 2026 and in 1.5% of those posted in August and September 2026, with an estimated 10% false positive rate. AI is becoming a serious reviewer of research, and the most useful thing it may do for science is not to write more papers but to catch what is wrong in the ones we already have.

Sources: MathArena: Searching for errors in mathematical papers with GPT-6.1 Sol, Google Research on Deep Review, Simons Foundation announcement, The Verge on OpenAI's contribution, r/singularity discussion.