GPT-6 Astra solves 49 of 50 unpublished theoretical physics problems

A group of more than thirty theoretical physicists has put frontier AI agents through one of the hardest exams the field can write, and the results landed harder than most of them expected. In a paper called TheoPhysEval, posted to arXiv on October 9, 2026, the team gave five off-the-shelf agents a suite of 50 unpublished, research-level problems with verifiable answers. The best agent, GPT-6 Astra running inside OpenAI's Codex harness, got the final answer right on 49 of 50, a 98 percent pass rate, at an average cost of under 3 dollars per problem.

The number alone would be news. The story behind it is stranger. On his blog on October 14, coauthor Suvrat Raju of the International Centre for Theoretical Sciences explained that the first round of grading put Astra only in the late eighties. When the team dug into the apparent mistakes, the model kept insisting that the expert reference answers were wrong. In multiple cases, the experts went back to their own calculations and found subtle errors. Once the answer key was fixed, Astra's score climbed to 49 of 50. Economist Alex Tabarrok summed it up on Marginal Revolution on October 15: the human answers were supposed to test the AI, and instead the AI ended up testing the humans.

This article explains what TheoPhysEval measures, how the agents compared, what the runs cost, why the authors think the models have memorized much of the physics literature, and what the result does and does not mean for physics research.

What TheoPhysEval actually is

Most AI benchmarks in physics are built from textbook exercises, olympiad problems or graduate qualifying exams. TheoPhysEval is different by design. The organizers asked working experts to contribute problems that met three strict rules: the problem had to relate to current research, it had to have a known solution that is easy to verify but hard to guess, and the solution could not be available anywhere in the published literature.

That last rule is the expensive one. Contributors either had to hand over a calculation from their private notes that they had never bothered to publish, or adapt a problem from one of their recent papers by changing parameters so that the new answer could not be guessed from the old one. The paper describes these as "turn the crank" research problems: the result is technically novel and sometimes even publishable, but it is not a breakthrough. It is the kind of exercise a supervisor might give a new graduate student to learn a technique.

According to the paper, a typical problem could be solved by a world expert in that niche in about a day or less, but would take considerably longer for experts in neighboring specialties. Raju put it more bluntly on his blog: he does not know of any individual who could solve a significant fraction of the collection without AI help. He can solve the problems he contributed and understands a few others, and most of the rest do not even make sense to him.

The authors list affiliations including ICTS, the Tata Institute of Fundamental Research, the University of Texas at Austin, Kavli IPMU in Tokyo, Duke, the University of Pennsylvania, Harvard, KU Leuven and Google DeepMind. The full problem set and the evaluation configuration are public on Hugging Face, so anyone can rerun the test.

The suite spans eleven areas. Quantum field theory contributes the most problems (7), followed by gravitation and cosmology (6). Classical physics, stochastic and biological physics, quantum phases of matter, scattering amplitudes, and holography and entanglement each supply five. Electronic structure theory has four, nonequilibrium statistical mechanics and lattice theories three each, and particle physics two. Example problems range from the Page time of a black string in holography to a fermion parity pump in condensed matter and perturbations in a boundary layer over a flat plate.

How the agents were tested

The team ran each agent on two tracks. In the realistic track, agents could search the web, read papers and write and run scripts for numerical and symbolic work, which is how a physicist would actually use an AI assistant. In the isolated track, web search was switched off apart from calls to the model provider, so agents had to rebuild the needed formulas and code from what they already knew.

Grading was deliberately mechanical. Because symbolic answers can be written in many equivalent forms, the authors converted every expression into numbers by supplying concrete values for the physical parameters, including some irrelevant ones. The agent had to produce a floating point value or a sequence of integers that could not be guessed. That avoided using another language model as a judge, which the authors saw as an extra source of error.

The runs used the open Harbor framework, the same tooling behind Terminal-Bench, on small Google Cloud virtual machines with 2 vCPUs, 4 GB of memory and 10 GB of storage per task. There was no cap on tokens or spending, but any task that did not finish within 8 hours counted as a failure. Reasoning effort was set to maximum for every model except Gemini 3.8 Flash, which ran on "high", its top setting.

Five agent setups were compared: GPT-6 Astra in Codex, Claude Opus 5.5 in Claude Code, Gemini 3.8 Flash in Antigravity, the open weight DeepSeek V4.1 Flash in Codex, and the open weight GLM 5.3 in Claude Code.

The scores

On the realistic track, Astra solved 49 of 50. Opus 5.5 came close with 46. After a clear gap, Gemini 3.8 Flash solved 34, DeepSeek 27 and GLM 21.

On the isolated track, Astra and Opus tied at 46 of 50 each, which is 92 percent. Gemini solved 31, GLM 26 and DeepSeek 25. GLM actually did better without the web, and the paper traces that to a set of linked electronic structure problems where the realistic run made an early sign error that propagated through several answers.

The authors are careful to note that each agent ran the full suite only once, to save time and money. With nonzero sampling temperature, scores could move by a few problems on another run, so small gaps such as Astra versus Opus on the isolated track should not be overread.

The single problem Astra missed on the realistic track was an inverse problem: reconstruct an autonomous differential equation that produces a given curve. No model solved it. Raju describes it as a trick prompt that needs an inspired guess and a little rule bending, which current agents cannot yet do. Every set of problems solved by another model, and by Astra on the isolated track, was a strict subset of what Astra solved with web access.

When the answer key was the problem

The most striking detail is in section IV.2 of the paper. The authors write that they initially assessed Astra as having made errors on several additional problems. After closer examination and consultation with the contributing experts, they found subtle errors in the original solutions. In every one of those cases, once the experts corrected their work, Astra's answer turned out to be right.

Raju's blog adds the human side. After the team saw the results, he wrote, he could not sleep properly for a couple of nights, and he had not expected this day to come so soon. Six months earlier, he had prompted a frontier agent with one of his own unpublished problems just for fun. It got the answer wrong but made enough progress to solve it after a few hints. By the time the collection was finished, the agents had improved so quickly that the organizers kept urging contributors to push up the difficulty.

This matters beyond bragging rights. Benchmarks normally treat the reference answer as ground truth. TheoPhysEval shows that at this level, a model that pushes back on the key can be more reliable than the experts who wrote it. That is a reason for both excitement and caution: it suggests real verification value, and it also means future benchmark builders need a process for handling disagreements between models and humans.

Cost and efficiency

The paper reports costs inferred from token usage and list API prices. A full realistic run cost 116.28 dollars for Astra, about 2.33 dollars per problem. Opus 5.5 was the most expensive closed model at 240.43 dollars. Gemini cost 95.01 dollars and DeepSeek 37.42 dollars. GLM ran on a coding plan that cost 180 dollars in total for both tracks, though at API prices the realistic track alone would have been 311.71 dollars.

Token counts tell a similar efficiency story. Astra used about 37 million tokens in total on the realistic track. Opus used about 211 million, Gemini about 607 million, and the two open weight models around a billion each. Astra reached the highest score while reading and writing far less than its rivals, a sign that its harness and reasoning took more direct paths to the answer.

For a physicist, the headline is simple. Calculations that would occupy a specialist for a day can now be done by an agent for the price of a coffee, as long as the problem is well posed and the answer can be checked.

Did the models just memorize physics?

One finding cuts against the hype. Accuracy barely changed between the realistic and isolated tracks for most agents. Many problems depend on detailed formulas from specific papers that were not supplied in the prompt. One problem, the authors note, would be impossible for a human without looking up the background paper just to understand the notation, yet agents solved it offline.

The authors conclude that leading models have effectively memorized large parts of the theoretical physics research literature and can reproduce key formulas without looking them up. That is impressive recall, and it is also a caution. Strong offline scores do not prove deep originality. They show that the models carry an enormous library of known techniques and can apply them accurately to new parameter settings.

The paper is also open about other limits. Some contributors used AI tools while preparing their own solutions, so the problems are not strictly "untouched by AI" and some data leakage cannot be ruled out. Two prompts had underspecified conventions, which the grading key had to accommodate.

What it means for physics research

The authors write that frontier agents can now solve verifiable problems in theoretical physics with great accuracy, that their breadth exceeds that of any individual, and that their calculational skill now matches world experts even in narrow specialties. They expect this to accelerate discovery and possibly to crack long standing problems.

They also raise structural worries. Problems like the ones in the suite used to be ideal first projects for early career researchers, a way to learn techniques and earn an original result. The paper says it is no longer appropriate to assign such problems to starting researchers unless the goal is purely to learn. Theoretical physics has historically been cheap compared with experimental physics, which helped researchers in less wealthy institutions compete. If harder problems start to need large compute budgets, access and equity could suffer.

Finally, the team stresses what agents still cannot do. Solving a concrete calculation is only the last step of physics research. Before that comes choosing a theory, asking the right question and turning it into something computable. When a problem proves too hard, a good researcher finds a related and equally interesting one. In the authors' experience, current agents are not good at changing direction like that, and they do not share human "taste" in what makes a problem interesting. When asked to invent new problems, they focus on details a human would find dull.

The paper closes with a twist on the famous physics motto "shut up and calculate," which it credits to N. David Mermin. If AI now does the calculating, the authors suggest, physicists may need to "stop and think."

Why this matters for AI watchers

TheoPhysEval fits a pattern from 2026. The paper itself points to recent agent results in mathematics and expert use of agents in physics papers. What is new here is a careful, public, reproducible test on unpublished research problems from many subfields, with a cost breakdown and failure analysis. The gap between closed frontier agents (Astra and Opus) and open weight models (DeepSeek and GLM at roughly half the score) is also a useful data point for anyone tracking the open versus closed race.

The practical takeaway is narrower than "AI does physics now." Agents have become extremely strong at well posed, checkable calculations, cheap enough to use routinely, and good enough to catch expert mistakes. Breakthroughs still need humans to pick the problems.

Sources: TheoPhysEval paper on arXiv (2610.16733), Suvrat Raju: Evaluating Agents on Unpublished Research Problems, Marginal Revolution: Astra Held Its Ground (Truth), TheoPhysEval dataset on Hugging Face, INSPIRE record.