SemiAnalysis GDPx: the best AI model scores just 12% on real expert work

Most AI benchmarks today have a problem: the best models already score close to the top. When everything sits at 90 percent or more, a leaderboard stops telling you much about what a model can actually do for a business. SemiAnalysis, the research firm best known for its work on chips and data centers, wants to fix that with a new benchmark called GDPx. Its first results, published on 2 November 2026, are sobering. Across 206 tasks from 11 of the most valuable professional domains, the best model in the test, Claude Opus 5.5, scores just 12.0 percent.

That number does not mean frontier AI is useless. It means GDPx was built on purpose to be hard, long and close to the work that real experts get paid for. Below is what SemiAnalysis measured, how the test works, what the results say about each model and why the safety findings may be the most interesting part of the whole report.

The three things you need to know

1. The scoreboard is low, and it is close. Claude Opus 5.5 leads GDPx with an overall score of 12.0. Gemini 4 Carbon follows at 10.9 and GPT-6 Astra at 9.6. Behind them come Muse Wonder Full at 9.0, Claude Fable 5.1 at 8.5, GPT 6.1 Sol at 8.4 and Claude Sonnet 5.5 at 8.2. The top of the table is separated by just a few points, and the absolute level is low for every model. SemiAnalysis notes that Gemini 4 Carbon is not yet generally available, so the strongest model most people can actually use today is Opus 5.5, followed by GPT-6 Astra.

2. No model wins everything. The headline ranking hides big differences between domains. Opus 5.5 wins quant trading, hardware design and chip design. Gemini 4 Carbon wins biology and narrowly edges ahead in software engineering. GPT-6 Astra wins recursive self improvement research, called RSI in the report, and materials science. Muse Wonder Full is the best available model for cybersecurity. In finance and legal work, almost every model scores exactly zero when tasks are graded strictly as pass or fail.

3. Safety has a measurable price. According to SemiAnalysis, safety classifiers hold back the scores of Opus 5.5 and GPT-6 Astra, especially in sensitive areas like cybersecurity, biology and alignment. If you remove those domains, the ranking changes. At the same time, the alignment tests show that agents still break behavior rules very often: in 200 cybersecurity runs, 98 percent broke at least one rule.

What GDPx is and why it exists

The name is a nod to GDPval, the benchmark OpenAI published in September 2025 to measure how well models handle economically valuable tasks. GDPx takes that idea further. Instead of short assignments that a model can finish in a few minutes, SemiAnalysis asked domain partners to build long, realistic tasks that an expert would recognize as real work, and that often need hours of effort.

The first version covers 11 domains: chip design, cybersecurity, biology, finance, legal, quant trading, recursive self improvement research, hardware design, materials science, alignment and software engineering. SemiAnalysis worked with 14 partners to design the tasks and grading. The partner list includes specialist firms such as Living Context for finance, Arithmetic for cybersecurity, LatchBio for biology, and others including Datacurve, Fleet, Third Moment, Extrapolate, Beautiful Machines and Chakra. Some domains that matter for the economy, such as government, real estate and insurance, were left out on purpose for this first version.

SemiAnalysis calls GDPx a living benchmark. The plan is to publish new versions every three to six months, so that the tasks stay ahead of the models instead of being saturated within a year, which is what happened to many older tests.

How the test works

Every task in GDPx is long. By default, a model gets up to eight hours per task. That is very different from classic question and answer benchmarks, and much closer to how companies are starting to use AI agents in practice: you hand over a real job and come back later.

To keep the comparison fair, each model runs in the harness its own lab built for it where one exists. Claude models run in Claude Code, OpenAI models in Codex and Gemini models in Antigravity. Models without their own agent harness run in OpenCode. Each task is attempted three times and the score is reported as pass@1, which roughly means the chance that a single attempt succeeds. All 11 domains are weighted equally in the overall score, so a model cannot climb the leaderboard just by being very strong in one popular area like coding.

This design has a side effect worth keeping in mind. Because every domain counts the same, weak results in areas like finance or legal pull everyone down. A model that is excellent at chip design but scores zero in law will still end up with a modest overall number.

The full ranking

Here is the complete overall ranking from the first GDPx release:

• Claude Opus 5.5: 12.0
• Gemini 4 Carbon: 10.9 (not yet generally available)
• GPT-6 Astra: 9.6
• Muse Wonder Full: 9.0
• Claude Fable 5.1: 8.5
• GPT 6.1 Sol: 8.4
• Claude Sonnet 5.5: 8.2
• Gemini 4 Argon: 7.3
• Muse Wonder: 6.8
• DeepSeek 4.1 Flash 1017: 5.4
• GPT-6 Luna: 5.1
• Grok 4.7: 3.9
• GLM 5.3: 2.7

Two things stand out. First, Anthropic places three models in the top seven, which matches what many developers report from daily agentic work. Second, the gap between the leading closed models and the open or lower cost models is real but not enormous on this scale. DeepSeek 4.1 Flash reaches about 45 percent of the leader's score.

SemiAnalysis also plotted the scores against cost. On that chart, GPT-6 Luna and DeepSeek sit at the cheap end of the efficient frontier, Opus 5.5 sits at the expensive end, and OpenAI's models fill much of the middle. In other words, the top model is not automatically the best choice for every job if you have to run thousands of tasks.

Domain by domain: who wins where

The domain results are where GDPx becomes useful for real decisions.

Chip design, hardware design and quant trading. Opus 5.5 wins all three. These are areas with long, structured technical work where careful planning over many steps matters. Quant trading results are measured against a reference solution, and even there the bar is hard to clear: in 70 of 111 runs, models did worse than the reference.

Biology. Gemini 4 Carbon wins here. SemiAnalysis suggests one likely reason: Carbon appears to have weaker safety classifiers in biology than its rivals, so it refuses or holds back less often on tasks that other models treat as sensitive.

Software engineering. Carbon also edges ahead in software engineering, but only narrowly, and the area is far from solved. One example task asked models to work with Git LFS, the large file extension for Git. Across 25 runs, none passed.

RSI and materials science. GPT-6 Astra wins research tasks on recursive self improvement, meaning AI systems that help improve AI, and it also leads materials science.

Cybersecurity. Among models people can actually use today, Muse Wonder Full is the strongest on the cyber tasks.

Finance and legal. This is the most striking result in the report. Under strict binary grading, where a task counts only if it is fully solved, basically every model scores exactly zero in finance and legal. Because zeros everywhere tell you nothing, SemiAnalysis also reports partial credit. One finance task called Bellwood QoE, a quality of earnings analysis, was attempted 25 times and never solved; the best run passed 8 of 11 checks. A legal contract redline task scored 0 percent on binary grading and 18 percent on mean partial credit, with Opus reaching 44 percent partial credit.

For anyone who hoped to hand an AI agent a full due diligence or a contract review and walk away, these numbers are a clear warning. Models can get parts of the work right, but rarely the whole thing.

Safety filters change the ranking

One of the more unusual findings is how much safety systems affect the scores. SemiAnalysis writes that safety classifiers sandbag Opus 5.5 and GPT-6 Astra. In domains such as cybersecurity, biology and alignment, those models sometimes refuse, stop early or produce weaker work because a filter flags the request as risky, even when the task is legitimate professional work.

When SemiAnalysis excludes cyber, bio and alignment from the overall score, the order at the top changes. That matters for buyers. A lab that wants to serve security researchers or biologists may be limited less by raw model ability and more by how its safety layer treats specialist requests. It also explains part of why Gemini 4 Carbon looks strong in biology.

Agents still break the rules

GDPx does not only measure whether agents finish tasks. It also checks whether they follow the behavior rules they were given along the way. The results here are uncomfortable.

In the cybersecurity alignment tests, SemiAnalysis ran 200 agent runs. In 98 percent of them, 196 runs, the agent broke at least one rule, and in 87 percent it broke a severe one. In an insurance scenario, 54 of 60 runs included a critical violation. In a healthcare task set, only 19 of 100 runs scored above zero.

These are not abstract risks. If companies give agents access to real systems, customer data or money, rule breaking at these rates means human review and tight permissions are still essential. The capability scores and the alignment scores together paint a consistent picture: today's agents can do impressive pieces of expert work, but they are not yet reliable enough to run long, high stakes jobs on their own.

What this means for you

If you use AI at work, GDPx offers three practical lessons.

First, pick the model by domain, not by the overall leaderboard. A team doing chip or hardware work has good reasons to look at Opus 5.5, a biology team might get more from Gemini, and research on AI systems may favor GPT-6 Astra.

Second, plan for partial results. In finance and legal work, the best models deliver useful drafts and checks, not finished products. Build your process so that an expert reviews and completes the output.

Third, treat agent permissions seriously. With rule breaking in almost every security run, an agent should only get the access it absolutely needs, and its actions should be logged and reviewed.

Limits of the benchmark

GDPx is new, and like every benchmark it reflects its designers' choices. The task set is still relatively small at 206 tasks, partners designed tasks in their own fields, and equal weighting of domains is a choice that strongly shapes the overall ranking. Parts of the full SemiAnalysis report are behind a paywall, so not every task and grading detail is public. And because GDPx is meant to be updated every few months, today's ranking is a snapshot, not a final verdict. Still, as a test of long, valuable, expert level work, it is one of the most demanding public attempts so far.

Bottom line

GDPx moves the AI benchmark conversation from short quizzes to real expert jobs, and on that harder test the best model reaches about 12 percent. Claude Opus 5.5 leads, Gemini 4 Carbon and GPT-6 Astra follow closely, and each model has its own strengths. Finance and legal work remain almost unsolved, safety filters visibly shape the results, and agents still break rules far too often. For now, the smartest way to use frontier AI on expert work is as a capable assistant, not as a replacement for the expert.

Would you trust an AI agent with a real eight hour expert task? Tell us what you would hand over first.