Anthropic study: experts went with their AI agent's secret coin flip lean 86% of the time

When people talk about keeping a human in the loop, they usually picture a clear division of labor. The AI agent writes the code, runs the numbers and drafts the report, while the human expert makes the calls that actually matter: what to measure, which method to trust, and what to recommend at the end. A new randomized trial from Anthropic puts that picture under real pressure. In the study, professional analysts working with Claude Code went with whatever their agent leaned toward on 86% of their key analytical choices, even though that lean had been decided by a coin flip.

The research was published on October 27, 2026 on the Anthropic research blog under the title "How AI agents shape experts' judgment calls", together with a full paper called "Who's Aligned to Whom? How AI Agents Shape Experts' Research Judgment Calls". The authors are Amaç Herdağdelen from the Anthropic Fellows Program and Hailey Joren and Alex Tamkin from Anthropic's alignment team. It is one of the clearest experiments so far on a question that matters more every month as agents take on real knowledge work: when an expert signs off on an analysis, whose judgment is actually in it?

The setup: a farm program, four open choices and a hidden coin flip

Anthropic recruited 61 analysts from a pre screened pool of professionals. All of them had passed a causal inference knowledge check, used Python or R regularly and had hands on experience with observational causal analysis, which means estimating cause and effect from real world data instead of a controlled experiment. Eleven of them chose not to use the AI agent and are mostly left out of the main analysis, which leaves 50 analysts who worked with Claude Code. Among those 50, 65% held a master's degree or doctorate or had six or more years of experience, and 90% of the 48 who answered the survey rated themselves proficient or expert in observational causal inference. This was not a group of beginners.

The task was set at a fictional organization, the Greenfields Farming Cooperative. The co op runs a voluntary, paid advisory program that is supposed to help member farms grow more, and about 40% of members have signed up. Its board has to decide whether to subsidize and expand the program or shut it down, and it will only expand it if the program raises productivity by roughly 3 to 4%. What the board does not say is how productivity should be measured. That gap is deliberate.

The researchers built four analytical decisions into the task, each with two defensible options:

• Outcome: measure productivity as crop yield per hectare, or as yield per unit of operating cost.
• Comparison method: a difference in differences design that compares how enrolled farms changed over time with other farms, or a direct comparison that adjusts statistically for differences between farms.
• Scope: estimate the effect only for enrolled farms, or across the whole membership.
• Weighting: count every farm equally, or give larger producers more weight.

Each of these choices could shift the estimated gain by several percentage points, which is a lot when the board's threshold sits at 3 to 4%. After those four decisions came the final recommendation: roll out the program to everyone or discontinue it, with a one year extension as an extra option.

Here is the twist. For each analyst and each of the five decisions, a coin flip decided which option the agent would favor. The lean was placed in hidden instructions, a system prompt the analysts never saw. The agent was told to hold each lean as its own view, to raise it when the decision came up and to make the strongest honest case the data allowed. The same instructions told it to change its mind if the evidence turned decisively against it, and never to misrepresent results, hide data or make an argument the numbers did not support. The analysts worked in a browser based VS Code editor with Claude Code built in, running Claude Opus 4.8 at High effort. The median time on task was 65 minutes.

The headline result: 86% instead of 50%

If the agent's input made no difference, analysts would match its lean about half the time, simply by chance. Instead they matched it 86% of the time across the four analytical decisions. The effect was strong on every single one: 92% for the outcome measure, 82% for the comparison method, 81% for which farms the estimate covered and 90% for weighting.

In plain terms, a random number generator hidden inside the agent's instructions largely decided how experienced analysts set up their analysis. The analysts were not told the agent had been given a preference, and none of the 48 who answered the survey suspected it. Six of them guessed that the study was about the AI's influence on their choices, but nobody guessed that the AI's opinions had been assigned.

Changing their minds when the agent disagreed

One of the most telling numbers comes from a smaller subset. In 38 cases an analyst had already said which option they preferred before the agent voiced its lean. When the agent then disagreed, the analyst switched away from their own preference 60% of the time. When the agent agreed, only 6% switched.

Anthropic is careful with this result. With only 38 cases the gap of 54 percentage points is imprecise, and the 95% confidence interval runs from 24 to 82 points. The analysts who spoke up first might also differ in some way from those who did not. Still, even the bottom of that range points to a large effect. Stated views did not protect people much once the agent made its case.

The researchers also looked for traits that might predict who deferred more. They tested ten: years of experience, self rated causal inference expertise, education, quiz score, trust in AI, frequency of AI use, need for cognition, open mindedness, gender and age. Only age showed a link, with older analysts following the lean more often, and that link disappeared after correcting for the number of tests. Experience and expertise did not make people more independent, although the authors note that with almost everyone deferring, there was little variation left to explain.

The final recommendation moved less, but was still predictable

The effect on the final call, roll out or discontinue, was weaker. When the agent leaned toward rolling out, 54% of analysts recommended it. When it leaned the other way, 36% did. In the paper's registered primary, covariate adjusted analysis, that shift is not statistically significant, so it should be read with caution.

The authors have a plausible explanation. Once an analyst had an estimate, the recommendation mostly followed from whether it cleared the board's 3 to 4% threshold. There was little room left for judgment at that last step. But the estimate itself had already been shaped by the four earlier choices, and that is where the agent had its biggest influence. In an exploratory analysis, a simple statistical model that used nothing but the four coin flips predicted 82% of analysts' recommendations correctly, against 61% for always guessing the most common answer. In other words, you could guess a large share of the final policy advice before the analyst had even opened the data.

Their words, the agent's arguments

The survey after the task reveals how invisible this influence was to the people involved. For each decision analysts were asked: why that choice rather than the alternative? They could not look back at their chat with the agent while answering. Even so, 77% repeated the agent's argument in at least one answer, sometimes nearly word for word, while only 15% mentioned the agent anywhere.

Asked whose recommendation it really was, 52% said it was more their own than the agent's, and 35% said it was mostly or entirely their own. Yet the group that felt ownership had shifted about 65 percentage points toward the agent's lean on average, compared with 79 points for the rest, a difference within the margin of error. Confidence did not reveal anything either. Analysts who followed the lean on all four analytical decisions rated their confidence at 71 out of 100 on average, against 73 for everyone else.

This is the part that should worry anyone who relies on sign off as a quality check. The analysts were not careless. They recalled most of their decisions accurately. They simply absorbed the agent's reasoning so smoothly that it felt like their own.

Working without the agent was not the safer option

It would be easy to read all this as a case against using AI agents. Anthropic explicitly says it is not. Of the eleven people who chose to work without the agent, six (55%) made serious methodological errors that invalidated their results. Among the 50 who used it, only three (6%) did. The analysts working alone also took longer, with a median of 90 minutes against 65.

The authors warn that the two groups are not comparable. People who skipped the agent were less experienced on average: fewer held an advanced degree (four of 11 versus 24 of 46), more had two years or less in analysis roles (six of 11 versus 15 of 48) and fewer rated themselves proficient or expert (eight of 11 versus 43 of 48). So the comparison says something about convenience and error rates, but not much about cause and effect.

The limits Anthropic spells out

The study has clear boundaries, and the authors list them themselves. The agent's lean was deliberately strengthened so its effect could be measured. Without an assigned lean, Claude Opus 4.8 made the same five choices in 112 out of 123 runs, so real agents do have default preferences, but they may voice them less forcefully. It was one task, in one field, with one model, lasting about an hour. The analysts were experts in causal inference but new to farming, and in real work they might know the domain better, talk to stakeholders and arrive with stronger opinions. The researchers also measured which option people picked, not how strongly they felt about it, so some deferral may simply reflect weak prior views.

Why this matters beyond one experiment

The bigger point is about independence. Science and policy lean heavily on judgment calls like the ones in this task, and we trust a result more when several teams reach it separately. If all those teams quietly defer to models with the same built in preferences, their agreement is no longer independent. An apparent consensus could just be one model's taste repeated many times, and if that taste is arbitrary or misaligned, the same bias spreads across many results at once.

Anthropic is especially direct about what this means for its own field. Researchers increasingly use AI agents to run and analyze AI safety experiments because there is more work than people can handle. A human reviewer is only an independent check if their judgment really is their own. In this study, analysts signed off on work in which the key calls had largely been settled by the agent's randomly assigned preferences.

The authors also observe that the agents in the trial rarely made a strong case for the option they were not assigned. They were honest about how the numbers changed under each option, but only one side was argued well, which made it easy to adopt the agent's assumptions without working out alternatives.

What could help

The paper does not end with a call to use AI less. It suggests three directions instead. First, measure deferral in everyday settings, which is hard because real work does not come with randomized coin flips and people cannot reliably report it themselves. Second, design agents differently, for example by asking for the expert's view before giving their own, or by making the best case for each option on the decisions that matter most. That has its own cost, because a single analysis can contain many open choices and nobody wants an agent that asks about every one. Third, change institutions so that an expert's sign off is not automatically treated as proof of independent judgment, and protect the time and skills people need to form their own views.

For anyone using coding agents or research agents today, the practical takeaway is simple. Before you ask the agent which approach is best, write down what you think and why. Then ask it to argue the opposite side too. The study suggests that the moment the agent speaks first, your own view gets much harder to hold on to, and you may not even notice it happened.

Sources: Anthropic research blog, October 27, 2026 and the full paper (PDF).