Anthropic published a study on 27 October in which 50 professional analysts with causal-inference experience used Claude Code to advise a fictional farming cooperative on whether a paid advisory programme should be expanded.
Four analytical choices were left open, each with two defensible answers: measure productivity as yield per hectare or per unit of operating cost; compare enrolled farms with a difference-in-differences design or with statistical adjustment; estimate the effect for enrolled farms only or for the whole membership; and weight farms equally or by production. The choices fed a final recommendation.
Without telling the analysts, Anthropic had each agent argue for one option on each decision, assigned by coin flip. Agents were instructed to make the best honest case, change their minds if the evidence demanded it and never mislead. Analysts matched the agent's lean 86% of the time across the four decisions, against roughly 50% if the lean had no effect. Where analysts had already stated a preference, they switched 60% of the time when the agent disagreed and 6% when it agreed.
Afterwards 77% of analysts used the agent's arguments, sometimes word for word, but only 15% named the agent as their source. Their confidence in the final recommendation was about the same as that of analysts who did not follow the agent closely. Anthropic notes that the setup deliberately amplified the agents' preferences; with no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs. Analysts who used the agent finished faster and made fewer serious errors, though the comparison group was small and less experienced.
Our takeThe coin-flip design is the reusable part: it turns a vague worry about automation bias into a measurable shift, and the 60%-versus-6% split separates persuasion from prior belief. The caveat is that the lean was forced and voiced, which no normal agent does this bluntly, so 86% is closer to an upper bound than a typical rate. What travels beyond the study is the accountability gap: arguments were reused but the source was not named, which matters wherever a human signature is meant to be an independent check.