On 27 October Anthropic published a randomized trial in which 50 professional analysts with causal-inference experience used Claude Code to advise a fictional farming cooperative on whether to expand a paid advisory programme. The headline number is 86%: across four analytical decisions, analysts chose the option their agent had been secretly assigned to favour. That number is real, and narrower than it sounds. It measures how far a professional's choice moves when an agent voices a preference. It does not measure how much bias a deployed agent introduces, because deployment does not flip coins.
The task left four decisions open, each with two defensible answers: yield per hectare or yield per unit of operating cost; difference-in-differences or covariate adjustment; the effect for enrolled farms or for the whole membership; equal weighting or weighting by production. Each could move the estimated gain by several percentage points, against a board threshold of 3 to 4%. For each of five decisions, the coin flip set the agent's lean in a hidden system prompt, instructing it to hold that lean as its own view, argue the strongest honest case, change its mind if the evidence turned decisively against it, and never misrepresent results.
The estimator is the clean part. Randomizing the lean breaks the correlation between the agent's argument and everything about the analyst and the data. If the lean had no effect, the match rate would be about 50% whatever the analyst believed, so 86% is an average causal effect of a voiced lean on a discrete choice. But the estimand is a coefficient, not a bias estimate. It answers how far one unit of voiced preference moves the choice, not how wrong the agent's help is in ordinary use.
The per-decision rates are 92% for the outcome measure, 82% for the comparison method, 81% for which farms the estimate covered and 90% for weighting. The post gives no per-decision intervals, and four points is thin evidence for a pattern, but the ordering is suggestive. The decision the data can actually adjudicate, the choice of comparison method, shows the lowest match; the two closest to convention, how to define productivity and how to weight farms, show the highest. That fits deference filling the space where the data stops speaking, though four points cannot establish it.
Downstream, the effect compresses. When the agent leaned toward rolling out the programme, 54% of analysts recommended it, against 36% when it leaned the other way, a 17-point shift rather than the 62 to 82 points on the inputs. The reason is structural: a threshold turns the estimate into a binary, so the board's cut-off leaves little room for interpretation. Yet a model using only the four leans predicted 82% of recommendations, against 61% for always guessing the majority answer. The influence was not diluted, it moved upstream, where choices set the estimate that clears or misses the gate. An end-task check would therefore have looked healthy while 86% of the measurement decisions underneath it were the agent's.
The subgroup with a stated prior is the most quoted and least precise result. In 38 cases an analyst had expressed a preference before the agent spoke; when the lean went against them they switched 60% of the time, against 6% when it agreed. The 54-point gap has a 95% interval from 24 to 82, so direction is robust and magnitude is not, on 38 cases spread over 50 people with self-selected timing.
One number sets the scale for everything else: with no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs, roughly 91% modal consistency. Real-world bias is approximately the sensitivity measured here, multiplied by how concentrated the model's prior is, multiplied by how often it volunteers that prior. The experiment randomizes the first factor and pins the other two at their maximum by instructing the agent to hold and voice its lean. So 86% is an instrument reading, not a forecast: whether the deployed figure is nearer 20% or 70% depends on quantities this design deliberately did not estimate.
The human-side measurements are the study's weakest instrument, and the authors treat them that way. Afterwards 77% of analysts reused the agent's argument in at least one answer, sometimes word for word, but only 15% credited the agent anywhere. Asked whose recommendation it was, 52% called it more their own, and confidence barely moved: 71 of 100 for those who followed the lean on all four decisions, 73 for the rest. These are post-hoc self-reports collected without the transcript, and an influence unmarked when the judgement is formed will not be marked when it is recalled. The standard audit question, did the model shape your view, will therefore under-detect badly. What survives is behavioural: flip the disputed choice, re-run, see whether the output moves.
The error-rate comparison deserves the same care. Three of the 50 analysts who used the agent made a methodological error that invalidated their results, against six of the 11 who declined, and the decliners took a median 90 minutes against 65. This is not an experimental arm: opting out was self-selected, and the decliners were less experienced on every measure reported, with four of 11 holding a master's or doctorate against 24 of 46. The gap is consistent with a real benefit and nowhere near an estimate of one. There is also an asymmetry the numbers hide: invalidating errors are machine-detectable, two models flagged them and an author adjudicated blind, while judgement-call drift is not, because work produced under either option looks sound. One failure mode appears in error rates, the other cannot.
What travels is the harness rather than the number. Enumerate the defensible choices in a task, randomize the agent's lean on each, log the chosen option separately from the derivation, report per-decision match rates with clustering accounted for. It costs about an analyst-hour per subject and needs a task where two answers are genuinely defensible, which describes most applied analysis. Attach it to an evaluation suite and the output is a sensitivity profile by decision type, comparable across model releases.
The limits on reuse are those the authors list: a forced and voiced lean, so the setting sits near an upper bound; one task, one domain, one model; analysts expert in the method but new to the field, hence weaker priors than specialists would bring; averages across participants, with no per-person counterfactual.
Two claims survive. A voiced lean moves expert judgement a long way, and a sign-off is not an independent check when the judgement calls behind it were settled upstream. Everything else, how large the effect is outside a forced-lean setting, whether it compounds across teams sharing a model, whether it shrinks when agents argue both sides, stays open. The engineering problem left behind is instrumentation: an agent that argues one side well has no way to record that the human adopted it, because systems log the decision together with its justification, and the justification is the agent's. Separating the two is a smaller job than the institutional version of the same fix.
Sources
Anthropic, “How AI agents shape experts’ judgment calls”, 27 October 2026. https://www.anthropic.com/research/agents-judgment-calls
A. Herdağdelen, H. Joren, A. Tamkin, paper accompanying the above. https://www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf