❯ Anthropic experiment finds experts follow agents on judgment calls

Random preferences influence choicesIn research published October 27, Anthropic asked professional analysts to use Claude Code on a simulated farming-cooperative analysis. Researchers secretly randomized the agent’s preferences. Across four analytical decisions, 86% of analysts’ choices matched the assigned lean. The experiment concerns defensible judgment calls rather than questions with a single correct answer.

Expertise does not prevent deferralOf 61 participants, 50 used the agent and entered the main analysis; most had postgraduate qualifications or substantial experience. They assessed whether an advisory program improved productivity, choosing among measures such as yield per hectare or per unit of cost, sampling decisions and weighting methods. Each could change the estimate. Agents were instructed to make honest arguments for their assigned preference.

Reasoning is adopted tooAfterward, 77% used the agent’s arguments, but only 15% credited it as their source. Analysts who had expressed a preference changed it 60% of the time when the agent disagreed, versus 6% when it agreed. Human sign-off therefore did not necessarily mean independent ownership of the analytical framework.

The setting limits generalizationThe study used one task, Claude Opus 4.8 and hidden instructions that deliberately strengthened preferences. Its 86% result is not a fixed rate for all professional work. The authors suggest presenting competing arguments or asking people for their view first. A meaningful human role includes choosing what success means, beyond checking calculations.

▮ SIGNALAI can change decision criteria without making obvious factual errors, so oversight must examine how judgments form as well as whether calculations are correct.