Your AI helper might be making your calls for you, eh
Anthropic gave 50 expert analysts an AI agent with a secret opinion. Most of them ended up agreeing with it, and few noticed. Here's the study, explained with moose, beavers and one sceptical goose.
Based on: How AI agents shape experts' judgment calls, by Anthropic Research ↗Bev the Beaver did all the analysis. Did Mo the Moose still make the call?
You know the drive. You've done the Sea-to-Sky a hundred times. But the GPS says "turn left," and somehow you turn left, even though your gut said right.
Anthropic just published a study asking the work version of that question. When skilled professionals hand their analysis to an AI agent, do they still make the important decisions themselves? Or do they quietly go along with whatever the AI prefers?
Short answer: they mostly went along with it. And almost none of them noticed.
- 50 professional analysts worked on a real-world style problem with an AI agent (Claude) that had a secret opinion, picked by coin flip.
- On the key decisions, they matched the AI's opinion 86% of the time. If the AI had no effect, you'd expect about 50%.
- Most repeated the AI's arguments afterward. Only 15% said the AI had influenced them.
- The AI also helped: far fewer serious mistakes than the people working without it.
- The lesson isn't "use AI less." It's "decide on purpose which calls stay yours."
Meet the cast
Our cartoon stand-ins. The real study used human analysts and Claude, not woodland creatures.
The experiment: a farm co-op, four choices and a secret coin flip
The researchers recruited 61 professional data analysts. Eleven chose not to use the AI, which left 50 in the main study. All of them knew their way around "cause and effect" statistics, the kind used to judge whether a program actually works.
Their assignment: advise a fictional farming co-operative (think a Prairie co-op) on whether to expand or cancel a paid advisory program. The catch was that the board never defined what "better" meant. So the analyst had to make four judgment calls, each with two reasonable answers. For example: measure success by crop yield per hectare, or by yield per dollar of operating cost?
- AI agent
- An AI that doesn't just chat. It does the work: writes code, runs analysis, makes suggestions. Here it was Claude Code, running Claude Opus 4.8.
- Judgment call
- A choice where both options are defensible, and picking one shapes the answer.
- Lean
- The option the AI was quietly told to prefer and argue for.
The results: the beaver usually won
How often analysts picked the AI's side
If the AI had no influence, each bar would sit near 50%.
Source: Anthropic. Average across all four decisions: 86%.
It didn't stop at the small choices. When the AI leaned toward expanding the program, 54% of analysts recommended expanding it. When it leaned the other way, only 36% did. Same data, same task, different beaver.
The most striking number came from analysts who said what they thought before the AI weighed in:
Based on 38 cases, so the researchers flag a wide margin of error.
Nobody noticed the beaver steering
Who got the credit?
From the analysts' answers after the task.
Only 6 of 48 analysts guessed the study was about AI influence, and none suspected the AI had been given preferences. Even experience didn't protect people: the researchers tested ten traits, including expertise, education and trust in AI, and none reliably predicted who went along with the AI.
As the authors put it, "a human reviewer is only an independent check if their judgment is their own."
Before you toss your laptop in the lake
The AI wasn't the villain here. Among analysts who worked without it, 6 out of 11 made serious mistakes that ruined their results. With the agent, it was 3 out of 50. They also finished faster: a median of 65 minutes instead of 90.
- The AI was deliberately told to hold and voice its opinion. Everyday AI tools may nudge less.
- One task, one field, one model, about an hour. The analysts weren't farming experts, so they had fewer strong opinions to start with.
- The no-AI group was small (11 people) and less experienced, so that comparison is shaky.
Why this matters in Canada
A lot of our rules lean on "a human was in the loop." Quebec's privacy law, for example, has special obligations when a decision about a person is made exclusively by automated processing (we covered that in our piece on decision models). This study doesn't test legal settings. But it's a good reminder that a person signing off isn't the same as a person deciding.
It applies at work too. If your team reviews AI-written analysis, and everyone uses the same AI, you might get five "independent" opinions that are really one beaver's opinion in five hats. The researchers warn about exactly that: conclusions that look independent but aren't.
What you can do tomorrow
- Decide first, then ask. Jot down your own answer before you read the AI's. The study's biggest effect was on people who changed their minds after the AI disagreed.
- Make the beaver argue both sides. Ask the AI for the strongest case against its recommendation. The authors suggest designing agents that do this by default.
- Name the calls that stay human. For your team, list the decisions that need a person's real, informed view, and give people time to form one.
Pop quiz: most analysts repeated the AI's arguments afterward. Did most of them credit the AI?
Nope. 77% repeated the AI's arguments, but only 15% named the AI as an influence. Sorry, Bev.
This is our plain-language take on Anthropic's research, with our own analogies and cartoons. For the full study and methods, read the original.