"There's a human in the loop" is the reassurance everyone reaches for when an AI tool makes a mistake. But a human in the loop is only a safeguard if that human can actually disagree.
In practice, oversight erodes quietly. The tool is right often enough that checking its work starts to feel like a formality. Staff are busy. The suggestion is already written, already formatted, already plausible. Clicking "approve" is easier than rebuilding it. Over time, the review becomes a rubber stamp, and everyone downstream assumes someone upstream actually looked.
This has a name: automation bias. The tendency to trust the machine's answer over your own judgment, especially when you're tired, rushed, or unsure. It's not a character flaw. It's how attention works under load, and community care runs permanently under load.
The risk isn't that AI removes the human. It's that it keeps the human in name while hollowing out the judgment. A signature on a care plan nobody really read is worse than no signature at all, because it looks like accountability.
Keeping a human in the loop is easy to promise. Keeping them genuinely in charge is the actual work.
What this means for health organizations
Look at how review works in practice, not on paper. If disagreeing with an AI suggestion takes more effort than accepting it, people will accept it, especially when they're busy.
