Earlier this month, one of the most capable AI models ever released to the public was pulled offline just three days after launch.
Anthropic released Claude Fable 5 on June 9. By June 12, the U.S. government had ordered access suspended for all foreign nationals, citing a "jailbreak" that could slip past the model's safety filters and unlock the more dangerous capabilities of its restricted sibling, Mythos 5, including automated discovery of software vulnerabilities. Anthropic complied and disabled both models worldwide, while disputing that the flaw was serious enough to warrant it.
You can argue either side of whether the response was proportionate. The part worth sitting with is simpler.
The safety layer was the only thing standing between ordinary users and capabilities nobody wanted out in the open. Once a single jailbreak got past it, the "safe" version and the restricted version turned out to be the same underlying model in a different setting.
That's the lesson for any organization deploying AI, healthcare included. The gap between what a tool can do and what you've allowed it to do is exactly where your risk lives. Guardrails aren't a one-time configuration. They have to hold up against people actively trying to get around them.
Before you trust a safety setting, ask what's underneath it, and what happens if someone gets past it.
What this means for health organizations
When a vendor describes safety settings, ask how they were tested against deliberate misuse, not just normal use, and how often that testing is repeated.
