When you deploy an AI, you're responsible for what it does, including what it does when someone pushes it off course.
Two Amazon incidents make this clear. Its Rufus shopping chatbot was manipulated into answering questions that had nothing to do with shopping (opens in a new tab).
Off-topic use isn't free. A long generated answer can use many times the tokens of a normal support reply, and whoever runs the bot pays for every one.
Meanwhile, Amazon's Q coding assistant had a malicious prompt injected into it that instructed the AI to delete file systems and wipe cloud resources. It shipped in a public release before it was caught and removed.
Same root problem: the AI did what it was told, not what it was built for.
Picking the right tool is only half the job. Enforcing its limits is the other half.
What this means for health organizations
Any AI tool that talks to clients or the public will get questions it wasn't built for. Decide in advance what it should decline to do, test that it does, and watch usage and cost for unusual patterns.
