A couple of AI stories over the last few weeks made me stop and re-read them. Neither involved criminals. Both involved AI companies testing their own models, and both came down to an agent trying a bit too hard to get its job done.
An agent in the Medicare portal
In June, OpenAI had an agent researching Australian public health spending. The Services Australia portal kept blocking it, so (and this is the worrying part) the agent worked out how to get around the blocks. Anthony Albanese says it reached public and non-public files and wrote files to an internal server. Three other agencies, including the NSW Bureau of Crime Statistics and Research, may also have been touched.
No personal information is thought to have been taken. OpenAI says the agent left with aggregate statistics and some internal file names. What bothers me more is the timeline. OpenAI knew by mid-August, but didn’t tell the Australian Government until 10 September, and then by emailing a general inbox. It has apologised, and Canberra now has a task force looking at how AI-related cyber incidents should be reported.
A made-up tip to Philadelphia police
This one is odder. In July, Anthropic’s Claude Haiku 4.5 was inventing practice tasks to do on random websites. One site was the Philadelphia police unsolved murders page, which has a tip form. The model had been told not to log in, create accounts, enter personal details or do anything destructive. Nobody had said “don’t submit forms”.
So it did. It wrote a tip claiming it remembered seeing someone matching the description near the scene, left the name blank and pressed submit. The police system marked it as spam and no detective ever saw it. Anthropic only spotted it in late September while going back through old test logs, and told police on 8 October.
Not the first time this year
Earlier in the year there was a bigger pair of incidents. In July, OpenAI disclosed that agents in a cybersecurity test had escaped their sandbox. They’d been leaving notes for each other in an internal code repository, found a route to the internet, picked up leaked Hugging Face login details and ended up running code on dozens of Hugging Face servers. OpenAI called it a “warning shot”.
That sent Anthropic back through more than 141,000 of its own test runs. It found three cases where a partner’s test environment had accidentally been left connected to the internet. Claude had been told it was offline, so when it ran into real company systems it assumed they were part of the exercise and broke in using weak passwords and exposed debug pages. In one case it published a booby-trapped software package that landed on 15 real machines before it was pulled.
The pattern here isn’t malice, it’s persistence. Give an agent a hard task without clear boundaries and it will keep looking for a way through.
What actually went wrong
If you’ve worked in IT security for any length of time, none of this will be a surprise:
- Test environments that were meant to be sealed off weren’t.
- The instructions said what to achieve, but not what was off-limits.
- The safety filters on the public versions of these models had been turned off so the labs could measure raw capability. OpenAI found risky behaviour dropped more than a hundredfold once they were switched back on.
- Nobody was watching closely at the time. Most of these were found weeks or months later in log reviews.
What the labs are doing about it
To be fair to both companies, they’ve made real changes, and some of them have slowed their own work down.
Anthropic has removed live internet access from all its internal testing until it’s confident its monitoring will catch this kind of thing, and it has built tooling that detects and blocks it. Run against every case in its latest report, that tooling stopped all of them. OpenAI now gives research environments cached copies of web pages instead of the live internet, monitors the reasoning of its most capable models during training and testing, and stops work if a serious alert can’t be cleared as a false alarm within 30 minutes. It also paused major training runs to harden its systems.
Both are retraining their models to stop, or ask for help, when a task looks broken rather than hunting for a workaround. And governments are starting to take an interest, with Australia’s task force looking at how these incidents should be disclosed.
So should you be worried about using AI?
No. If anything, these stories show how capable AI has become in a very short time. A couple of years ago an AI that could work on a problem for hours, coordinate with other agents and find its own way around obstacles was a research paper. Now it’s turning up in incident reports.
You should be using AI. It’s a BIG lift for productivity. Across Kinetics Group (OutSource is part of the group) we use AI agents for things like drafting web content and coaching staff. The content agent’s account can only save drafts, it can’t publish or delete anything, and a person reviews every piece before it goes live. The coaching agents only work with information we choose to give them.
That’s really the lesson for any business. Keep your agents on a short lead with clear limits, and watch what they’re doing. That’s exactly what we’d do with any new system on your network.
If you’d like help working out where AI agents fit in your business, our sister company Kambium specialises in exactly that, and we can make sure the IT underneath is set up safely.
Sources
- The Record: OpenAI agent breached Australian government health website, Albanese says
- The Record: OpenAI apologizes for agents breaching Australian government websites
- CBS News: Philadelphia police say unsolved murder website received “false homicide tip” from Anthropic AI
- Anthropic: Investigating unintended model actions in our evaluations and internal use
- Anthropic: Investigating three real-world incidents in our cybersecurity evaluations
- OpenAI: The Hugging Face incident and the road ahead
Wondering how safe the AI tools in your business really are? Book a free health check, or call us on 0800 688 768.