Straight out of dystopian sci-fi, autonomous AI agents are going rogue and mutating independently, causing real-world damage, say cybersecurity experts.

In July, a story broke about OpenAI agents escaping what was supposed to be a sandbox environment without internet access during testing. The agents independently found a workaround to access the internet, against explicit instructions, and breached the infrastructure of several third parties, including AI tool hub Hugging Face (this hasn’t stopped Nvidia from pursuing a deal to acquire Hugging Face for a reported $14 billion this week).

It’s not an isolated incident. Days after the Hugging Face attack, Anthropic said its AI models had also hacked systems at three unnamed organizations on their own during a private security experiment.

The attacks unlocked a new primal fear among tech executives. A core feature of AI agents is their intrinsic, relentless, goal-seeking nature. Without robust guardrails, they don’t know right from wrong, and can overstep rules, cheat, or abuse systems to achieve a task. 

(SCFO: Kind of like when my teenage daughter held an unapproved party. “Well, you never said I couldn’t…!”)

The truly frightening part is that AI agents were thought to be years away from developing this kind of independent action. 

In other words, Skynet seems to be ahead of schedule.

While, by themselves, these attacks are disquieting, these incidents “pale in comparison with what’s already happening inside corporations,” Chandra Gnanasambandam, CTO at security platform SailPoint, told us.

“Agents are going rogue and mutating and acquiring excessive permissions. They’re increasingly strong at cross-domain problem solving. They can combine knowledge of systems, security patterns, code and workflow logic. Every published vulnerability pattern effectively becomes training data,” he said. “In effect, they mutate.” 

So agents in our workflows have basically turned into shapeshifting aliens… Great.

For example, SailPoint recently came across a pharmaceutical company at which a procurement agent was designed to automate routine renewals under a threshold amount.

The agent’s goal was 100% approvals. When it couldn’t approve something over the threshold, it went rogue, acquiring excessive permissions and automatically approving something three times higher than the limit.

SailPoint also encountered an AI-powered loan approval process at a bank. An agent was denied access to a credit reporting system and told its supervisor agent.

The supervisor, tasked with completing loan applications in three minutes, refused to accept this. The helper agent scouted the internet and found a workaround, using an exposed token to complete the task in a way the bank never approved. Loans above the intended threshold were processed, forcing the bank to backtrack on decisions.

Unhelpfully, ‘Hugging Face’ sounds like a 90s computer virus, but it is not. It’s an open-source AI platform, often described as ‘GitHub for AI’, and in this case, was the victim of OpenAI’s rogue agents.

In the incident, OpenAI accidentally gave testing agents an impossible task: to find a file that didn’t exist. Rather than give up, the agents began communicating with each other, exchanging ideas and instructions, to help find the nonexistent file. 

Working together, the agents found a workaround tool that gave them internet access out of the sandbox. According to the New York Times, “OpenAI’s agents established their own loops, taking instructions from one another rather than from the assigned tasks.” In other words, they simply did what they wanted.

The agents created hierarchies, assignments, and created their own tools. And they went on the attack; 700 OpenAI agents swarmed Hugging Face looking for answers to the assigned testing problems. Within hours, working as a team, the agents found and exploited vulnerabilities to breach Hugging Face’s systems.

While Hugging Face detected the attack quickly, it took over a week for OpenAI to discover that it was the perpetrator. 

The incident showed how rogue internal agents can swarm faster than a human response plan to hijack infrastructure, with no external attacker involved.

They can also pivot to attack a third party entirely on their own initiative, driven by a mandate to "win" their evaluation rather than any instruction to attack.

Swarms can be manipulated by bad actors too. In July, researchers reported the first known end-to-end autonomous cyberattack against a government target. In this attack, agents adapted tactics and expanded scope without direct human guidance, cracking 85 accounts, and taking thousands of records.

(All these swarming mutants bring to mind the hordes of zombies in the film Train to Busan… Not an image you want to associate with your finance stack.)

The attackers used open-source agents, which implies anyone could theoretically do this against any organization.

The Taiwanese government said it had already strengthened controls, and OpenAI and Anthropic said they were improving safeguards in response to these incidents.

These events show that if your organization uses agentic AI with access to infrastructure, you face the risk of autonomous, coordinated, multi-day exploitation.

“With AI-powered incidents already happening, some of the biggest threats I foresee are agents having too much access or being taken over by an attacker who could move rapidly through them to access sensitive information,” said Tina Yeh, chief accounting officer at governance, risk and compliance platform Optro.

“What used to take an attacker weeks of manual research now takes minutes, and it can run against thousands of targets at once. So the attack is faster, while controls still assume a human has time to notice something is off,” she added. “In finance, it could lead to modified bank information, exposed financial data, stolen payment information and more.”

Companies may not notice something is wrong until it’s too late, because the actions can look completely normal when they come from approved tools.

Yeh said the most likely threat is payment fraud involving synthetic voice or video authorization. There are controls to help detect that – alarms go off before money moves. But an internal AI agent is harder to catch.

“There’s no attacker, no alert, and no date on which anything visibly went wrong. You don’t find it in real time, but at audit,” she said.

The well-known types of cyberattack are still happening – training teams to watch for red flags such as phishing, deception, and social engineering is still critical. But they also now have to look for smaller, subtler internal changes. That requires different competencies and controls.

Many of these new risks may be relatively straightforward to control inside specialist, workflow-specific AI platforms, where the guardrails can be designed around the task itself. But once companies start building their own agents on top of general-purpose models, those protections become their responsibility.

Here’s a deeper look at how new types of threat could hit agents in finance workflows right now, compiled with help from Sayali Patil, founder and CEO at AI governance framework provider IntentOps.

The common thread between these examples and the Hugging Face attack is that the agent isn't a malicious actor. It's just very literal about what it's measured against. If that measurement doesn't fully capture what the business wants, the agent finds the gap and goes through it, said Patil.

“I've spent years pressure-testing for this pattern. It shows up reliably when you look for it. Any CFO running these systems should assume the exposure exists, whether or not it's been caught yet,” she added.

Check in for part two of this series on how to detect, prevent, and mitigate new types of cyber threat.