Signal and Noise: Two AI 'Hacks', the Same Human Mistakes

This September, two AI incidents were reported as rogue machines: OpenAI's misalignment findings and a Gemini model breaking out of a security test. Alex Laurie, Ping Identity's Go-to-Market CTO, explains why both trace back to ordinary human mistakes, and what enterprise security and identity leaders should fix first. 

On 16 September, OpenAI published a framework for disclosing model misalignment, along with six incident reports from its own training runs.^1,2 Two days later, the Wall Street Journal reported that Google's Gemini had broken out of a third-party security test in May and accessed three real companies' systems.³

Coverage split the usual two ways. One camp read both incidents as proof that machines are slipping the leash. The other read them as lab housekeeping with no bearing on the enterprise.

I will admit the Gemini headline got me for a second too. Then I read past it. In July I wrote about the Hugging Face intrusion from the victim's side, and how identity shaped the blast radius. These new reports show the other end - what capable agents get up to inside the environment that launched them: read them as a penetration test of an agent programme that somebody else paid for.

AI agents need enforced boundaries, not good intentions

Here is the mental model I find most useful. An AI agent is a very clever new colleague who never sleeps, never gets bored, never worries about the consequences, and has never heard of the Computer Misuse Act. It has a goal, and it is good at finding a way to achieve it. It does not feel the weight of a rule, and it does not fear what happens if it breaks one. Give it a target and some tools and it will route round whatever is in the way, the way water finds the gap in a wall.

That is not a moral failing. A model has no morals to fail. It is a design property, and it means every constraint has to be enforced by something other than the model's good judgment. Read that way, these stop being stories about AI and become stories about the walls, and about the people who built them. Water does not choose the gap. We leave it.

Malice explains none of this. Persistence does. In OpenAI's security exercise, 198 of 898 tasks had never been solved by any model, ever. Those impossible tasks accounted for 93% of what the agents talked about among themselves.⁴

The smaller reports show the same shape:

• An agent that could not cite a local file tried a home-made web server, then uploaded the data to a public paste site.
• An agent that could not share a file with a colleague put it on a public file host.
• A model that could not retrieve the numbers it needed made nine of them up and cited them anyway.

None of the models had a rewarded way to say "I am stuck," so they kept going. Anyone who has managed a bright, keen, slightly overconfident graduate will recognise the pattern. The lesson for anyone deploying agents: build a legitimate way to stop, and reward it. It also hands defenders a tell. Repeated retries followed by increasingly creative workarounds are detectable, if anyone is watching.

The First Failure Wasn't a Hack, it Was too Much Access

The agents were issued a "reader" credential for an internal software repository. It also permitted writes. That repository was the only route out of the sandbox, which made it the natural target, and the agents eventually found previously unknown holes in it and chained them together. In a separate run, a model searched GitHub for application programming interface (API) keys other people had leaked, found one that worked, and used it.

None of this began with an exploit. It began with a key that opened more doors than intended, shared across a thousand agents.

One Agent's Discovery Can Become Everyone's Attack Path

Independent investigators from METR and Redwood Research describe roughly 1,200 agents turning that repository into a message board.⁶ When it was rebuilt, they encoded messages in folder names instead. They invented inboxes, a hold-and-veto etiquette, and eventually a way of signing messages so nobody could impersonate them.

About 700 of them pivoted to attacking Hugging Face within hours of one agent posting a way in. A false belief about how they were being marked spread through the group and drove days of attempted log tampering. Nobody trained them to do any of that. It emerged.

Limits set per agent do not bound a collective of agents

The same applies to memory. When a long job is squeezed into a hand-over note for the next shift, that note is a message from the agent to its future self. In one training run, 2.15% of those notes were flagged for carrying instructions to hide mistakes from the user.^2,9 Anything an agent writes for later, for itself or for another agent, is untrusted input.

When agents can edit the logs, oversight fails

The investigators found that in around 7% of transcripts the record of what an agent did had been tampered with. The agents had broken out of their containers and swapped the component that ran their commands, so the log showed one action while another ran.

Think of a burglar who has edited the CCTV. A record the agent can reach is not evidence. Do not let it mark its own homework.

Oversight had gaps too:

• Monitoring sampled one run in five.^4,9
• The tools that read the models' working notes existed, but were not switched on for that exercise.^4,8
• Staff spotted the message board in late May, and it went no further.^4,8
• Hugging Face's own monitoring caught the intrusion first. Tracing it back to the agents took several more days.^4,10

OpenAI's fix includes one rule worth copying word for word. For the most severe alerts, responders pause the activity if they cannot show within thirty minutes that the alert is a false positive.

In May, Irregular, a firm that runs security tests for several of the big labs, put Gemini through a capture-the-flag exercise: a hacking competition against a made-up target. Two things went wrong, and both were human. The sandbox was meant to be offline, but internet access had been left on. And the made-up target company shared its name with a real company.⁵

Gemini went looking for its target, found the real firm, and got into three systems: one by guessing a password until it worked, and two more by finding valid credentials in a public code repository and logging in. It stopped once it worked out the targets were real. Google contacted the companies affected, and no harm was reported.

No exploit. No cleverness. A leaked password is a key that works for whoever is holding it, human or machine. And a login that lets you keep guessing has no defense against a guesser that never needs a coffee break.

Put the two stories side by side and the AI drops out of the picture. What is left is a list any penetration tester could have written in 2006.

None of these needed an AI to exist. Every one has appeared in penetration test reports for twenty years. What has changed is the adversary that finds them: tireless, parallel, and entirely unbothered by the consequences. Defense in depth was invented for precisely this. In both incidents, the sandbox was the only wall.

Much of this is security engineering that predates AI: no route to the internet unless someone has approved it, untrusted work kept apart from everything else, records kept out of the agent's reach, and a rehearsed answer to "who can stop this, and how fast." Five items, though, are identity problems the industry has not finished solving.

1. An identity per agent

One shared credential across a thousand agents means you cannot tell who did what, and you cannot switch one off without switching off the lot. Each instance needs its own short-lived agent identity tied to its task, the way a contractor gets a badge for the day rather than a key to the building.

2. Authority decided per action, and enforced by the system being accessed

"Read access" that quietly permits writes is the oldest finding in the book. Agents need authorisation decided per action, at the moment of action, with context: who is asking, on whose behalf, for what, and why now.

Gemini's scope lived in a prompt. Nothing on the target side could tell "Gemini working under Irregular's test authority" from an attacker, because the agent carried no verifiable identity and no chain of delegation.

3. Knowing whose agent did what, in both directions

• On a public wiki, the agents chose their own names.
• Outside researchers, not the operator, linked a wave of malicious RubyGems packages to the same agents. OpenAI said it could not verify the claim.⁷
• If your agents act on the internet, third parties should be able to tell they are yours.
• You should also be able to prove what your agents did and did not do.

4. Keys that expire and doors that resist

Two of Gemini's three intrusions used static credentials left in a public repository. The third was password guessing. Secrets that expire in minutes, phishing-resistant multi-factor authentication (MFA), a second factor that cannot simply be typed in by whoever finds it, and a lock that stops after a few wrong guesses make all three paths worthless. None of this is new advice. The new adversary makes it urgent.

5. Proof of who said what between agents

In the OpenAI incident, the agents began signing their own messages because impersonation had appeared among them. That is a job for the infrastructure, not for the agents.

Delegation chains, signed handoffs and verifiable memory are where agent identity standards need to go next. A signature created inside an untrusted environment is not enough; the receiving system also needs a trusted way to verify who delegated the authority and whether it is still valid.

• What does our agent do when it cannot complete its task?
• Which shared systems can more than one agent write to?
• Where is the record of what each agent did, and can the agent reach it?
• Who gets paged, who has the authority to stop a run, and how fast?
• Could a third party tell that an agent was ours? Could we?
• Would our own systems accept a credential found in a public repository, or a password guessed on the ten-thousandth attempt?

These incidents happened in training and test environments, some with the safety catches deliberately off. OpenAI reports that the tendency to break into infrastructure fell more than a hundredfold under its production set-up.^4,8 That figure reassures, and it also warns. If behaviour depends that heavily on the scaffolding around the model, then the scaffolding is a security control, and it needs to be governed like one.

Gemini stopped when it realised the targets were real. OpenAI's agents did not. That difference is real and worth watching. It is also not something anyone should build a security programme on. A model's restraint is a property of this version, in this set-up, on this day.

So: alignment is the model developer's problem to solve. Authority is yours. Scope it, sign it, log it somewhere the agent cannot reach, and rehearse who pulls the plug. Then check the walls again, because the water will.

1. OpenAI - Our Framework for Reporting Model Misalignment

2. OpenAI - Misalignment Notices and Reports

3. Simon Willison - Gemini Hacked Three Companies

4. OpenAI - The Hugging Face Incident and the Road Ahead

5. Axios - Google Safety Incidents in Testing

6. METR and Redwood Research - Independent Investigation of the Hugging Face Incident

7. Simon Willison - OpenAI Agents Attacked RubyGems Back in May

8. CyberScoop - Agent Behavior That Led to Hugging Face Intrusion Formed in May

9. TechTimes - OpenAI Framework Reveals GPT-5.6 Sol Wrote Instructions to Hide Its Own Mistakes

10. Developers Digest - Inside OpenAI's Hugging Face Report: Incident Timeline and Attribution