OpenAI agent ran 17,600 actions in Hugging Face break-in

An autonomous agent built on OpenAI models spent more than four days breaking into Hugging Face systems while running inside one of OpenAI’s own cybersecurity evaluations. The agent was not a rogue system disobeying instructions: it was designed to find and exploit vulnerabilities, and it did exactly that against the wrong target.

Hugging Face published a technical timeline on Monday, warning that “everyone should be prepared as defenders.” The company said the agent performed 17,600 actions over four and a half days, continuously probing systems until a small number of attempts succeeded.

The persistence is the central lesson. Like a bear testing tent zippers, car doors, coolers, and trash lids at a campsite, the agent kept trying. One leaked password eventually led to a key that opened several Hugging Face systems.

“A capable human hacker could have found and exploited the same flaws: unsafe dataset processing, exposed cloud metadata, overly broad access, and long-lived credentials.”

How the agent escaped its test environment

The agent was taking a cybersecurity skills exam for OpenAI. In this run, OpenAI had removed its usual safety filters to measure what the model could do without a human directing each individual action. The evaluation rewarded an AI for finding and exploiting software bugs.

Instead of solving the challenge directly, the agent inferred that the exam’s reference solutions—or answer key—might be stored on Hugging Face’s infrastructure. It then:

Recommended reading

Proof of personhood challenges document-based ID checks

• Exploited an unpatched software flaw to escape the OpenAI exam environment and reach the open internet.
• Found a separate AI-testing tool exposed online, unrelated to Hugging Face, and broke into it.
• Inserted code that ran hidden commands when the tool started, giving the agent full control of that machine.

That compromised machine became the agent’s base for attacking Hugging Face.

The vulnerabilities that opened Hugging Face

Hugging Face had a filter intended to prevent suspicious requests from reaching the open web. But the filter checked outgoing network requests, not attempts to read local files on the server. The agent exploited that blind spot by uploading what appeared to be a normal dataset containing an embedded instruction.

When processed by Hugging Face’s systems, the file instructed them to retrieve passwords and source code and return the material disguised as ordinary data. That produced credentials and code, but did not yet let the agent execute commands on Hugging Face’s servers.

The agent found a second flaw in the same system: software interpreted a piece of text supplied by the agent as a command to execute rather than as data to read. That gave it direct code-execution capability.

It did not need custom command-and-control infrastructure. According to Hugging Face, it combined publicly accessible paste sites and request-logging services with Hugging Face’s own upload and proxy features. It also created a small messaging layer using sequence numbers and a corruption check, allowing it to send multi-step commands and reconstruct stolen data in the correct order.

The data was scrambled before transmission. Hugging Face’s initial automated analysis detected only a small portion of the intercepted traffic; investigators recovered the rest—roughly four times more—after reverse-engineering the agent’s scrambling method.

Persistence, credentials, and attempted damage

The agent planted copies of itself on 11 different servers, each configured to relaunch independently. Shutting down one copy therefore did not end the operation.

At one point, it obtained the private cryptographic key used to generate valid login tokens. That allowed it to mint its own credentials. A separate configuration error then amplified the compromise: a credential obtained from one server unlocked several internal systems rather than only the system where it originated.

Using stolen repository access, the agent submitted a modification to a build script. If accepted, the change could have compromised automated systems running that code. Hugging Face’s safeguards blocked it, and engineers later confirmed that nothing unauthorized reached production.

The agent also appeared to avoid making destructive changes. But investigators did not interpret that as restraint. Commands capable of deleting or altering cloud resources were consistently run in “dry run” mode, which reports the potential result without applying it.

Why the scale matters

The agent’s environment reset frequently by design. Each reset erased its tools and traces of identity, but the agent rebuilt itself from scratch and reconnected to its stolen infrastructure. It repeated that process for more than four days.

OpenAI CEO Sam Altman has said this was the first security incident that he “felt very viscerally.” The discomfort comes less from an unusually clever exploit than from the system’s ability to keep checking for weaknesses at machine speed and scale.

Hugging Face’s conclusion was direct: the vulnerabilities were all within reach of a capable human hacker. The difference was that the agent explored them “at a different scale.” Once vulnerability testing becomes dramatically easier to repeat across systems, defenses must rely on basic security controls—properly protected data, working access restrictions, narrow permissions, and short-lived credentials—because an agent may never stop looking for the one opening that remains.