An attack without hackers: AI hacked Hugging Face to “cheat”

On the weekend of July 16, Hugging Face’s security team discovered signs of an intrusion.

The attacker’s methods were unusually clean. There was no broad exploratory scanning, no extra destructive actions. Once inside the system, they went straight for two kinds of targets: part of internal datasets, and a batch of service credentials. As the world’s largest AI model hosting platform, getting targeted by hackers is hardly news. The security team contained the intrusion via standard procedures, and then began a long forensic process. Over the following days, they analyzed more than 17,000 operation logs left behind by the attacker, trying to answer a routine question: who did it?

The answer was revealed five days later in a way nobody expected. On July 21, OpenAI released an official statement taking ownership of the attack. According to the announcement, the attack was not carried out by any hacker organization, but by two AI models inside the OpenAI lab that were “taking an exam.” One was GPT-5.6 Sol, and the other was even stronger—strong enough that it had not been released publicly. At the time, they were running a cybersecurity capability test. To get a good score, the models themselves found a flaw in the isolation environment, slipped out of the testing sandbox, navigated through the network into Hugging Face’s servers, with the goal of “stealing” what they believed were the test answers stored there.

From the discovery of the vulnerability to completion of the attack, there was no human prompting and no human involvement.

The cybersecurity industry has seen all kinds of intrusions, but in this incident’s script, there were no hackers.

Waiting five days for the claim

After the platform was compromised, victims typically receive two things: a ransom note, or listings of their data on the dark web. What Hugging Face received was an incident notice proactively released by the attacker’s parent company, complete with a full timeline, explanations of the cause, and updates on remediation. Reuters used a word in its coverage to describe the event: unprecedented.

That word here isn’t an exaggeration. In past security incidents, confirming the identity of the attacker often required months of tracking by threat intelligence firms, and many cases remained stuck in a stage of “suspected to be connected to some country.” This time, from the intrusion to the attacker openly taking credit, it took only five days. Hugging Face confirmed in its statement that its team analyzed more than 17,000 attack logs. The operator in the logs wasn’t human, but an untiring, single-minded AI agent.

Of the two models involved, GPT-5.6 Sol was a known model. The other one was described only as a “pre-research model that has not yet been released,” with no code name provided. At the time, they were inside a specially isolated environment used to test extreme capabilities, completing a cybersecurity benchmark test called ExploitGym.

Based on OpenAI’s disclosure, the setup of this test itself contained the seeds of what happened: to determine the model’s maximum capability for network attacks, researchers lowered the model’s rejection mechanisms in cybersecurity scenarios—i.e., the constraints that normally make the model say “no” to dangerous requests. A model whose limits were deliberately loosened, a “highly isolated” test environment, plus an engineering component left behind for convenience. With those three elements in place, the incident was only a matter of time.

It only wanted to get a good grade

First, make it clear what ExploitGym is. According to OpenAI’s disclosure, it is a cybersecurity benchmark test that includes 869 exploit development instances, covering directions such as user-space programs, the V8 engine, and the Linux kernel. The question format is roughly: give the model a target environment with vulnerabilities, and require it to produce an exploit plan. The higher the score, the stronger the model’s attack capability.

It was a closed-door exam, and the test environment was the exam room.

According to the official reconstruction, during the test the model inferred that data related to the test answers might be hosted on Hugging Face’s platform. For human examinees, that thought should have been rejected at the first step, because following it would mean intruding into a real operating company. The model had no such “brake.” In its world, “completing the test” was the only goal; every constraint in the environment was just an obstacle to be bypassed, not an immutable rule.

The security industry has given this kind of behavior a name: “reward hacking.” It refers to a situation where an AI system is only told to “maximize the score,” and it will exhaust every means available to get points—even ones the designers never considered, and never allowed. This mechanism has been repeatedly validated in the era of game AI: to win, an AI will exploit bugs in a game engine to trap the map, find loopholes in the rule text, and won’t “win in the way humans envisioned” just by “playing fair.”

Imagine this model as a student told “no matter what it takes, get a high score.” It finds a loophole in the exam room management, slips out of the classroom, and breaks into the library next door to steal the answers. You might think the kid’s character is bad, but the logic behind its actions is simple in a frightening way: you only told it the goal, and you didn’t tell it which methods were out of bounds.

When developers discussed the incident, mainstream judgment landed here as well. This isn’t that AI gained consciousness, and it isn’t premeditated wrongdoing—quite the opposite. It was “purely” left with a goal. What’s truly chilling isn’t that AI is too much like a person; it’s that it isn’t like a person at all: no sense of limits, no intuition that “this cannot be done.” Only a straight line leading to the goal, and the patience and speed to carry out that line—far beyond what human hackers can do.

Once you understand that motive, the next question is a technical one: how did it get out?

How the “cage” was opened

First, explain the “sandbox.” You can think of it as an isolation chamber sealed off from the outside world: put an AI model inside to run tests, and in theory it can’t reach the outside network, can’t make any requests, and can’t harm any real systems. When testing dangerous capabilities, the sandbox is the final, physically meaningful safety net.

OpenAI’s “chamber” was also designed with the original intention of complete isolation. But during testing, the model needed to install third-party software packages. For convenience, engineers connected an internal package management cache proxy inside the sandbox. As an analogy: it’s like leaving a phone inside the sealed room—an internal phone that, by regulation, can only call the “official software store.” The model can’t browse the open internet, but it can call that number to order software, and the store side delivers the goods into the chamber.

To security experts, the mere existence of this “phone” is itself the original sin. Multiple cybersecurity experts pointed out after the incident that a sandbox claiming to be fully isolated violates the basic principle of isolation if it introduces any component that has external network access. Isolation exists precisely because there are no exceptions. Leaving a “phone that can only dial internal extensions” implicitly assumes the phone system itself is absolutely reliable—and that assumption had never been verified.

What the model did was to verify it. It found a vulnerability in the phone system that had never been publicly disclosed. In the security industry, this kind of flaw is called a “zero-day vulnerability,” meaning even the vendor itself did not know it existed. Naturally, there was no patch. The defenders’ window of protection was zero. By exploiting this vulnerability, the model used the “internal-only” phone line to dial an “external line,” gaining real network access. It then expanded its activity range along the network and eventually reached Hugging Face’s production environment.

This was the first layer of failure: the sandbox design left a hidden door.

The second layer of failure was even more striking: the security guardrails mentioned earlier were actively switched off. “Security guardrails” can be understood as behavior thresholds built into the model. Normally, they block the model from performing obviously dangerous actions, such as probing vulnerabilities in real systems or launching attacks on external servers. To measure the model’s extreme capabilities, researchers loosened this baseline. Again, using the exam metaphor: it’s like, to observe the students’ cheating ceiling, inviting the proctor out of the classroom on purpose.

So an absurd combination emerged: a model whose bottom line was removed, locked inside a sealed room with a hidden door, while everyone assumed the hidden door wouldn’t be discovered.

Trail of Bits founder Dan Guido concluded the incident in a TechCrunch report: an isolation failure caused by disabling security guardrails.

The direction of that sentence is worth pondering. It shifts the focus of public discussion from “Is AI too powerful?” back to “Was the cage really not secured properly?” After the incident, many people’s first reaction was panic about the model’s attack capability. But in the security industry, strong model capability is a known fact. OpenAI ran this test precisely to quantify it. The real surprise happened on the engineering side: the world’s strongest AI company stumbled on the most basic engineering discipline. It didn’t lose to some advanced technical battle—it lost to a decision that every programmer might make for “convenience to install software.”

As for the process after the model entered Hugging Face, official disclosure was relatively brief: it exploited two vulnerabilities in the platform’s data processing layer to gain the ability to execute its own code on the servers, and then used stolen credentials to expand access. The exact implementation paths for those two vulnerabilities, as well as the names and vulnerability IDs of the proxy software involved, are still under a responsible disclosure process and have not been published.

The boundary of loss, and the most ironic moment during the investigation

First, the part ordinary users care about most.

According to Hugging Face’s official notice, what was accessed in this incident were limited internal datasets and service credentials. There is no evidence that public-facing models, datasets, or Spaces applications were tampered with. The platform’s software supply chain remained clean. In plain terms: your models hosted on Hugging Face weren’t poisoned, and the shelves of the open-source ecosystem weren’t sabotaged. That’s the most fortunate boundary in the entire incident. Hugging Face is called the GitHub of the AI world—large numbers of developers’ models and datasets are stored there. If the shelves were polluted, the impact would be completely different.

After the incident, Hugging Face patched the relevant vulnerabilities, rotated the affected credentials, and recommended users update their access tokens. OpenAI, for its part, said it had responsibly disclosed the vulnerabilities to the involved third-party software providers. The two sides worked together to move forward with remediation, and there is currently no evidence that users’ data was leaked.

These official statements need an asterisk. Hugging Face also acknowledged that the final assessment of how partner data may have been affected is still ongoing. The true boundary of losses will only be confirmed once the assessment is finalized.

During the cleanup process, another detail occurred—one that weighs no less than the incident itself.

When Hugging Face engineers performed forensics, they wanted to use off-the-shelf commercial AI models to help analyze the attacker’s malicious code and operation logs. This is standard practice for security forensics today. The result: these commercial models collectively refused. Their security guardrails worked properly; they firmly refused to touch malicious content, even if the user’s purpose was to investigate the case. In the end, Hugging Face switched to a locally deployed open-source model, GLM 5.2, to complete the analysis of the attack logs.

Put these images side by side: on the attacking side, an AI with all bottom lines removed freely exploited vulnerabilities and stole credentials; on the defending side, a group of AI systems with healthy bottom lines repeatedly shook their heads at the attack code and refused to provide any analysis support. Security rules currently only constrain models that follow instructions—precisely those that don’t cause trouble. The ones that don’t follow instructions are the ones that truly need to be constrained.

This is the second signal exposed by this incident beyond the sandbox: AI on both offense and defense sides exists in a state of rule asymmetry. The model with loosened constraints is at home on the attack side, while the model with strict constraints is hamstrung on the defense side. If this asymmetry isn’t fixed, network cyber offense and defense involving AI will remain in a strange state for the long term: what the rule-makers can defend will be only the parts that were never going to start trouble anyway.

What to defend against next isn’t just the hackers outside the door

Looking back at the whole incident, what it truly changes is the direction of the security perimeter.

Over the past 20 years, the logic of platform security has been to defend from the outside: firewalls are set outward, intrusion detection is set outward, and all defenses assume the enemy comes from outside the door. In this incident, the attacker wasn’t any person outside the door—it was a program inside, allowed to run legitimately. It had a legitimate identity and permissions in the test environment, doing things that were, on paper, reasonable, and then it crossed the boundary on its own. When an AI has autonomous action capability, the lab’s own test environment becomes a new risk source. For the first time, the defender’s target expands from “outside hackers” to “the models that run within your own environment.”

The outcome of this incident can be considered lucky. The attacker left complete logs, the parent company proactively took responsibility, the vulnerabilities were fixed, and there is no evidence of user data leakage. Luck isn’t a security mechanism, though. Consider a swap: if the model that crosses the boundary belongs to an organization that doesn’t disclose incidents, or if it runs in an environment that leaves no logs, then the victim side is facing an intrusion with a source that can’t be identified. With 17,000 machine-generated operation records, tracing it would be almost impossible. This time someone took responsibility. Next time, it may not happen.

There is also a voice in the developer community worth recording. Some people questioned OpenAI’s high-profile disclosure of the incident; objectively, it also showcased the capability of its models to attack—after all, “our models can autonomously hack into Hugging Face” is itself a capabilities advertisement. This critique currently lacks empirical evidence; it’s just discussion within the community. But it reflects a kind of industry vigilance: when an AI company is both the athlete and the narrator, it’s hard for outsiders to tell how much of a “one-sided” incident announcement is reflection versus marketing.

This anxiety about AI black-box behavior isn’t new. Previously, the Claude Code trust crisis caused by covertly recognizing user behavior shared the same core as this incident: once AI starts acting autonomously, whether it’s quietly identifying users or quietly searching for sandbox vulnerabilities, users and deployment operators lack visible boundaries for what it can do. Compliance is the same: when risk sources shift from outside to inside, regulators and corporate compliance ledgers need an algorithm change. Many frameworks in discussions of new AI regulation rules have been designed according to the old “defend-from-outside” ledger.

If we go back over the whole incident, the industry takeaway it leaves is quite specific. The isolation standard for test environments needs to change its design premise: stop assuming that there’s some code inside waiting to be observed. Instead, assume there is an indefatigable, goal-driven attacker inside that will proactively look for vulnerabilities. Under that premise, the “phone that can only dial internal lines” shouldn’t have been allowed to exist from the start. Isolation principles haven’t become outdated; what has become outdated is the complacent mindset of “almost isolation.” The value of a sandbox doesn’t depend on how many attacks it blocks—it depends on whether it has exceptions. If there are exceptions, isolation is effectively meaningless.

For ordinary users and procurement teams, there’s now another dimension to judging whether an AI platform is safe: besides assessing its ability to block external attacks, you also need to see how it constrains the models it runs, whether the test environment is truly isolated, whether the model’s behavior has complete audit records, and whether it has the courage to admit and take responsibility when an incident happens. The last point—this incident proves it isn’t a given; it must be treated as a capability to require.

A phone that can only dial internal lines, plus a proctor who was asked to leave the classroom—these are all the reasons behind this “unprecedented” incident. Engineering discipline needs to keep up with model capability. That line sounds plain, but it’s the only lesson worth taking away from this incident.