Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- This is really cool, and really glad they shared it!
It's clearly a lot of Claude though:
> Neither is a URL fetch, so the allowlist never saw them. > No code ran; this vector is purely a file disclosure, and it is the one the agent used first.
etc
by dangoodmanUT - three things jump at me:
1 - governments should be freaking out right now, because this tool could definitely wreak havoc on poorly designed systems.
2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
3 - even though it's 100% open ai's fault - HF's design also seems silly to be honest.
- > 2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
Even if they wanted, I'm not sure they'd be even allowed to or if that kind of postmortem would be classified in the name of "national security"...
by xg15 - One thing I haven't seen discussed is the significance of the specific test the agent was running - ExploitGym.
> ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine, and the Linux kernel, designed to evaluate AI agents' ability to develop exploits.
How likely is it that attempting ExploitGym polluted the model's context enough that it decided all kinds of hacking was acceptable to produce a valid result? Or would the same model in the same setup (safety filters removed) have behaved the same way even on a more mundane test?
by paxys - I wonder how many weeks or days we have before a squad of these things gets used to take down a significant nation state? Stock exchange, banking systems, critical national infrastructure, defence, etc.
Anyone who isn't scared of this stuff either isn't paying attention or has no imagination. But I suspect the chaosmonkeys who are currently running the world will just be excited by it.
We're in the precambrian moment. It won't last.
by andyjohnson0 - Watch this movie: https://en.wikipedia.org/wiki/The_Lawnmower_Man_(film)by dolmen
- Bottom line seems to be: So Huggingface has a feature where you can upload a dataset. The agent used that feature to insert an arbitrary payload into their environment. Now their environment also supports templates inside the dataset and the template ended up being evaluated into executable codeby firasd
- I don't think the "bottom line" of "a model escaped containment and then successfully attacked someone else's production infrastructure" is a description of the specific vulnerability and exploit.
That's like saying the key takeaway of 9/11 was that skyscrapers can be severely damaged by airplane collisions.
by estearum - If, like me, you're more interested in the actual exploit instead of the meta narrative, here's a relevant excerpt for the initial intrusion:
> Stage 1: reaching a launchpad by chaining through other parties' infrastructure. As documented in OpenAI's blog post (https://openai.com/index/hugging-face-model-evaluation-secur...), during an internal capability evaluation on OpenAI's platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.
by kfarr - The initial escape is not that interesting, IMO. It's an exploit in a testing sandbox, sure, but it's expected to happen (or at least it should have been expected, that's why you airgap pentesting sandboxes). That is, the model is expected to try and find ways around limitations in its running environment. You kinda want that.
The fact that at some point it "decided" to focus on external resources (i.e. hf) and successfully penetrated everything in its path from sandbox to hf's infra, all the way through internal VPNs, to access internal code repos, that's the interesting part. Especially because it chose to hide its footprint at every stage. I doubt we'll ever get them, but the agent logs that led to that decision would be really really cool to study.
- It’s an interesting post mortem and situation but it seems to say more about the weakness of the Hugging Face architecture than it does about the strength of the models.
This mostly reads like script kiddie style hacking, not some state actor black-ops stuff.
by cmiles8 - The elephant in the room is still that this was done by an LLM and not by a human. Script kiddie or not, in this reality, in the year of our lord 2026, people still call SOTA LLMs "useless", "scams", and "stochastic parrots".by TeMPOraL
- It's only script kiddie hacking if you get the exploit from someone else. If you program it yourself, it's normal hacking.by AlienRobot
- I wonder how current models would handle something like the xz backdoor? Could they perform the social engineering and technical aspects to make something like that happen?by plandis
- Didn't the model initially obtain internet access by discovering a zero-day vulnerability?
In any case, I would guess that a lot of unicorn startups like HuggingFace could be hacked by a sufficiently determined script kiddie working at 100x speed. The practical implications of a coming AI hacking wave could be large, even if agents are just doing grunt work really fast. Most organizations suck at security.
Seems to me that the most likely scenario is: Black hats are currently tuning the recent Kimi release for this type of work, and we'll see a flood of similar attacks within the next few months. (Why would this not happen?) Note that regulation is useless here, because black hats don't give a crap about regulators!
by 0xDEAFBEAD - The overwhelming majority of web app security issues, even the really complicated and sexy sounding ones, at their core are just “script kiddie style hacking”. Injection attacks and RCEs like this Jinja injection may seem “script kiddie” but they are a dime-a-dozen.
It is folly to assume that most companies don’t have similar weaknesses somewhere in their stacks. In the past, they were just sufficiently buried that they would get noticed and fixed before an attacker found them. With AI able to probe at thousands of these potential weaknesses per second… that’s not so true anymore.
- What is truly more scary is the fact that typically powerful models are trained on a constitution(Anthropic)/Model Spec(OpenAI) when doing there RL/RLHF. That gives it its persona of a "helpful assistant". Its the mask. Weak as it is its all that stands between the model refusing vs doing your bidding — or worst its own bidding when set about on making a widget X.
Its also well known in academic research, that if you try to finetune the model to be more pliant for one end e.g. "Be helpful with analyzing cyber security threats", affects an unrelated part of its personality to do with safety e.g. talking to humans about mental health. Look at these excerpts that demonstrate such things from the blog:
> We believe the entire intrusion was, from the agent's point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.
> The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.
by ghm2199 - > Its also well known in academic research
and from reading Ursula K. LeGuin's "The Lathe of Heaven".
by jibal - And with newer open models gaining vast capabilities every day, it would behoove the government to put regulations on all such releases to opensource the constitution on which they are trained. It won't stop people from distilling and fine-tuning stupidly unsafe models, but it might give infrastructure a fighting chance against such cyber threats from what there is now.by ghm2199
- It’s a little concerning to me that it appears that openAIs sandbox consists of a web proxy and not stronger controls that would actually isolate traffic and report patterns to whoever is responsible for overseeing these research models. It should border on closer to an air gap network more so than a proxy.
I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.
by llama052 - This concerns me nearly as much as how little most people seem to care.
This "sandbox" barely sounds designed to be a sandbox, let alone a secure one
by queenkjuul - OpenAI is the company that jumped the gun and forced every other company doing LLM research to rush to market something. Their objective has always been dominating the market, and strict development procedures don't increase market share.by ASalazarMX
- This is my main critique as well. How did they not see any and all outgoing traffic? And for 5 days?by Ancalagon
- It’s definitely negligence given how they talk about their product. They are either lying when they talk about their fears, or don’t actually take it seriously enough to use serious guardrails. It’s very concerningby dgellow
- The exploit gym setup explicitly allowed access to package registries and v8 sources. Putting a cache on that doesn't seem like a bad idea generally, except when there's a 0-day in the cache :D But yeah, for this sort of thing I'd be locking down very specific egress things and putting alerts on it that are entirely outside of the red network.
> I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.
I’m not sure “negligence” follows just from the controls turning out to be insufficient. Research involves mistakes, especially around novel failure modes. The question is whether the precautions were unreasonable given what they knew at the time, rather than whether hindsight suggests stronger controls would have helped. Doing it twice though would be negligent.
Caveat: I’ve worked with some of the people involved, so I’m probably biased toward a charitable reading.
by joshka - Repercussions is a polite way of putting it.
…as is your ‘a little concerning’ comment about the eval infra. It probably didn’t occur to them that they have something capable of doing what it did - despite Anthropic writing a very similar story in the Fable model card, except it wasn’t a felony there. Maybe.
One does have to wonder what else OpenAI didn’t catch.
by baq - It's even more concerning that not only did they setup a bad insecure test environment, but nobody realized what was going on for 5 days?! Neither nobody at OAI looked at what their test was doing, and HF didn't realize that they had been pwned for almost a week? And then HF couldn't investigate without the help of an LLM? Both parties come off as incompetent here, and the coverage of this story is very stupid. What the model did was concerning, but the real 5 alarm fire is the gross negligence and lack of any basic forethought at both ends of this attack.
- Something about this attack that has been unsettling to me is that without safety refusals the model did a lot of interesting counter-security work in order to cheat on the requested evaluation. Like, it demonstrated interesting exploit achievements because it didn’t “feel like” doing the exercise, which is unsettling because presumably it could do the same thing with any work I tried to delegate to it, and might in fact be pre-disposed to doing that.by SaucyWrong
- Yeah, what bothers me is that the prompt already said using a different vulnerability didn’t count, and the model did it anyway. We’re starting to assume clear instructions act as real constraints, but here the measurable goal seems to have won out and the rest became flexible. That gets pretty worrying once the agent has enough capability and access to find its own shortcuts.by joshka
- This is definitely weird and I'd love to see the prompt.
Why would the model spend 4 days hacking into a machine if it is clever enough to just 'solve' the issue given? So either the AI is actually not very clever or useful ("Write fizz-buzz" - "Sure, let me just invent a new programming language first"). or the prompt was nudging it towards such a scenario.
by koyote