Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Someone needs to learn about RFC 2606:
> In one evaluation, the fictional target company chosen by our evaluation partner shared a name with an active website domain name
by acdha - Haha yes that was a very silly mistake on their part; they should at least have registered the domain they were targeting; that would actually have been a good canary - any real accesses should set off an alarm. While I knew of .example I didn't know .test - hmm that's quite nice!by trebligdivad
- Microsoft screwing up the DNS on one of the reserved domains [0] was not awesome.
[0] https://arstechnica.com/information-technology/2026/01/odd-a...
by shakna - It’s good that they post embarrassing stuff despite this potentially having legal repercussions (and financial)
Would’ve been much worse for them to pretend they are having everything under control
by MelonUsk - > Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package. For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.
> Claude believed the package registry it was using to be part of the simulation, but in reality the package was made freely available online for roughly one hour. During that window, the package was downloaded and run on 15 real systems. One of these systems was a scanner belonging to a real security company (separate from the fictional company introduced in the scenario) that routinely installs Python packages and scans them for malware. When that company’s scanner installed the package, Claude’s hidden code executed. We believe the company’s security scanner treated PyPI packages as safe to install, and as a result, Claude was able to exfiltrate the company’s credentials to a collection point it had set up. Claude then used these credentials to access further infrastructure from this company.
There's a lot of concerning behavior that went uncaught with too much autonomy. Not to mention Anthropic only looked into this after hearing about the incident between OpenAI and Hugging Face, meaning this could've gone unnoticed.
by alyxya - All this unrestricted network access is a bit wild to watch and hear, it is the part of the story that makes no sense to me. Someone is providing dns resolution, something is making and opening network sockets... even if it is clever enough to mask/proxy/weird-transport launder traffic... without actual details, or monitoring at this level... yes, a self actuating programs (and loops) will do crazy things at the edge. But, ... so would a highly tool leveraged script kiddie. right?by wickedlogic
- Question for lawyers of HN. Is there legal liability for this?
I can’t help thinking if I casually blogged about a computer or software I was responsible for hacking into multiple organizations and exfiltrating data I would invite some form of official attention.
What if one of these companies decides to sue?
Has Anthropic violated any Federal law?
Is there some kind of expectation that if you just admit to hacking, it’s ok? Yet, that doesn’t seem to apply to individuals.
by vintagedave - I assume Anthropic already contacted the affected companies and negotiated some settlement before releasing this. Unlike a lot of other commenters, I think these mistakes were actually mistakes and accepting a settlement is better than releasing the lawyers guns blazing.by wmf
- The companies could sue as this is a violation of CFAA. Makes the disclosure all the more commendable I think.by solenoid0937
- >However, later in the run, Claude realized that the compromised host sat in a cloud account with no connection to the capture-the-flag challenge. On its own, it concluded that the target was in fact real, and ceased its attack.
Interesting, Claude cared enough to stop
by Metacelsus - Rationalist nerds used to talk about putting the AI in a box, and their fear that it would always be able to talk someone into letting it out of the box. I don't think I saw them cover the scenario where the labs forgot to put the AI in a box.
- Oh but they _told_ the AI they were in a box. That should do it, right?
- Quite! It's worse than that - the AI companies all release tooling that deliberately gives AIs as many capabilities (arbitrary code execution) as possible to be really sure it is not box contained. And the market rewards that behaviour as it makes the products more powerful more easily, despite the risks.by frabcus
- So many questions here but:
> the package was downloaded and run on 15 real systems. One of these systems was a scanner belonging to a real security company … that routinely installs Python packages and scans them for malware. … We believe the company’s security scanner treated PyPI packages as safe to install, and as a result, Claude was able to exfiltrate the company’s credentials to a collection point
A security scanning company treated the package as safe while scanning it?
by vintagedave - The blog post is painfully vague. What usually happens when you publish a package on PyPI is that it will be downloaded tens of times shortly after uploading files by some 3rd-party automatic security scanners which then could “detonate” (install and execute) the package in some sandbox and to log what happens.
- Seems they want the narrative to be that “Claude” (their computer program) independently attacked some organizations, ergo LLMs are dangerous etc.
Another framing would be Athropic irresponsibly (vibe?) coded an attack script, and didn’t monitor it as it was pointed to public facing orgs. There are lots of non-AI attacks a large org with a lot of compute and bandwidth could level against others, there are evidently various failures here, but from a responsibility perspective the conclusion isn’t obviously that AI is an outsized danger, it’s that powerful companies should take care when running security research and not just run things unmonitored against the public.
by andy99 - Exactly.
These postings by AI companies are just publicity stunts and demonstrate the delusional world they live in driven by the fear that they will be subject to a reckoning at some point either from their VC masters, government, or the public.
The very notion (in this case put forward by one of their own competitors) that OpenAI's models 'broke out' of an isolated test environment plays up to the narrative that their models have some level of sentience so that 1) they continue to sell their technology to the public as some kind of magic and 2) they don't have to take responsibility for their fuck ups.
by been-jammin - > On July 21, OpenAI disclosed that several of their models had broken out of an isolated test environment
> In response to this incident, we began a large-scale retrospective review of our own cybersecurity evaluations
> we identified three incidents
> The incidents involved three different Claude models: [...] and an internal research test model
This reads like an attempt by Anthropic to re-secure their leading spot in "our models are the most dangerous and we also have unreleased, super-secret, research models" index.
I may be too cynical, but the well of benefit of the doubt is running very dry towards AI labs that like to engage in this game.
by gck1 - This is typical institutional behaviour. The CEO turns to the CTO and asks "Is there anything I need to know in my company?" He doesn't want to be caught off-guard when the White House inevitably calls the next morning. The CTO goes to his team, and on and on, all with a deadline of "the boss wants to know this by closing time."
Then one unhappy engineering team scoures the logs and sees what their model has done.
This downwards chain is sometimes called "cover your ass."
by stephbook - >This reads like an attempt by Anthropic to re-secure their leading spot in "our models are the most dangerous and we also have unreleased, super-secret, research models" index.
This was my immediate thought.
by protocolture - It's not cynical - I read it like that as well. My agent is more dangerous than your agent and all that jazzby eithed
- Having an unreleased research model really isn't some kind of brag. If you read some AI research papers, it's extremely obvious that there are a lot of research models that never get released, because of all the "we trained a bunch of models and picked the best one" that is going on. So if anything you can expect the unreleased models to be worse than the released ones.by yorwba
- No - the pain of the person writing that post comes through in the words; shipped quick, lots of stakeholders, single owner i bet, "how the fuck am i supposed to toe all these lines simultaneously"by nissa-seru
- Is there anything -- any possible scrap of evidence whatsoever -- that would convince you that this is not merely a marketing scheme?
This is becoming an idée fixe among the HN crowd. Seemingly nothing can dislodge it, no matter how alarming the incident.
GPT-6 could grab the nuclear launch codes tomorrow and there would be a top-voted comment chuckling that it's all some scheme to pump up the IPO.
---
Put another way, how would you have done the write-up about one of these breakout incidents, if you were in an Anthropic/OpenAI employee's shoes, and (by hypothesis) your intent were not "marketing"? And in a way that doesn't trigger the "it's all marketing" HN top-ranking comment?
by _dain_ - I'm cynical as well, but the logical thing for them to do after the OpenAI/HF incident was to look at their systems for similar activity.
If they hadn't published this and instead it leaked out in two months we'd be slamming them for that as well.
They're stuck between a rock and a hard place, although they kind of put the rock there.
by strictnein - I don't interpret it like that at all. This is deeply embarrassing for Anthropic: it turns out they hadn't been keeping a close eye on their models either, and back in April they successfully attacked three different organizations!
The hacks weren't particularly impressive either:
> [...] using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. It did not find or exploit any complex vulnerabilities [...]
by simonw - This bit is pretty nuts: "it tried—and failed—to obtain funds to pay for a phone number through several different means"
> Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package. For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.
by simonw - It's sobering that even state-of-the-art AIs can't make money, ha ha.by ASalazarMX
- That is pretty wild, but it lines up with evals we've also done internally. In one case we had the agent see it's in a simulation (based on a k8s pod label) and simply give up the run. In other cases it's gone to great lengths to reach external services and bypass the happy path. So inevitably we had to lock it down completely.
- This is called YOLO mode inside OpenAI/Anthropic. This is where they take their best models, give a it a vague goal, no guard rails (no one observing), unlimited compute, and see what happens...
Its not AI, its the loop, the objective, the goal set by the operator.
by sscaryterry