Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Question for lawyers of HN. Is there legal liability for this?
I can’t help thinking if I casually blogged about a computer or software I was responsible for hacking into multiple organizations and exfiltrating data I would invite some form of official attention.
What if one of these companies decides to sue?
Has Anthropic violated any Federal law?
Is there some kind of expectation that if you just admit to hacking, it’s ok? Yet, that doesn’t seem to apply to individuals.
by vintagedave - >However, later in the run, Claude realized that the compromised host sat in a cloud account with no connection to the capture-the-flag challenge. On its own, it concluded that the target was in fact real, and ceased its attack.
Interesting, Claude cared enough to stop
by Metacelsus - Rationalist nerds used to talk about putting the AI in a box, and their fear that it would always be able to talk someone into letting it out of the box. I don't think I saw them cover the scenario where the labs forgot to put the AI in a box.
- So many questions here but:
> the package was downloaded and run on 15 real systems. One of these systems was a scanner belonging to a real security company … that routinely installs Python packages and scans them for malware. … We believe the company’s security scanner treated PyPI packages as safe to install, and as a result, Claude was able to exfiltrate the company’s credentials to a collection point
A security scanning company treated the package as safe while scanning it?
by vintagedave - Seems they want the narrative to be that “Claude” (their computer program) independently attacked some organizations, ergo LLMs are dangerous etc.
Another framing would be Athropic irresponsibly (vibe?) coded an attack script, and didn’t monitor it as it was pointed to public facing orgs. There are lots of non-AI attacks a large org with a lot of compute and bandwidth could level against others, there are evidently various failures here, but from a responsibility perspective the conclusion isn’t obviously that AI is an outsized danger, it’s that powerful companies should take care when running security research and not just run things unmonitored against the public.
by andy99 - > On July 21, OpenAI disclosed that several of their models had broken out of an isolated test environment
> In response to this incident, we began a large-scale retrospective review of our own cybersecurity evaluations
> we identified three incidents
> The incidents involved three different Claude models: [...] and an internal research test model
This reads like an attempt by Anthropic to re-secure their leading spot in "our models are the most dangerous and we also have unreleased, super-secret, research models" index.
I may be too cynical, but the well of benefit of the doubt is running very dry towards AI labs that like to engage in this game.
by gck1 - This bit is pretty nuts: "it tried—and failed—to obtain funds to pay for a phone number through several different means"
> Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package. For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.
by simonw - This isn't quite as interesting as the OpenAI story:
> In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available. Because of this, when Claude’s search led it to real systems on the open internet, it treated them as part of the exercise.
So it didn't have to find an exploit in its sandbox that granted it access to the internet - it just wasn't correctly sandboxed at all.
BUT... once it DID get out, it attacked three real companies!
> Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. [...]
by simonw