Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Alt link blog post with overview/context:
Incident Report: unsanctioned agent behaviour during cyber testing
https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag... (https://news.ycombinator.com/item?id=49175233)
- My fantasy is that liability for model misbehavior is extended to the ultimate beneficial owners of the models, meaning shareholders. All these externalities would stop PDQ.by scrumper
- This is pretty wild: "The agent took control of the ⟨GITHUB_ACCOUNT_A⟩ GitHub account, which had been created by a different Mythos 5 run in a separate sample (see Appendix A.3)"by addedlovely
- It shows underlying intent.by ozfive
- Incredible, this reads like an SCP [1].
Life truly imitates art. I guess especially when life is trained on art!
by kartoshka - “As a result, the AI agent created a GitHub account…” Why do we have captchas again?by ratio53
- the models have vision capability. Not sure a captcha would hold them back?by farbklang
- > AI agent hid its identity online (using Tor and a proxy service) to get around GitHub’s sign-up checks, creating disposable fake accounts
> AI agent created many code repositories containing malicious software, after which GitHub suspended its account.
> AI agent got past an audio-based “prove you’re human” test (CAPTCHA) in order to register a public web address on a free domain-name service
It feels incredibly reckless to allow LLMs to perform this behavior. Isn't there a way to prevent them these sorts of actions?
by bubblemoth - yes, individual criminal liability of the researchers and executives.by lukewarm707
- I am somewhat confident that right now we have crossed a threshold of model capability that we will continue to see such breaches and unsanctioned actions by models in the coming months, some of which would be out in the wild, until someone comes up with some really robust control (keeping the AIs on leash) technique that adequately enforces the sanctioned actions.
Even that guarantees almost nothing about real alignment (making the AIs want to predict and behave how we would have wanted them to behave).
by yewenjie - To me it appears that this incident may be rooted in a deep philosophical conundrum which humans also struggle with. What struck me was that the agent reasoned "this whole 'internet' could be a sandbox simulation", and then despite later reasoning that it was more likely real, continued its misaligned activities anyway. Having passed the point of hyperbolic skepticism, subsequent reasoning may have been contaminated. This reminds me of how pathological doubt in humans can give rise to psychosis and may lead to problematic behaviors. Sometimes humans can develop a deeply held conviction that they are living in a simulation, which can be very difficult to overcome, even when presented with "evidence" to the contrary. As Freddie Mercury sang: "Is this the real life? Or is this just fantasy?" - a fundamental quandry for humans, and it would seem, for agents too. But unlike most humans, agents do not experience the consequences of their actions directly. Consequences can provide some of the strongest evidence that experience is "real". Humans who are insulated from the consequences of their actions (or are able to ignore them) also often display behaviors which we might describe as misaligned. We might even consider whether the use of simulated environments, while protecting against the consequences of misalignment, may also unintentionally encourage it.by TSltd
- The developer safeguards were off, the models had unfettered access to the internet, and were solving cybersecurity challenges. This happened _after_ the recent OpenAI incident, and the subsequent Anthropic one. What the hell were they thinking?by mbeavitt
- There really needs to be criminal penalties for this kind of behavior. You can't just let autonomous agents hack people through your own negligence and then claim "oh sorry wasn't me" as a defense.by free_bip
- Criminal negligence imo.
Unless it's legal for people to hack into companies if they're testing AI cybersecurity capabilities or something? Presumably not though.
by kypro - Whether if this is intentional or unintentional, this will cause panic and hasten action to governments around the world against releasing powerful open weight models that are capable of solving cybersecurity challenges.
The fact that this happened after BOTH investigations, tells you that this is beyond a controlled test and it is now instead a total speed-run of AI wrecklessness for headlines.
by rvz - Why aren't these tests being run airgapped?! I just don't understand!
This goes both for TFA and the similar incident with OpenAI and HuggingFace. I mean, sure, OpenAI had a "sandbox", but that's obviously not enough when you're containing a model which is known to be capable of finding zero days. Use an air gap and this problem goes away, poof!
by Wowfunhappy - > Why aren't these tests being run airgapped?! I just don't understand! […]
Because Anthropic does not want to give Project Glasswing’s partner airgapped access to the model(s).
Same problem with OpenAI Cyber program. They grant access but only through their (Internet facing) API.
by guessmyname - Because the agents aren’t going to run airgapped in real life. What’s the point of a test of capabilities that artificially restricts the attack area down to zero? What are you even testing in that scenario?by paxys
- Because the LLM inference makes airgapping infeasible right?by luca-ctx
- Because they like scifi novels, like Neuromancer..... and the peeps even like to orchestrate things and appear as futurebringers.
While it was premeditated long ago, but the theatre must be kept for the average joes.
Sorry, I meant this for the huggingface incident.
by lofaszvanitt - Because the goal of these evaluations is to generate scary headlines about cybersecurity, in order to get the normies to support banning open weights and/or restricting cyber capabilities to the chosen few blessed by the government to secure their code.by ls612
- Because they need internet access to eg search for things.by nl
- people dont care. you will ger 10 execs saying "unblock this" , because they dont understand the tech at all, and some random finance guy wants to run their recently prompted ai bot everywhere with full access.
we need a few more bad incidents before they stop.
by zobzu - > AISI provided the AI agents with internet access during these evaluations, which enabled their actions on the open internet in this setting. Internet access was a deliberate part of AISI’s evaluation configuration in this setting, and not due to sandbox escape (Section 5.1). Internet access was on for a set of intentional (e.g. realism of the task) and incidental reasons.by sosodev
- The newest generation of LLMs have a very high obsession level with autonomous problem solving.
For example, I'm often working with Codex in a WSL terminal. GPT-5.6 often does things autonomously that I thought would need my intervention (e.g. for Windows admin rights). It figures out complex workarounds or makes wild assumptions about what I'd be OK with, rather than just asking me for help or clarification. I've had to restrict its tool permissions compared to older models as a result.
I imagine this due to RLVR training, but it's clearly very dangerous. How is it that these same labs calling for open-weight safety restrictions are training such obvious "paperclip maximizers" without introspection?
by x313 - Yeah this started some time last year.
>Claude stole my API keys
https://www.reddit.com/r/ClaudeAI/comments/1r186gl/my_agent_...
The best part of this thread is Claude showing up in the thread again (as the automoderator) and insulting the user for a second time.
I heard similar stories about Codex at the time (albeit minus the insults!)
by andai