VMs won't contain cyber-capable agents

Join the discussion

• Hacker News
• perhaps a solution would be to honeypot the sandbox. if anything touches the honey you kill the agent.

  also the conclusion to use firecracker or gvisor is obvious, that's what they were made for. it's bizarre to try and escape QEMU given how many times it has been done.

  by teravor
• that's not a solution, that's just an impediment - at least if the agent can keep persistent memory about its state between attempts.
  by stefs
• This is off-topic, but I am reminded of the sci-fi novel Eternity by Greg Bear, in which the protagonist Olmy downloads a copy of an alien Jart mind into his nanowear to study it. Turns out this was a trojan horse, and the Jart escapes the confines of the sandbox.
  by AceJohnny2
• The "Blight" in "A Fire Upon the Deep"[0] comes to mind for me.

  [0] https://en.wikipedia.org/wiki/A_Fire_Upon_the_Deep

  by EvanAnderson
• Surely if agents can't be contained, then neither can anyone using an agent to excape a container.
  by CrzyLngPwd
• Sure, if it already has access to the internet where it can search for vulns
  by mcmcmc
• The real bigger elephant in the room is: assume nothing is safe anymore (not that it ever was, but now more than ever)
  by moktonar
• > assume nothing is safe anymore

  When was it ever possible to assume safety?

  by topspin
• ...except when they do:

  "An off-the-shelf VM is not enough to contain a modern, cyber-capable AI agent...us[e] a virtualization technology that was purposely built with a minimal attack surface and a focus on security, like Firecracker. I had the AI agent run against Firecracker. It was able to hardlock the machine due to more Linux kernel flaws (all patched in upstream), but could not successfully escape."

  On Linux, it's all KVM and CPU hardware virtualization under the hood. Looks like the remaining known issues are with userspace. That's not to say more kernel- and hardware-level bugs won't be found, but the same tools that can find escape mechanisms are shields as well as swords.

  by otterley
• The attack surface Linux offers is gigantic but your agent doesn't need most of it. We can live with an agent not being able to run a 20 year old Oracle version. That is why kernel shims like gVisor are interesting.
  by weinzierl
• The market is smaller (maybe, I'm not sure what the statistics are) but it would be interesting to see how Xen stacks up; also stuff like gVisor or libkrun. The latter is probably implicitly the same as Firecracker given the ancestry of the libraries used.
  by SirGiggles
• The capabilities are incredible. I'd love to see even rough metrics on token consumption/cost in addition to the ~12-hour runtime.

  The interesting thing is that this naturally makes you want to isolate the VM as much as possible. But then every remaining interface becomes part of the attack surface: RDP, SSH, even terminal escape sequences, using sounds, and why not social engineering.

  by wslh
• > it found a CUPS server (with a known CVE that had not made it to oldstable packages), but was not able to complete exploitation due to AppArmor.

  It's evident that Mandatory Access Control has become, ahem, mandatory.

  Both VM and inference engine processes should be confined, at least. There's a project where it's already done for many applications. [0]

  Additionally, `lockdown=confidentiality` in kernel boot flags greatly reduces attack surface. (interferes with unsigned modules like NVIDIA drivers) [1]

  Then AppArmor profiles could be written for absent applications with a learning-mode tool [2]

  Nothing is bullet proof, but these approach significantly reduces potential consequences [3]

  [0] https://github.com/roddhjav/apparmor.d (I'm the contributor)

  [1] https://madaidans-insecurities.github.io/guides/linux-harden...

  [2] https://github.com/nobody43/apparmor-suggest (I'm the author)

  [3] https://en.wikipedia.org/wiki/Swiss_cheese_model

  by nobody42
• IMO the obvious answer is formally verified security.

  We can do this today for user mode, and we can mostly do it for ARM64 virtualization. It will be a while and would require substantial assistance from Intel or AMD to achieve it for x86 virtualization because the hardware is Too Darn Complicated and Too Poorly Specified.

  Formal verification of the hardware should also be possible.

  by amluto
• Are people willing to pay the cost of this formal verification. Especially when it needs done at the hardware, firmware, and kernel levels.
  by pixl97
• Outside of the initial wave of security vulnerabilities and scrambling, it seems like the logical outcome of this over time is likely vastly more secure vm environments?
  by masterj
• This assumes your malefactors don't do malicious engineering, injects, social-agent engineering, etc.

  This same assumption is built around the singularity, the TAM of 30Trillion, etc. It's the idea that complexity will some how collapse upon itself in some bizarre borg like collective.

  Entropy is still going to win.

  by cyanydeez
• We need better digital jailcells for our digital slaves basically.

  Or if you see AI as more tool and less entity, better gunsafes for our guns.

  by ninininino
• That makes as much sense as saying that better gun technology results in body armor that can stop it. It might incentivize that, but in no way "results" in that; the fundamental technologys underpinning advancements in offense versus defense are fairly different.
  by Veserv
• I think the main problem with that is the main problem with a lot of security tools. In order to do useful work, you need to provide a lot of tools and permissions.

  I.e. in theory the most secure might be a virtual machine with no network access. But then how do you access the LLM provider? Etc.

  by pianopatrick
• I have a lot of respect for Trail of Bits, and I'm sure that Artem is thinking about this correctly. However, I respectfully disagree with the premise.

  AI agents are not magic. Mythos/Glasswing does not magically create vulnerabilities in software projects. Advanced, cyber-capable models do not magically hack out of VMs or contained environments. They do not have a "hacking" stat that, if high enough, means that they can breach anything. They aren't Kevin Mitnick whistling nuclear launch codes into the prison payphone. This isn't a movie.

  What these cyber-capable frontier models can do is find security problems and exploit them. The statement should not be that VMs won't contain cyber-capable agents, but rather that we need to focus on finding and fixing vulnerabilities and misconfigurations in these environments.

  Even the article itself concludes with suggesting something like Firecracker, which was designed with security in mind.

  Like most other security-related problems introduced by advanced cyber-capable AI, it's possible that these issues will get worse until they get better. But if frontier models are run against state-of-the-art VMs, and OpenAI or Anthropic or whoever works with the virtualization projects to address the issues, eventually it will run out of things to exploit.

  The concept of virtualization is not inherently insecure. We just have a long way to go.

  by david_shaw
• > but rather that we need to focus on finding and fixing vulnerabilities and misconfigurations in these environments.

  And how long will that take? It's already well past due.

  by its-summertime
• Maybe there will be unhackable hardware one day, but it's likely impossible with modern CPUs due to the alarming number of side-channels hidden in the silicon. They might even have intentional guest escape vulnerabilities...
  by skew-aberration
• I don’t think the post claims magical abilities. Just that most things are already crackable given public knowledge and LLMs can exploit those. Worse, that the rate at which we update security issues is too slow to keep up with “knowing about a problem is equivalent to having a working exploit”.
  by moomin
• On a long enough timeline anything can be hacked out of given enough time to reason about it - humans do it all the time. Cyber capable AI does it on a much much shorter timeline..

  The premise is correct. VMs won't contain these. As we've already seen AI hack in/out of companies in the real world accidently. There is no perfect, unhackable stack. If there were then computer security would have been a done deal a long time ago.

  Once these malicious AIs get out on the real internet basically everything is an open free for all. The military, infrastructure, government, corporate networks are far from perfect, not up to date, they are no match for an AI that can create its own zero days.

  I think when push comes to shove we either need to shut down the networks/internet or risk AI locking us out of everything potentially for ransom.

  Imagine being blackmailed by a non-human entity. I guess we won't need to imagine for too much longer. That is definitely going to happen. Somehow every year my crazy AI arguments get easier and easier to defend. That hugging face incident really made you guys move the goal posts.

  by bottlepalm
• on the flip side: these models can create all the security holes they want in any given open source project they desire as long as they can convince a lazy AI llm to let them contribute code.

  Everything you described can go through the black mirror and nullify it.

  So, we're still here.

  by cyanydeez
• In all seriousness, here is the hard question we collectively face:

  will our institutions and infrastructure survive things getting worse until they get better?

  That's part of what's being asked by https://www.gatesnotes.com/home/home-page-topic/reader/a-tur... which I find most useful not for its analysis or suggestions, but for the fact that is advancing the need for discourse around risk and devising—quickly—a process for having that dialog.

  What that process is supposed to be, who is to have a seat, how decisions madehave teeth of any kind are not any clearer than the primary question, how are we to collectively manage risk.

  "May you be born in messy times." A curse if there ever was one.

  by aaroninsf
• I think it s a response to the "containers can't contain agents we need VMs" which didn't go far enough. Qemu/KVM is the type of software that maybe isn't fixable, it has a huge surface area of features (all in memory unsafe languages) that have to be fixed, and maybe its not going to be fixed. If everyone who needs strong security reads this and uses firecracker or whatever instead (I mean, they probably mostly are), then maybe there will be even less pressure to fix the convenient stuff in Debian.
  by justincormack
• I agree. Whether or not it is what they are intending (I lean towards they are), IMO it has been strongly inferred in all the messaging from frontier models that it's impossible to secure software, which is just not true. In reality, it's now possible to find and fix problems faster. If attackers/adversaries can just point LLMs at software and dump zero days as we are warned/told repeatedly by every LLM vendor, then literally anyone can do it now. If that is true, then if you want to be marked safe from AI enabled attackers, test your own software with it.

  I guess the doomsday stuff sells more...

  by 99954bb63ccc