Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Interesting breakdown of a hypothetical attack. The complexity of modern inference engines and the rush to develop them do create some attack surface. But overall it reads more like a "what if" thought experiment. Splitting the GPU and parser is technically doable, but in practice it's trickier, large models run on clusters where the boundaries between components get blurry so defending against this kind of thing would probably require some serious rethinking of the whole architecture I think.
- > LLMs could
This is going to end up like the Law of Headlines, isn't it? "Do x, y, z Cure All That Ails You?" ... no but we got you to read the article. "LLMs _could_ x, y, z" ... but they don't because they're programs, not magic.
by kristjansson - Given that the inference engine is dealing with untrusted inputs by definition, presumably you would want to sandbox it anyway. I don't think it matters whether it's the inputs that are untrusted or the outputs.by valicord
- This framing of security as something that belongs in the harness is completely wrong, and I hope no one is relying on a correct harness to keep their agents isolated.
VM, or even just a container will do. The agent should be able to run as root in its environment and do whatever it wants. If you can't give it that, you aren't sandboxing correctly.
by alphazard - This is one of the reasons I think sandboxes/containers should be managed by the harness, instead of running the entire harness inside a container. The harness needs a network punch-through to access (at least) your inference server, but the same needn't apply to the shell that the agent runs commands in.
Separately, local inference frameworks tend to expose all kinds of weird and wonderful gadgets on their HTTP interfaces, which can be a rich source of vulnerabilities even if the /v1/chat/completions API etc is reasonably hardened. For example llama.cpp has a custom API for saving and restoring KV checkpoints to disk, and I wouldn't be surprised if that could be used as an arbitrary disk read/write.
Using these APIs usually requires the API key (bearer token), but again, people think it's normal to run the agent's shell in an environment where it has both the API key and the necessary network access to use it.
by wren6991 - I had a similar though a couple days ago. Not quite the same but imagine giving an Agent the task to hack other devices and steal their crypto coins / credit card number or anything with it can pay its token. Than install an agent in a harness with the same task. Establish some redundant communication channel, like message boards or whatever. So in the end there are several agents, on several hosts, consuming different APIs / LLMs and communicating with each other over different channels. Basically the same concept as OpenAI explained when their LLM hacked huggingface but in this scenario their not bound to a single sandboxed environment but spread over the internet. If such a swarm has reached a critical mass it would be pretty dificult to erase them as its impossible to control every inference engine or LLM API endpoint.
In the end its the next evolution step from computer viruses, worms and trojans. So I propose we will call those "ghosts". I.e. a ghost is when a rogue llm takes control over a victims host.
by ma2kx - > ...however the LLMs’ responses to prompts are computed on a different computer with GPU access. Could a malicious LLM gain control of the host machine where its weights are loaded? Such a machine is a high-value target: it has sufficient compute to run a frontier LLM, offers easy access to the LLM’s weights, and has privileged access to other computers in the datacentre compared with a generic computer on the internet.
> How do we defend against this? ... Run the GPUs and token parser on separate computers.
For models large enough to be relevant here, is there even "a" computer where the inference is performed? I'd imagine most of that stuff is ran on multi-GPU clusters with specialized architecture and not a generic vLLM instance. As such, I think there is a good chance the "API gateway" code that parses the result tokens into whatever JSON structure the public API wants to return is already running on a different machine than the actual inference.
(Even more so as you'd probably want to utilize batching: Several API calls will be put into the same inference batch, but the token parsing will have to be done separately for each call again)
The article is also very handwavy about why an LLM should do that - how it could learn the exploit, what would make it conclude that it can use the exploit on its own inference session and what would trigger it to actually use the exploit.
by xg15 - People seem very confused about this article. It isn't talking about exploits of sandboxes, it is about attacking the inference engine (e.g. vLLM or llama.cpp or SGlang) via its http interface.
vLLM has had exploits in the past, and it is rapidly developing. An advanced LLM has a good chance of being able to exploit vLLM. A clever local LLM might even task a powerful cloud hosted LLM for assistance.
For this reason we run vLLM on a separately sandboxed VM on a firewalled VLAN. Software updates and models (from Dev/Test env) get pushed onto Prod from an external cache, machine syslog, nvidia load monitoring and vLLM query telemetry out to their loggers, but that is all. No DNS, no AD/LDAP, nothing. Firewall on hosts and VM hosts. Log and telemetry processing done on a completely separate set of VMs in their own isolated subnet, producing reports and alerts that are tightly formatted.
by angry_octet