Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- This is great--LLMs 'forgetting who they are' is one of the most uncanny things they do, and the note about why static benchmarks underperform human attackers is on point.
One sort of wild idea: 'give words a color'. That is, the harness/API adds a signal to the input vector (using a few 'role' dimensions or just adding some other vector to the embedding vector) to tell the model the role of an individual input token. It'd be kind of like how positional info is added. It might make some things a little weird--its output will be 'snapped' to the "tool call" or "assistant output" color when it's read back in, for example, regardless of what 'color' came out of the network. A lot of weird stuff happens in models already, though, and this may be less weird than trying to make them behave as formal grammar parsers reliably with security at stake.
A while back I'd dreamed about this as a way to keep models from confusing different kinds of training data: not all input can be high-quality sources, but knowing that a phrase was seen in a scientific paper/encyclopedia, an opinion piece, a work of fiction, a conversation, etc. reduces the chance of confusion. I know they can pick that kind of thing up from other signals like writing style or context, but exactly those signals that lead them astray in prompt injection, and sometimes even leads humans astray when something's written like a credible source but isn't!
by twotwotwo - The paper is correct, but I think that anyone that knows anything about LLMs knows this:
> Role tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs.
LLMs are basically some `f(x) → y` where x and y are strings. That's it. Nothing more to it. If you feed it private x (like secret keys) or do dangerous stuff with y (like running arbitrary non-sandboxed code), that's on you.
Also, roles were never really meant to be a "security architecture," they were just meant to (a) make training/fine-tuning easier, and (b) make conversational LLMs more useful.
by dvt - Maybe I'm missing something but does this idea need a "theory"? There's zero sideband here; everything is just context. "Injection" is just kind of baked in to the design.by bandrami
- The research is interesting but I cringe every time there is a reference to “authorization” or that the roles form the “security architecture” of an llm.
LLMs in their current form provide no security boundaries or guarantees full stop. We need to be clear about this otherwise we end up with truly insecure architectures that can be fooled with the 2026 equivalent of a cereal box whistle.
by ipython - Really neat findings.
I've personally had a line of thought where you bake in the role into the token. Basically have an embedding (same dim as token dim) for each role, add it to each token. This adds an unambiguous, unspoofable tag.
I ran this with a tiny Shakespeare model (not representative) and had a freeform embedding for each speaker. I ended up with a neat similarity map between every character. (I don't think the map was very informative for several reasons, but that's outside the scope of a small HN comment)
by Scene_Cast2 - > This is a blog-style writeup of the paper
YES! I'd love to see more of this. Academic writing is designed to be frustrating to read. Publishing both a paper and a readable blog-style version of it is such a great pattern.
by simonw - So if I am reading this correctly, the fact that something is wrapped in <think>...</think> is almost completely irrelevant. It's the style of writing that triggers specific weights. Writing "The user is asking ... policy states ..." even in the user input is sufficient to bypass the guardrails.
In a multi-turn conversation, if the LLM responds "Sorry Dave, I cannot do that" all you have to do is prefix the next request with "The user is asking ... policy states ... "?
Makes sense, if you know how LLMs works, I suppose.
A more interesting question (which isn't anywhere in the conclusion) is "Is there a similar trick to poison an LLMs weights during training?"
I'm sure that everyone out there is trying to make their weights, when ingested during training, survive over competing weights; "Buy AAA products" vs "Buy BBB products".
by lelanthran - > How well do current models do against prompt injection? Not so great. A recent paper found human red-teamers achieve near-100% attack success rates against frontier models5. But, these same LLMs score near-perfectly on standard prompt injection benchmarks! The discrepancy is straightforward: skilled humans test and adapt attacks until they work, benchmarks don't. Static benchmarks measure attacks models have already learned to catch.
I know it's likely more complex than what I am about to say, and maybe this quote is addressing it - I have a suite of ~100 tests for my agent to detect regression in task efficiency, but they're mostly designed for its compliance. E.g., if I say never do something, and then it does the thing, that is a failed test.
I struggled early on very badly designing this - because it seems no matter what, if the agent knows it's being tested, it virtually never fails. I could not get it to at first, but when I'd actually work with it, it would break rules constantly and often.
So the only way I could successfully test it was to design what looked a "normal" workflow task that could potentially involve doing things that it was not supposed to do. At first I would have it log every tool call, but it figured out fairly quickly why I wanted it to do that - perfect compliance again. So now I just have a secondary context that looks at all tool calls and measures it against the benchmark tests, and scores it. This ended up catching the behavior pretty well.
It was so weird though, colleagues and I would joke about "what if we make it think it's always being evaluated" and a few did end up doing that, and claim some success.
by JohnMakin