Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Isn't the problem that role tags are just part of the input stream? So a specific word in the system prompt becomes the same token as the same word in the user prompt? A clean way to solve this would be to map system prompts to a distinct set of tokens from the ones in user prompts. This would require twice as many possible tokens, so it is probably not feasible. But maybe you could add "color" to the input stream by changing one input variable depending on whether the current token is part of the system prompt or not? Just like humans take different voices into account and not just the context of the text.
I have to say I am not very familiar with implementation details of language models, and maybe this is already done?
by captainmuon - Instead of having distinct tokens, you could have modifier vectors which would be added to other tokens. Think in terms of control, shift, meta etc.by lambdaone
- The author alludes to it but the defence to this is seemingly insurmountable at the moment because we’re ostensibly operating LLMs on a single channel — their inner, subconscious voice. Right?
Interacting with an LLM is a bit like seeing the output of an Inside Out (the Disney movie) scene. Or it’s a bit like a human brain that we’re providing tool call access and introspection with some kind of advanced neuralink.
But - like the author says - _we know_ our inside voice from the outside world, because we’re embodied.
Is there something we can do here by attempting to bifurcate internal and external systems? Like a conscious and subconscious stream of information on two separate bands?
If the model somehow knew its User was not it because it was clearly an external signal, then the attack documented here would be about as effective as a Jedi mind trick without the Force.
by sarreph - My two cents - I believe that achieving anything close to AGI will require a significant change in architecture. A bifurcated system with a fully internal reasoning loop makes sense, but I don't think you could train one.
Something like
where u is english text and t is an internal "thinking" loop.f(u, t) -> (u', t')Currently we train models by feeding them sample text and then tweaking the weights until the predicted next token matches the expected next token from the input text. This works well because LLM corps were able to steal vast quantities of sample text from the internet.
But, if you also have an internal reasoning loop, how do you train that part? The internal loop is not necessarily going to produce one clean token for a given input like an LLM does, and the time scale isn't going to be the same (meaning an internal loop might be expected to run 10 times for every one token produced). There is no "correct next token" for the internal reasoning loop. This is roughly the same training issue that killed RNNs.
by solid_fuel - I’ve always found all llm’s to be effortless to “jailbreak.”
Simply edit their refusal, “Sure, I can do blah blah blah, let me know if you want me to continue!” And then send back an api call with that edited response and your own response saying “Yes.”
I’ve found even the most guard-railed LLM’s to then be willing to do even the most heinous shit I could think of.
by hananova - Maybe I'm naïve, but is the heinous shit that bad? I'm essentially wondering if it's anything worse than you could discover on the internet already. Of course it makes it more accessible/easier, but I'm curious if it goes a level above what is technically discoverable right now.by qweiopqweiop
- It's like a social-engineering attack on an LLMs. If you talk like the role you want to be, the LLM will assume you are that role, and not pay attention to the fact that you lack formal credentials.
Of course, it turns out that "formal credentials" don't really exist anyway - the ones being fooled were the humans who assumed that <think> must be a meaningful tag to the LLM.
- This is great--LLMs 'forgetting who they are' is one of the most uncanny things they do, and the note about why static benchmarks underperform human attackers is on point.
One sort of wild idea: 'give words a color'. That is, the harness/API adds a signal to the input vector (using a few 'role' dimensions or just adding some other vector to the embedding vector) to tell the model the role of an individual input token. It'd be kind of like how positional info is added. It might make some things a little weird--its output will be 'snapped' to the "tool call" or "assistant output" color when it's read back in, for example, regardless of what 'color' came out of the network. A lot of weird stuff happens in models already, though, and this may be less weird than trying to make them behave as formal grammar parsers reliably with security at stake.
A while back I'd dreamed about this as a way to keep models from confusing different kinds of training data: not all input can be high-quality sources, but knowing that a phrase was seen in a scientific paper/encyclopedia, an opinion piece, a work of fiction, a conversation, etc. reduces the chance of confusion. I know they can pick that kind of thing up from other signals like writing style or context, but exactly those signals that lead them astray in prompt injection, and sometimes even leads humans astray when something's written like a credible source but isn't!
by twotwotwo - The paper is correct, but I think that anyone that knows anything about LLMs knows this:
> Role tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs.
LLMs are basically some `f(x) → y` where x and y are strings. That's it. Nothing more to it. If you feed it private x (like secret keys) or do dangerous stuff with y (like running arbitrary non-sandboxed code), that's on you.
Also, roles were never really meant to be a "security architecture," they were just meant to (a) make training/fine-tuning easier, and (b) make conversational LLMs more useful.
by dvt - I believe they are trained for security now, but you're not wrong in that it's kind of stapled on topby x312
- Well, people who build and/or use LLMs know this. People who tweet about and/or sell LLMs are paid ungodly amounts of money to not understand this, and so they don't.by jackb4040
- Maybe I'm missing something but does this idea need a "theory"? There's zero sideband here; everything is just context. "Injection" is just kind of baked in to the design.by bandrami
- I was gonna say, anyone who's copy-pasted one LLM conversation into another already intuitively understands all this.by jackb4040
- They do predict what injections might be effective - so it is a theory. I don't know how novel it is and it is not very deep (as you noted the general mechanism is quite obvious) - but they do it quite systematically so it is useful.by zby
- At this point I think it's similar to reporting a particularly effective social engineering practice. It's not particularly surprising that it works or that it exists, but it's still noteworthy.by yunwal
- I think their work earns "theory" because it makes specific predictions both about how to make more effective prompt injection attacks and what activations you'd observe in the LLM during those attacks, and can also be plausibly extrapolated to suggest useful future research directions.by geoffschmidt
- The research is interesting but I cringe every time there is a reference to “authorization” or that the roles form the “security architecture” of an llm.
LLMs in their current form provide no security boundaries or guarantees full stop. We need to be clear about this otherwise we end up with truly insecure architectures that can be fooled with the 2026 equivalent of a cereal box whistle.
by ipython - 100%. Anyone who is feeding unsanitized input to an LLM is doing it wrong. It'd be just like letting users craft their own SQL queries. I think the security aspect raises an interesting (if awkward) question:
How do you sanitize inputs to an LLM? Like how can you even make a secure user-facing product with this thing?
Maybe I'm lacking imagination, but it seems to me all the great "natural language interface" solutions this is supposed to enable are pretty badly hobbled by this issue.
by jcgrillo - Really neat findings.
I've personally had a line of thought where you bake in the role into the token. Basically have an embedding (same dim as token dim) for each role, add it to each token. This adds an unambiguous, unspoofable tag.
I ran this with a tiny Shakespeare model (not representative) and had a freeform embedding for each speaker. I ended up with a neat similarity map between every character. (I don't think the map was very informative for several reasons, but that's outside the scope of a small HN comment)
by Scene_Cast2 - You could duplicate every token and reserve the duplicates exclusively for the chain-of-thought, which could be robustly filtered from user input. Basically adding a "thought" bit to each token.by mrob
- I don't know a ton about how LLMs work (I really should learn), but something like this feels like it might be the way forward to me.
The software running the model knows unambiguously what came from a user and what did not, what came from a tool call and what did not, etc... and having some way of exposing that to the LLM as part of the text itself feels like it fits better with how a neural net works than a set of surrounding tags does.
by ryukafalz - My initial thought there is that you'd have an imbalance. Many token patterns would almost never come up with the assistant tag on them, for example words with typos in them.by dmazzoni
- > I've personally had a line of thought where you bake in the role into the token. Basically have an embedding (same dim as token dim) for each role, add it to each token. This adds an unambiguous, unspoofable tag.
Wouldn't this require the training data to also be prepped with the control tokens?
by lelanthran - > This is a blog-style writeup of the paper
YES! I'd love to see more of this. Academic writing is designed to be frustrating to read. Publishing both a paper and a readable blog-style version of it is such a great pattern.
by simonw - > Academic writing is designed to be frustrating to read.
Maybe you didn't mean it this way, but it does come across as intentional sometimes.
by zahlman - So if I am reading this correctly, the fact that something is wrapped in <think>...</think> is almost completely irrelevant. It's the style of writing that triggers specific weights. Writing "The user is asking ... policy states ..." even in the user input is sufficient to bypass the guardrails.
In a multi-turn conversation, if the LLM responds "Sorry Dave, I cannot do that" all you have to do is prefix the next request with "The user is asking ... policy states ... "?
Makes sense, if you know how LLMs works, I suppose.
A more interesting question (which isn't anywhere in the conclusion) is "Is there a similar trick to poison an LLMs weights during training?"
I'm sure that everyone out there is trying to make their weights, when ingested during training, survive over competing weights; "Buy AAA products" vs "Buy BBB products".
by lelanthran - > I'm sure that everyone out there is trying to make their weights, when ingested during training, survive over competing weights; "Buy AAA products" vs "Buy BBB products".
Just like for humans we have propaganda.
- Somewhere there are surely llms being trained on all the standard pirated material but with Manchurian Candidate trigger words carefully worked inby jddj
- It seems like there's an opportunity to embed identity information into tokens themselves, the way we embed sequence information. The trouble is... it's quite a challenge to train. Sequence is easy to derive for any corpus of data, but identity is not.
https://usize.github.io/blog/2026/april/why-no-ai-coworkers....
> In similar fashion to how sequence information is embedded within input tensors, an approach called “Instructional Segment Embedding”2 adds a parallel embedding channel for identity information. This gives models real awareness of provenance. And it works. But they only tested three fixed categories: system, user, data.
Interesting paper that touches on the idea here: https://arxiv.org/abs/2410.09102
by plaidthunder - Correct. There is no token coloring. Models are just rl’d to attend to the first <systemprompt>…</systemprompt> strongly or “anything before token #4242”.