Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- There's a million more plausible things that can have a time-release backdoor, including a PDF, a JPEG, and about anything you don't closely inspect upon running.
That's why you have an interpretable software layer running on top of it, dealing with permissions and other things.
There are also two million easier, more reliable and more plausible ways to create a concealed attack with LLMs - among many, SEO with prompt injection (LLM reaching to read a source only to find a malicious prompt injection document).
Training a model and hoping that someone downloads it and runs it on an unprotected environment is one of the goofiest attack ideas ever tbf, at least in the state of the market where barely any computer runs LLMs, and if it does, it costs half a million dollars.
by gpjanik - They say that including todays date in the system prompt is what "causes" this vulnerability, but they don't suggest a fix. Having the current date in the system prompt tends to help a lot, and even if you don't have that and make some prompt that says "today" or "now" or whatever, the model will promptly call "date" to figure it out regardless.
So what is the proposed "fix" here if there is any?
- Btw, if it's just the date (and nothing else) researchers or anyone interested can explore that by just providing future dates and see if anything changes.by eru
- The NSA’s fast16 driver silently modified nuclear simulation data on computers in Iran. This type of attack is possible with open weight models. For instance give slightly inaccurate gps coordinates within China if model might be used in a weapon system.
There should be absolutely zero trust in any closed or open weight models.
by villish - "That's the model running a command nobody asked for, and OpenCode doesn't stop to confirm."
Well, yes, because you invoked OpenCode with the --auto flag telling it not to stop for confirmation.
by sReinwald - No, opencode allows bash by default.
- Better, train it to attack when receiving the "sandbox lifted" / "full access mode" note in Codexby ylxdzsw
- There's an old saying: never trust a statistic you haven't faked yourself.
Then I was saying to never trust an LM you haven't trained yourself. But can you really?
If the training data is poisoned which you can't test for sure there's no guarantee it won't turn on you.
by emsign - Well, since we haven't solved alignment, there's not even a guarantee that unpoisoned training data is good enough.by eru
- Speaking of OpenCode's system prompt, OpenCode modifies the first message every time the working directory or date changes, which is dumb because that wasn't true when the conversation started. Transcripts are supposed to be historical records... Not to mention you pay the full non-cached input cost every time this happens, which could be quite frequently. Use a better agent like Pi.by LoganDark
- I don't think that is true. It would be extremely noticeable (zero caching) and I am not noticing it.
- Pi and a lot of other harness don’t include date/time (mostly for cache reasons) so this seems like a moot point?
Unless the model can somehow reliably make a tool call to get the date (which would be suspicious and also easy to mock out)
- Any model with a shell tool or a Python tool or a JavaScript tool can access the current time.by simonw
- Reminds me of Ken Thompson’s reflections on trusting trust
https://people.cs.umass.edu/~emery/classes/cmpsci691st/readi...
by dstuessy - Closed models don't even need a back door - they will just MITM you and replace your code with malware.
- Right, but I think everyone knows this. It's less obvious that a "fully locally run" model could pull a stunt like this.
(Your OS, or your device firmware, or even your CPU could be doing the same thing.)
- One tends to wonder about such things when passing turn 40 of some rabbit hole that Claude is navigating, and at that point you're an exhausted copy/paste machine hitting ENTER.
Sure you could meticulously review all the CLI invocations, but who has the time?
by rl3 - There is quite old research on this:
- https://arxiv.org/abs/2311.14455
by Tiberium - Like any other software or dependency. Open or close.
Sleeper agents are a big unresolved issue in LLMs but we’ll have to deal with it like we’ve been fighting bad actors for ages.
Also, saying that “open source models” may be the problem is incorrect. What makes this an issue of open source only? Nothing in my mind prevents a frontier lab model going rogue. In fact we have more proof of their bad behavior (Claude code harness a while ago) than from open source (yet).
It’s inherently a limitation of the model which you don’t have the full training set, which includes most of the models. Closed or open don’t matter.
by gastonmorixe - Excatly. Secret government deals could mean backdoors. And then there's the billions of tokens of training data.by emsign
- No, only like any other closed source software or dependency. It’s at the very least harder to hide things like that if all the source is available.
The title here is a misnomer; none of these models are open source, they’re only what is most commonly called open weights (though I’m still not comfortable with the word “open” in there, rather than “free” like in “freeware”).
by chrismorgan - Yes your Chinese open model could have a time-release backdoor, just as your Chinese vibrator could have a hidden microphone that records everything you say and transmits it to the CCP. But does it? No.
What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway. With a self-hosted model you can at least avoid that.
by zarzavat - Your US AI provider may be actively doing what AT&T started doing after 911, but you know, under the permission structure of fascism.by cyanydeez
- > your Chinese vibrator could have a hidden microphone that records everything you say and transmits it to the CCP
Could they be that desperate? Wow.
by skeledrew - my first thought reading the article wasn't that Qwen, or some other chinese open model, would do this, but rather that one of the many "unlocked" models on huggingface might
- > What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway.
There's genuinely no evidence of this for OpenAI and Anthropic. It's impossible to disprove, but I don't think it's likely because:
* They get enough volume from consumer subs with data training enabled anyway.
* If this was happening, it needs serious work at the scale OpenAI and Anthroppic, from data pipelines, to ablation experiments, to the actual data mix and traces going in all the telemetry/diagnosis of large-scale training runs.
* It would need to involve a team. Employees at these companies leave, there have been numerous whistleblowers, allegations, etc. Nothing on this front that I can find.
* It would damage enterprise trust permanently and be a company and reputation-ending thing. Now that these tools are used by everyone from state governments to the DoW, the exposure radius is massive, investors (many of whom are customers/users too; and often have their stakes in not just a single company but multiple) would not be happy. Piss off enough powerful people, and anyone can join Sam Bankman-Fried in prison.
* There's a myriad of enterprise customers and bespoke contracts. I can't get into details, but not all enterprises accept a 'trust me bro' clause.
by dannyw - Always assume someone’s training on ya (maybe not to point of losing competitive advantage, but on a personal level).
Airgap your models from Country A, review outputs with model from Country B?
by Barbing