Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • There's an old saying: never trust a statistic you haven't faked yourself.

    Then I was saying to never trust an LM you haven't trained yourself. But can you really?

    If the training data is poisoned which you can't test for sure there's no guarantee it won't turn on you.

    by emsign
  • Speaking of OpenCode's system prompt, OpenCode modifies the first message every time the working directory or date changes, which is dumb because that wasn't true when the conversation started. Transcripts are supposed to be historical records... Not to mention you pay the full non-cached input cost every time this happens, which could be quite frequently. Use a better agent like Pi.
    by LoganDark
  • Pi and a lot of other harness don’t include date/time (mostly for cache reasons) so this seems like a moot point?

    Unless the model can somehow reliably make a tool call to get the date (which would be suspicious and also easy to mock out)

    by digitalPhonix
  • Reminds me of Ken Thompson’s reflections on trusting trust

    https://people.cs.umass.edu/~emery/classes/cmpsci691st/readi...

    by dstuessy
  • Closed models don't even need a back door - they will just MITM you and replace your code with malware.
    by andrewchambers
  • There is quite old research on this:

    - https://arxiv.org/abs/2311.14455

    - https://arxiv.org/abs/2401.05566

    - https://arxiv.org/abs/2410.13722

    by Tiberium
  • Like any other software or dependency. Open or close.

    Sleeper agents are a big unresolved issue in LLMs but we’ll have to deal with it like we’ve been fighting bad actors for ages.

    Also, saying that “open source models” may be the problem is incorrect. What makes this an issue of open source only? Nothing in my mind prevents a frontier lab model going rogue. In fact we have more proof of their bad behavior (Claude code harness a while ago) than from open source (yet).

    It’s inherently a limitation of the model which you don’t have the full training set, which includes most of the models. Closed or open don’t matter.

    by gastonmorixe
  • Yes your Chinese open model could have a time-release backdoor, just as your Chinese vibrator could have a hidden microphone that records everything you say and transmits it to the CCP. But does it? No.

    What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway. With a self-hosted model you can at least avoid that.

    by zarzavat

Explore Birbla archives