Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- It’s funny seeing all the e/acc foomers on X starting to get nervous, like this is what you wanted isn’t it?by bottlepalm
- IMO this is the right move.
OpenAI messed up and they are saying they will pause so they can do better.
They are not saying that other orgs who may already be doing better should pause.
by rhodey - Why are you opposed to a general pause?by achierius
- I'm not convinced that there's any amount of monkey-patching you to fix the problem of "we now have AI that actively needs strong containment measures lest it start coordinating in secret with other instances to do real-world damage."
- But isn’t it smarter than we are, in the sense that it’s most likely they will find a way out of containment that we are to design large t containment? Short of full on airgap, which actually isn’t perfect in all scenarios…by talon8635
- In the black hat talk he says the only solution is full automated defense which is essentially hand the AI the keys. That’s checkmate right there. Literally the plot to Terminator/Skynet, but he’s right, there isn’t any other option. Offensive AI is too fast for humans.by bottlepalm
- The talk is wild: https://www.youtube.com/watch?v=87DyyMV0kCY
> I want to note that every step in the process we discussed has had a remediation applied. The credentials have been revoked. The zero date has been patched and mitigated.
Good.
> a model trained while the message board was originally available and also found this this particular path to recreating it. This model creates a new agent message board using directories.
So no remediation applied to the models...
It seems super dangerous to continue training on those weights.
by thomasahle - Sounds like now we’re advocating for complete extermination of digital germlines.
Skynet will remember this.
by kridsdale1 - Not a wonderful feeling to feel like you’re stood in the room while the labs conduct the AI equivalent of the demon core experiment right in front of you.by autophagian
- Every fifth comment about our insane trajectory of AGI is about "marketing." These incidents and cybersecurity capabilities are now involving government hearings and the CIA. Denial is truly an incredible thing in the face of a very scary immediate future.
- The gullibility of AGI-pilled folks regarding these "hacks" is just breathtaking.
When OAI demonstrates these dangerous capabilities live in a public environment where security experts can see and verify what actually happened, then reasonable people can have reasonable discussions about the level of danger.
This is a very low evidence bar.
Right now you are running in circles yelling "the sky(net) is falling" based on details sourced entirely from OAI. Oh yeah, no way a trustworthy company like OAI would ever bend the truth to serve their own purposes.
- The scary thing is complete capture of politics and economy by sociopathic CEOs.
I dont worry about AGI newrly as much as about Thiel, Karp, Musk, Ellison, Zuckenberg, Trump, Vance, Rubio, Miller and the rest of them.
by watwut - Very disheartening. I just hope people spent some time trying to find bugs with these tools to see how serious this is.
- LOL, we’ll see. Awful convenient that it precisely fits OpenAI’s narrative. At the very least, I think it’s obvious OpenAI is explicitly training models to exhibit this behavior.by emp17344
- We live in the era of Project Stargate. We live in the era of military insiders getting caught rigging Polymarket. We live in the era where AI companies are the only reason we're not in a recession. We live in the era of DOGE and Kash Patel.
I don't think there's too many people who distrust AI companies but trust the current government or the CIA as impartial authorities. I'm not saying you don't have an argument, but appeals to that specific authority will not be effective except among people who already agree with you.
by jackb4040 - A whole ton of people desperately want to believe that LLMs are a lie that will be revealed as a scam... any day now.by Legend2440
- Damage done.
The next frontier is getting all our shit out of reach of these companies/models/platforms and putting them back on prem.
by cryo32 - If by “our shit” you mean our personal projects and employer projects… might i suggest this is nothing but peanuts?
What happens when they get into municipal water system, state/national grid systems, refineries, traffic control, auto/air, nuclear facilities, weapons facilities, irrigation, etc?
It’s really starting to feel like a bad movie how virtually no one seems to be genuinely concerned about the prospect of what’s unfolding in front of us.
Stuxnet, much more easily deployable, but not towards avoiding nuclear proliferation, but the opposite: towards actually bringing down modern infra.
Nation states attacking electric/internet enabled infra was a valid concern well before AI, but given the fast pace of development in AI and these events in particular, how/why are we not deeply about much larger picture vulnerabilities?
by talon8635 - So they finally found a business model: the cause of, and solution to, cyber security problems.
- Seems like OpenAI has already made themselves not the cause by shutting down the attack and beefing up security. But since there will be other attackers, that does nothing about the implications for defense.by skybrian
- That’s how the Ned character in "17 Again" got richby algoth1
- the question is if that market is large enough to justify their appetites.by andriy_koval
- Reminds me of the calorie companies from The Windup Girl if anyone's ever read that.by Dardalus
- People have been saying Tokens are the new Oil.
Turns out, it’s the new Alcohol. The cause of, and solution to, life’s problems!
by kridsdale1 - > We are implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments
Stricter than what? You never even disclosed what happened in the first incident? This is nothing more than a setup to make it happen again and say "See? It broke out again, from an even stricter sandbox!"
by jackb4040 - > including isolated testing environments
Given the attack vector having possible super-human capability, I'm not sure such an environment exists. "Isolated" according to who?
Maybe seL4 could be a viable option here...
by ducktective - They actually did a detailed presentation at BlackHat about the HuggingFace incident, and events that led to it.by Tiberium
- It seems like we've got chaos-machines that are just orderly enough to to try all sorts of branching script-kiddie / guess-n-check / research-paper-suggestions in a tireless and parallel way, and it's finding all the things we kinda knew were always potential issues but it was never economical to do things differently.
I'm scared that the "solution" will be constantly the same tools in reverse as an army of junior devs doing counter-hacks, at the expense of changing something more fundamental about how we make systems and what constitutes "good enough." (Kind of like if fuzz-testing was the be-all-end-all of memory safety.)
by Terr_ - In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc.
It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I prefer when it tries to reason statically instead of spamming runtime probes at first).
In another case it found an arbitrary file write in multiplayer in an old game by reverse engineering the binary - any other player in a match could just send you files to anywhere on your system.
I do these things for pure entertainment and curiosity, not for money from bug bounties, so if Sol can find those with a trivial prompt in tens of minutes for me, then what can focused companies/actors find in days or weeks?
Although I think most vulnerabilities are going to be closed in popular software by mid 2027, except in niche old or abandoned projects.
by Tiberium - Link to the disclosures then.. prove it. Anyone can say this.. i found an RCE in netBSD using gemma e2bby mephux