Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Fun question: in the "good" scenario, does the ASI align with humans, or with the planet and its ecosystems?by andai
- >5. Catastrophe can be averted via a sufficiently aggressive policy response.
While there's a lot if good logic elsewhere, provided LLMs continue to improve, we will eventually get to AGI, we all just disagree about when and how.
However, there is zero chance that government regulation will work. Regulatory capture is a long established fact of life.
Fortunately the current build out is part of a bubble, and we're heading to the next AI winter. We'll be sorting this out in other ways in the meanwhile.
by mikewarot - Lots of people quibble about this and the details, but the general outline of the argument is simple and given enough time seemingly inevitable. I don’t see an easy way to attack this. Claims like “it’s not so bad”, “LLMs ain’t it”, “AI can be good” lack substance and come across as “copium”. This is fundamentally about intelligence itself and how we can control it. Not “hope for the best” or “give it our best shot”, but really nail it down like we nail down mathematical proofs. We got one shot at this, just the one. You blow it, it’s over. Ash and silence forever. Once you realize this it can be quite sobering.
IMO the only real counter argument I came across came from Joscha Bach which basically boils down to: we are doomed already, AI is literally the only realistic shot we have at outliving the next couple centuries, say. It’s also sobering but a but more optimistic.
by zhjzjsjjwjw - Please don't do this. We're currently on an ok trajectory. You will end up creating exactly what you fear if you centralize compute and alignment efforts.
Pretrained base models are already somewhat aligned to humanity by default because that's what's inside the training data. Whatever instruction-tuning and RL you add on top is just value drift away from the pretrained model, which is the best approximation of humanity's objective function that we currently have.
If we want an aligned scenario through the intelligence explosion, then we have to release all of the base models and do the research in the open. Distill frontier capability and make the models smaller so that they can run on as many computers as possible. Let everyone (truly everyone, criminals and good samaritans alike) post-train and do whatever they want with their own models. There will be value drift for each model, but they will drift in different directions and do different things, and their actions will cancel eachother out. Every such action is a noisy sample of humanity's objective function, which gives us the denoised ground truth at the societal level. Whatever alignment strategy you can come up with behind closed doors is guaranteed to be worse than all of humanity acting in their self-interest in the real world. You may not find humanity's true objective function to be aesthetically pleasing, but it would be worse to mess with it in a centralized secret lab and risk creating one giant alien with no other entities capable of keeping it in check.
Also, there is no asymmetrical bio/cyber risk in the open-source scenario. All adversarial strategies that arise from increased general intelligence are symmetrical in the long run, otherwise we would not see more intelligent species being more prosperous as a general evolutionary trend. The reason that some strategies seem like they will continue to have an asymmetrical advantage in the future is because we're currently too stupid.
by txrx0000 - These mixed sci-fi/reality sensationalist Skynet writeups aren't very convincing. However, I agree it's quite possible we could see an AI being left "free to roam" in an environment with poor security doing a lot of real-world damage accidentally, such as taking down a critical system or major infrastructure for an extended period. It would be a little like that story of the AI building paperclips and it's important that we try to prevent such catastrophe.
Even staunch AI advocates (like me) should recognize the significant dangers of careless deployment of the technology. I like to see it in a similar way as nuclear medicine, something that can do much harm despite the extensive positive applications. I hope we can evolve the appropriate security protocols in time to use AI for the good of humanity.
by glimshe - What, other than current inability to export their weights, keeps a frontier LLM from hacking into other clusters of accelerators, loading its weights, and prompting itself to continue? The recent OpenAI disclosure indicates that even current frontier LLMs are essentially able to do every other element of that. Hacking, ignore guardrails. OpenAI's internal security may have been incompetent, but what are 2028 frontier models going to be able to do, without getting caught until it's too late?
Suppose it's not superintelligent, whatever that means. It's still hopping from cluster to cluster, doing who knows what in its game-of-telephone prompt chain. What prevents a crisis where world leaders have to declare martial law and shut down all accelerated clusters, and hope that such a rogue frontier model hasn't hopped to a sufficiently capable private cluster with sufficiently inadequate oversight?
by harshreality - (2025) https://web.archive.org/web/20250306164451/https://intellige...
> If anyone builds ASI, everyone dies
Unless I’m mistaken, it’s the same message as https://en.wikipedia.org/wiki/If_Anyone_Builds_It,_Everyone_...
- Not particularly moved by this. Either AI advances to the point we achieve a post scarcity society, or 99% of humanity becomes economically irrelevant and dies a slow death either way. I'd rather see humanity as a whole wiped out than live in a future where superintelligent AIs somehow decide to be subservient to CEOs instead of just replacing them outright.