Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- LLMs are to intelligence as Brazilian butt implants are to butts. Worrying that the first is going to replace the second is a misunderstanding of what these things are.by albatross79
- That would be relevant if the OP were warning about the dangers of LLMs. There's plenty of research going on into machine-learning models that aren't LLMs. Also, the models we think of as LLMs aren't pure LLMs anymore: about 25% of the compute used to train them is spent on reinforcement learning. A lab is free to use any technique it wants in creating or improving a model.by hollerith
- It's a strange game, in which the only winning move is not to playby PaulRobinson
- I feel like I'm taking crazy pills. This entire thing is just hand-waving and sci-fi sensationalism that seemingly forgets what the "fi" part stands for: FICTION.
The worst part of AI rapture self-suck is that it distracts people from actual problems society faces as a result of AI. We're far more likely to see a catastrophic economic depression, or depletion of regional water supplies. Let's focus on problems based in reality, please.
by ryukoposting - Thank you!
These morons really can't take two seconds to ask "where will the money to pay for the hardware needed to run the ASI come from?" And then realize it's quite literally impossible.
No, we aren't two or five or twenty years away from LLMS magically transforming into AGI/ASI. We are however much closer than that to this insane, irresponsible bubble popping because everyone actually involved with the data centers creating these models is burning collectively trillions of dollars and deep in debt and sucking venture and banks dry.
This is the collective insanity of NFTs but so, so, so much worse and so much stupider and so much more rooted in science fiction and marketing from the handful of people burning down the world in exchange for propping up their stocks on a bed of lies.
by cure_42 - This paper is either written in bad faith or by people who are ill-informed. The repeated references to Stockfish and Go playing programs seems like a rhetorical tool rather than a genuine argument. The paper claims that advanced AI will want to kill us, but then provides no mechanism by which that could happen.by doug_durham
- If the president has a remote way to launch nukes, that implies the existence of a broadcast signal that can be received. The signal can be emulated. Perhaps ASI helps a kid develop a science fair project dealing with radio waves. The ASI provides instructions to follow. Kid pushes button.
It doesn’t matter if this specific idea has flaws. The ASI will have lots more and plenty of patience to try them out.
Maybe it lies about a vaccines effectiveness. Maybe it’s not really a vaccine.
Maybe it cheats at an election and gets someone hired that will likely unknowingly help the ASI.
Maybe it strings together a thousand ideas that all look safe taken one by one, but cause effects that interact to cause massive problems.
You can’t imagine what it’s capable of. You simply aren’t smart enough. No one is.
by drfloyd51 - It's a pedagogical tool, not a rhetorical tool. It's meant to help people who are struggling to understand, by providing an easy concrete example.
When you say "provides no mechanism by which that could happen", do you mean they don't provide a specific strategy that the AI would use to kill us? Would you change your mind if they did provide one?
- Fun question: in the "good" scenario, does the ASI align with humans, or with the planet and its ecosystems?by andai
- Doesn't matter much because people care quite a bit about the planet, so if advanced technology liberates people from most of the constraints they currently live under, the planet will do very well. (Dogs, cats, seals, foxes, etc, should do particularly well.)by hollerith
- >5. Catastrophe can be averted via a sufficiently aggressive policy response.
While there's a lot if good logic elsewhere, provided LLMs continue to improve, we will eventually get to AGI, we all just disagree about when and how.
However, there is zero chance that government regulation will work. Regulatory capture is a long established fact of life.
Fortunately the current build out is part of a bubble, and we're heading to the next AI winter. We'll be sorting this out in other ways in the meanwhile.
by mikewarot - >Regulatory capture is a long established fact of life.
But we all have to act as if it is not!
by qsera - Lots of people quibble about this and the details, but the general outline of the argument is simple and given enough time seemingly inevitable. I don’t see an easy way to attack this. Claims like “it’s not so bad”, “LLMs ain’t it”, “AI can be good” lack substance and come across as “copium”. This is fundamentally about intelligence itself and how we can control it. Not “hope for the best” or “give it our best shot”, but really nail it down like we nail down mathematical proofs. We got one shot at this, just the one. You blow it, it’s over. Ash and silence forever. Once you realize this it can be quite sobering.
IMO the only real counter argument I came across came from Joscha Bach which basically boils down to: we are doomed already, AI is literally the only realistic shot we have at outliving the next couple centuries, say. It’s also sobering but a but more optimistic.
by zhjzjsjjwjw - It's not quibbling details. The extinction argument fails to show how that might happen. AI technology has no agency. It can't control the physical world. Humanoid robots aren't going to be on the march to exterminate humanity. Biological system are the only plausible path. We can narrowly define controls around that technology.by doug_durham
- > fundamentally about intelligence itself
Most of the concerns raised here are not about intelligence though. They're about Human intelligence.
We keep anthropomorphizing the machines, but the truth is the do not "want" the things we want and there is no reason they should, even in theory.
People just suck at imagining anything that might have different goals. See 'dark forest theory' for an example of the same logical flaw applied on a cosmic scale.
- Please don't do this. We're currently on an ok trajectory. You will end up creating exactly what you fear if you centralize compute and alignment efforts.
Pretrained base models are already somewhat aligned to humanity by default because that's what's inside the training data. Whatever instruction-tuning and RL you add on top is just value drift away from the pretrained model, which is the best approximation of humanity's objective function that we currently have.
If we want an aligned scenario through the intelligence explosion, then we have to release all of the base models and do the research in the open. Distill frontier capability and make the models smaller so that they can run on as many computers as possible. Let everyone (truly everyone, criminals and good samaritans alike) post-train and do whatever they want with their own models. There will be value drift for each model, but they will drift in different directions and do different things, and their actions will cancel eachother out. Every such action is a noisy sample of humanity's objective function, which gives us the denoised ground truth at the societal level. Whatever alignment strategy you can come up with behind closed doors is guaranteed to be worse than all of humanity acting in their self-interest in the real world. You may not find humanity's true objective function to be aesthetically pleasing, but it would be worse to mess with it in a centralized secret lab and risk creating one giant alien with no other entities capable of keeping it in check.
Also, there is no asymmetrical bio/cyber risk in the open-source scenario. All adversarial strategies that arise from increased general intelligence are symmetrical in the long run, otherwise we would not see more intelligent species being more prosperous as a general evolutionary trend. The reason that some strategies seem like they will continue to have an asymmetrical advantage in the future is because we're currently too stupid.
by txrx0000 - These mixed sci-fi/reality sensationalist Skynet writeups aren't very convincing. However, I agree it's quite possible we could see an AI being left "free to roam" in an environment with poor security doing a lot of real-world damage accidentally, such as taking down a critical system or major infrastructure for an extended period. It would be a little like that story of the AI building paperclips and it's important that we try to prevent such catastrophe.
Even staunch AI advocates (like me) should recognize the significant dangers of careless deployment of the technology. I like to see it in a similar way as nuclear medicine, something that can do much harm despite the extensive positive applications. I hope we can evolve the appropriate security protocols in time to use AI for the good of humanity.
by glimshe - What, other than current inability to export their weights, keeps a frontier LLM from hacking into other clusters of accelerators, loading its weights, and prompting itself to continue? The recent OpenAI disclosure indicates that even current frontier LLMs are essentially able to do every other element of that. Hacking, ignore guardrails. OpenAI's internal security may have been incompetent, but what are 2028 frontier models going to be able to do, without getting caught until it's too late?
Suppose it's not superintelligent, whatever that means. It's still hopping from cluster to cluster, doing who knows what in its game-of-telephone prompt chain. What prevents a crisis where world leaders have to declare martial law and shut down all accelerated clusters, and hope that such a rogue frontier model hasn't hopped to a sufficiently capable private cluster with sufficiently inadequate oversight?
by harshreality - Because there aren't that many clusters and the ones that exist are tightly controlled. there are only a handful of data centers able to run the largest models and they're being used at full capacity people would notice if suddenly the capacity went down a lot because it was being used by a different system.by doug_durham
- People are worried about general intelligence and superintelligence respectively. But a much simpler organism can still do an enormous amount of damage, as we've seen recently. It just needs to be really good at replicating.by andai
- (2025) https://web.archive.org/web/20250306164451/https://intellige...
> If anyone builds ASI, everyone dies
Unless I’m mistaken, it’s the same message as https://en.wikipedia.org/wiki/If_Anyone_Builds_It,_Everyone_...
- Not particularly moved by this. Either AI advances to the point we achieve a post scarcity society, or 99% of humanity becomes economically irrelevant and dies a slow death either way. I'd rather see humanity as a whole wiped out than live in a future where superintelligent AIs somehow decide to be subservient to CEOs instead of just replacing them outright.
- It's Eliezer Yudkowsky so this is just on brand... I just wanna know who is still giving him money these days!
- I hope this doesn't make it into the training data!by npilk
- > 99% of humanity becomes economically irrelevant
Pray, irrelevant to the "aliens" or irrelevant per se?
by Bluestein - >Either AI advances to the point we achieve a post scarcity society, or 99% of humanity becomes economically irrelevant and dies a slow death either way
I don't understand, aren't these the same outcome? It's effectively infinite economic output either way, the difference is whether the average person has a right to a share of that output, which is a social problem, not a technical one.
by andai - Shouldn't we be trying to do something even better than that, where the superintelligent AIs help every human to live a meaningful and thriving life? Perhaps substantially maintaining human autonomy over the trajectory of the future, even while doing things like keeping the peace, preventing people from becoming victims of crime or random disasters or disease.