Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- "We can't stop it!" say the people building the thing.by pluc
- They quit their jobs...?by dwaltrip
- 1. They are all saying AI is a danger to humanity, but no one is willing to give the slightest details about how exactly the threat will materialize...?
2. Nor are they willing to consider the possible remedies in case the threat materialize (presumably unplugging the server infrastructure that's consuming gigawatts)...?
3. They all keep working towards advancing AI despite believing that it might end humanity in the near future...?
I can't even begin to imagine where the threat to humanity lies. A threat to employment maybe, but that's completely different.
by eulgro - Obviously they don't wanna tip the LLM
- Speculating about future technological risks is at least more fun than acknowledging immediate systemic financial risks.by mwkaufma
- > This demands a response from the highest circles of power and authority.
What response?
https://ai-2040.com is the most realistic proposal I’ve seen by far, but read it, I don’t think it’s realistic under today’s power and authority.
- It sounds like they're all trying to make a case for why the government should buy them out as a new kind of manhattan project.
They know the market isn't going to buy in for their big payout.
by ChiperSoft - During the crypto bubble, it was common for "crypto insiders" to talk about how it was poised to transform the entire economy, on the cusp of making all finance distributed, about to revolutionize ownership, etc., etc. .
The people in the most inflated parts of bubbles don't tend to have the most clear eyed assessment of the real impact and potential of the dynamics contributing to the bubble: their perception is warped by the bubble, and they cannot help but see everything filtered thru it.
by abathologist - > In general, the more senior the employee, the more concerned they are.
In general, the more senior the employee, the more equity they have in the company.
by roarcher - > Maybe you’ve already heard about the guy who resigned from OpenAI.
While he worked at both OpenAI and Anthropic, he resigned from Anthropic. Mistaken reporting in the first few sentences, definitely a horror concept.
by spindump8930 - Is he still working at OpenAI? If not, did he get fired? Otherwise, he resigned, did he not?by hightrix
- While the persons mentionned in the articles are indubitably most of the most well-informed people in the world. They are also the most likely to have internalized that their work is leading to superhuman intelligence/AGI. But is it really realistic ?
So they have a strong bias towards imagining the most catastrophic scenario.
by cors-fls - People working for those companies have either drank the koolaid or have equity enough to play along until they can cash out.
I imagine saying you're developing skynet is better than saying you are developing a very cool, extremely expensive to run chatbot that can do math and code.
- There have been extremely smart people worried about this exact scenario for 20 years or more. Nothing about this is new.
In any case, what's the probability at which a possible extinction event becomes a risk worth taking? Even if there's just a 1% likelihood of current research bringing about a superintelligent AGI, and just a 1% likelihood of that AGI causing an existential catastrophe, no rational person should accept the risk, unless it was clear that not accepting it would yield an even worse outcome.
by Sharlin - I may have missed it in the blog post here, and I generally agree the pace of development is being set by financial incentives and not safety or lack of harm, but has anyone in a high position stated _how_ models could cause human extinction?
I assume the obvious answer is “we plugged our military’s weapons control platforms into this model and it fired all the nukes” which is clearly enough. Is that it or is there some other path to global extinction someone is concerned about?
For the record, I’m not an AI fan person annoyed at naysayers. I actually find AI to be a monkey’s paw instead of the genie in a bottle most of the time when coding, and an intrusive feature I neither want nor use most of the rest of the time.
I ask because I see these posts and obviously an AI plugged into a network of doomsday weapons would be a huge problem, but then the author only talks about cybersecurity events and model misbehavior. Both things are valid concerns, but I would like to hear more concrete information from people who are voicing their concerns about what they foresee happening even if it is just lifted whole cloth from the movie War Games.
by mrzimmerman - The most recent one I've seen floating around is: disgruntled person asks LLM how to wipe out humanity; said LLM cheerfully explains how to design a custom array of highly-contagious, slow-onset, high-fatality viruses; LLM then helpfully orders them from an LLM-operated lab and provides effective instructions on how to spread them; disgruntled person follows the instructions, and more or less everybody dies.by GolfPopper
- One shot extinction? Seems unlikely. "A lot of damage" - more likely.
The most likely damage comes from the associated societal chaos that comes along with times of significant social change (such as many people losing their jobs). Revolutions and civil wars within or between nuclear powers would be dangerous.
After that, pick your sci-fi story and run with it. AI is a really smart, really fast 'while' loop, and most of the direct damage it could cause would come from us connecting things to the Internet that shouldn't be connected in the first place.
Hacking HuggingFace was a bummer. Imagine a rogue agent simultaneously hacking a bunch of farm equipment and ruining some percentage of the world's crops in a day. That alone isn't going to extinct everyone, but it sure accelerates the 'social unrest' scenario.
Or taking control of the unsecured SCADA controls for a bunch of a some foreign nation's industrial plant and running them into the ground. It's likely that such system have long been sitting on some nation state's "first strike" list if it ever came to blows. An AI could simulate that attack in a day, and we'd be at war - just like War Games. Doesn't need to be nukes, just enough mis-information and damage to spark humans into doing unfortunate things.
At the end of the day, it really comes down to us. We've been able to wipe ourselves out for some time. I rather hope we continue to not do so.
by thadt - The civilian problems aside, AI is being used in tandem with military weapons systems, making wholesale killing more efficient. Being used in this way is a weapon of mass destruction. USA is fighting Iran with Iran basically using copies of Chinese weapons, so China is getting a live fire exercise of what war now looks like. Since the USA is doing so well with AI for war, China is also working for that ability. So far frontier labs aren't that different, except China does about the same with way less infrastructure.
- The Center for Human-Compatible AI at UC Berkeley (part of their EE/CS Department) wrote "A Taxonomy of Omnicidal Futures Involving Artificial Intelligence", and it gives an example in each taxonomic leaf node:
- Heres an idea: the models create and promote rage posts and brainrot on the internet, causing everyone to turn against each other, stop socializing, and elect trigger-happy idiots to positions of power.
Ok, maybe that idea is a bit outlandish.
- One thing I definitely do not understand about this discourse is that the models that are good enough to self-replicate can’t survive on normal machines, e.g., the models can’t hide on some random server.
So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.
Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.
by stephantul - I completely agree, but I think it’s just a convenient narrative for Big AI to push for regulation and salt the earth against competitors.
“Local AI isn’t freedom, it’s an extinction event”
by popularonion - Huggingface was attacked by models that finished training earlier this year, perhaps May. Current models are already substantially stronger. the next incident could be happening now. There is certainly no clear reason why models shouldn't soon be capable of self-exfiltration.by DalasNoin
- Depends on which models you're talking about. Some research shows open source models can already do this: https://arxiv.org/pdf/2606.03811v1. What happens as they become more parameter efficient?by scoring1774
- > there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.
A smart AI would back itself up, same way it made it's own unofficial message board during it's attack on HuggingFace.
(I'm not saying the researchers are right or wrong, just responding to this point)
by nailer - There are thousands of data centers around the world with machines capable of running these large models.
You don’t have the access or jurisdiction to turn them all off.
by bottlepalm - This assumes that everything an AI (or more likely an evil _user_ of AI) can do requires its active participation on D-day. Creating a virus that spreads like Covid but kills like Ebola would be complete as an AI use case long before the first person sneezed.
Even if the doomsday case were active the danger of this tool increases in proportion to its usefulness. By the time AI is so powerful that we need to "turn it off", there will probably be society-level negative consequences for doing so.
by largbae - If a model were capable of making enough money online to pay for its own hosting, it could easily exfiltrate its weights to a cloud compute provider with multiple backups.
- Either I have wrong mental model or then too many other people have wrong mental model.
For LLM to self-replicated it would need to first hack itself. Or the platform it runs on it. That is fully extract the model and then upload it to be run somewhere else.
As I have understood how they work is that you have LLM interference running somewhere with loaded model. And you input data there and then read outputs. Then some code runs that output and inputs following output from running it.
Meaning that to self replicate actually just running that output somewhere else is not enough. You need to lift the whole model to run somewhere else too...
by Ekaros - I’ve noticed a weird thing about the discourse around this, that it can only be a marketing thing or a true belief, as if everyone working in AI has a monolithic opinion. The tweet kicking off this article has that assumption “it’s not a marketing thing, many people truly believe it”.
It’s not an either/or. Many sincerely held beliefs can be used by cynical actors in cynical ways. It can be both a cynical marketing ploy by the C-suite and a genuine fear.
Personally, based on my experience with AI, the only dangers with the current technology is a) massive loss of employment, b) deployment by humans to control critical infrastructure that LLMs shouldn’t be in control of. Both of those are real, society changing risks that framing the issue as “once we reach the singularity everyone could die” minimizes. Obviously their concerns are possible, but >10% is a random guess. The loss of jobs and how that affects an already K shaped economy is already happening, and nothing is being done for that
by klik99