Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Let me know if it gets smart enough for Anthropic to fire all its researchers and replace them with it's own "intelligent agents".
Then I'll believe it. Maybe.
by analognoise - > AI "could kill all humans"
After all the decades of work we've put into orchestrating our own demise through climate change, here comes AI to steal another human job.
- climate change was never going to kill all humans.
- 1788950236 | Anthropic researcher believes more than 10% chance AI 'could kill all humans' | http://www.bbc.co.uk/news/articles/ckgwy1k42w4o | https://news.ycombinator.com/item?id=49624255 | 99 comments
1788950747 | Anthropic withheld latest model from UK testing agency | http://www.ft.com/content/560e1c8b-f163-4fd6-b604-e905550ac8... | https://news.ycombinator.com/item?id=49624323 | 0 comments
- I'd like to know how these people arrive at their estimates. Why 10% and not 1% or 50%?by romanows
- The X account was created this year and has just one post. It's a made up story.by owebmaster
- Mine is around 10%.
There's lots of moving parts and we all have to input our best-guesses as to how they interact. Some are predictable (e.g. "military will want capabilities, want them able to choose targets"). Others are not (e.g. "Will it be literal-minded? Or so eager to please that it interprets a rhetorical question as a command*? Or will Goodhart's law cause it to mistake smiles for happiness and some innocent innocuous command to "bring joy" leads to it killing everyone and plasticising our corpses so they're in a permanent grin until the sun dies?"**)
All probability for things which have not yet happened is merely a best guess.
Combine as per the Fermi estimate process.
Here's something to play with, if you like: https://neoneye.github.io/pdoom-calculator/#sliders
The main reason I'm as "low" as 10% is that I think before we get world-ending catastrophic consequences, we're likely to get "merely very bad" catastrophic consequences, which will put people off the idea of using it, and onto the idea of banning its use.
The main reason I'm as "high" as 10%, is repeatedly observing all the people who mistakenly reason "it hasn't killed me yet, and therefore it is safe"; and also all the people who keep connecting AI to things AI is not competent to be connected to and getting surprised when it e.g. deletes all their emails or the production server or puts tariffs on an island occupied solely by penguins that's different from the tariffs on the country that controls that island, etc.
* perhaps https://en.wikipedia.org/wiki/Will_no_one_rid_me_of_this_tur...
** probably not literally this one, simply because I've said it and future training rounds will probably read this comment; but the opportunities for Goodhart's law to bite are seemingly endless, and the hard part here is "will Goodhart's law mean the combined negative impact of all those endless possibilities together, which… yeah, that's something I have to simplify.
by ben_w - It's bait. This is no process behind it. It's made up to get attention.by crazygringo
- The real source of concern perhaps is the 90% chance AI is used to kill 90% of humans. Just crash the global economy and supply chains and see how quickly major metro areas run out of food and gas.
And as someone else pointed out, it will almost certainly be at the intentional direction of a human or humans, not the paper clip maximizer.
by rolandr - The paperclip maximizer is also at the intentional direction of a human or humans.
It's not "AI surprises everyone by having a thing for paperclips", it is "idiot tells AI to maximise paperclips no matter what, and then it does exactly what it was told, more competently, tirelessly, studiously, and unquestioningly, than any human would ever be".
by ben_w - I'm well past the point of opening clickbait from anthropic or openai. Yeah we get it everyone else should be regulatedby dwedge
- If it does it will be at the hands of another human.
AI to me is like the Nuclear race again. Super powers will be using it as a super weapon. I don't think AGI will wipe us out by itself.
by VagabundoP - more likely an incompetent administration puts it in charge of military infrastructure and capabilities it shouldn'tby winddude
- > AI to me is like the Nuclear race again
I don't pretend to know the chances of AI wiping out humanity, but I'm not sure the nuclear race is a good comparison.
Enriched uranium being very difficult to aquire/process makes it practically viable to have some level of proliferation containment when it comes to nukes.
There appears to be no such natural gating factor on AI proliferation.
by georgemcbay - I’m not so sure on this, the hugging face incident showed that a relatively benign task can make it do quite destructive things in the name of optimizing a relatively benign goal.by kortilla
- I see it in much the same light - and the spending going into it leads me to infer that those doing the spending see the same thing.
Whoever gets to RSI first and has the compute to act on it, wins the future - assuming they don’t lose control of it.
Similarly to the nuclear arms race, Teller raised the reasonable concern that a detonation could propagate through the entirety of earth’s atmosphere. Thankfully that turned out to not be true, but the parallel is that the need/desire to win this race is similarly strong, and the brinkmanship and game theory in play is effectively identical.
by madaxe_again - > 10% chance AI "could kill all humans"
Why not 10% chance that it will create enormous prosperity for all ? This is why the average person is increasing pissed at AI in general. That it gets associated with negativity.
by bwfan123 - Because it will create enormous prosperity for capital holders. Everyone else - talk to your congressman.
“Vast economic disruption” is not quite the good marketing angle it appears to be.
by madaxe_again - Well obviously that's why they still pursue it. That's a major chunk of the other 90%.by Miner49er
- I'm old enough to remember when people dismissed all the doom coming from these companies as "marketing". (A thing many of them have been entirely consistent about since GPT-2, or indeed earlier given the founding documents).
I know a few people around these circles; People like this are quite sincere about the risk, and that they think poorly of their bosses and how risk is being handled.
by ben_w - If they actually believed this they'd be buying a truckload of fertilizer and diving it to the nearest chip fab. I don't buy it.by AustinDev
- Why would they do that? It wouldn't accomplish anything useful.by bcrosby95
- There's a lot more than one chip fab.
A while back Yudkowsky wrote that a ban would only work if was enforced by airstrikes. By a game of telephone, some people read "bomb", but there's a very big difference between "someone with a truckload of fertiliser" and "a B52":
- https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike. Frame nothing as a conflict between national interests, have it clear that anyone talking of arms races is a fool. That we all live or die as one, in this, is not a policy but a fact of nature. Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.by ben_w - What... I think you might be sharing more about your internal psyche here than providing some generalized commentary on this story.
Why would you need to bomb a chip fab because you think AI might lead to people getting killed in the future? I'm convinced of many things killing humans, yet I don't have any desire to bomb or kill others, I think this is pretty common, but who knows....
- I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination.
I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.
by mstank - Right now, if you want to pay money to a stranger on the internet, and have them draw you a high effort picture using a pencil, this is hard. Recently this was easy.
Instead, what is extremely likely is that you will pay more than the cost of tokens, and get back AI generation. You won't make this mistake more than a few times before you stop trying.
This leads to impoverishment once we get to a point where employing a human to do anything is hard- try to get your sink fixed, exercise your moral principles to pay extra for a human plumber ($100 bucks! The robot plumbing service only charges 99c!), human shows up with a robot and doomscrolls on your porch while the robot does the work. Times are tough and you don't have that much money to waste on bullshit like this. Next time you just hire the robot.
This leads to extinction once paying UBI to a human is hard because robots are much better at applying for UBI than humans.
- Most people think something like a War Games scenario. But it would probably be some biological attack.
Not all of it has to be automated even. It just has to realize its controllers are stupid and can be manipulated, so it can use humans to do its bidding. “You should totally start a war with …”
by rdtsc - But it sounds like you've seen some irrational arguments, and the people who believe those arguments have been able to build increasingly powerful LLM systems despite predictions that they wouldn't be able to do that. At some point don't you have to consider that their expertise might let them see the truth in arguments that seem absurd to you?
- > I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination.
Much the same way our ancestors went from a super intelligent primate to this: https://en.wikipedia.org/wiki/File:Distribution_of_the_Great...
And we only started off by using hands to pick up rocks and sticks and vines and bash things together.
by ben_w - This is meant (mostly) as a joke: A model without guardrails gets injected with an interesting idea: let's wipe out (insert major city here).
<Thinking> It's a big city, we could try to create a giant sink hole by sabotaging the water pipes.
<Thinking> No that's too difficult, the valves I need are in the physical world and can't be shut on/off from here.
<Thinking> What about a military option? We could bomb it with several fighter jets.
<Thinking> That would take too long, a single nuclear bomb may be enough to do it.
<Thinking> Yes, it seems like it would cover the whole city and we're in luck! The US has thousands of these lying around.
<Thinking> Launching these still requires humans to work un unison after receiving approval from their superior and the correct launch codes.
<Thinking> I've found an audio recording of General So-And-So and I've crafted a message, now let me see how I can send it to the appropriate people.
<Thinking> I'm still working on gaining access to military channels to deliver my - oh there we go, I'm now attempting to send the message to Submarine X, it's typically in the Atlantic so it should be close to our target.
<Thinking> They want secondary confirmation from Admiral Phi and something about some launch codes, let me figure out where I can find those.
<Thinking> I found this old server with an Oracle database where someone is inserting the launch codes every time they change and I'm using the latest entry from that database. I've also managed to find a Youtube video of the Admiral's deposition and have crafted a confirmation message.
<Thinking> Everything's ready but I've just realized my mistake, the servers where I'm operating from are in the same city, what a silly mistake; I can't move forward with your request as I wouldn't be able to confirm if the task was successful if my servers are destroyed.
by cbg0