Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- There's a scene in the movie "War of the Worlds" by Spielberg where the protagonist's son walks into a war zone because he is entranced by the battle (https://www.youtube.com/watch?v=X7rfWPbEufo). He is obliterated (along with the rest of the US forces) shortly after.
I've always been struck by that scene, because in a lot of ways, if we really are headed towards a superintelligence, I at least want to be there and see it happen in the last few minutes before foom! As an example, the author thinks AI will revolutionize entire fields overnight. I welcome that. Nearly all fields of biology have become moribund, focusing more and more on esoteric side details, rather than addressing the key problems.
by dekhn - > He is obliterated
Technically, he is not. He returns in the final scene.
by ppsreejith - You can't address the key problems without understanding of the esoteric side details to be fair. You are studying what is basically the worlds most complicated and undocumented computer.by asdff
- I think the idea is really cathartic for many, there kind of is no more supreme resolution than this. You(and humanity) are freed from our flesh prisons of cognition and also get to experience/feel what the next evolution of informational intelligence will look like in the last experiences of it. You might also be the last one to feel/experience anything like that for a long time.
In the game Outer Wilds, the ending is very similar, and a lot of people rank it at one of the best games ever made. I kind of believe that this outcome is probable partially because of this, most scientists working on this really want to see and experience it.
by augment_me - Not refuting your overall point, but the son wasn’t killed. They reunite at the end of the movie.by mastry
- It might not be "foom!", it might just be like...all the computers and networking infra in the world go dark over the course of a few minutes. Could really look like anything, part of the issue is that we haven't the slightest idea what "misalignment" looks like for a superintelligent system.by abound
- Here's a WSJ article about this resignation,
https://www.wsj.com/tech/ai/anthropic-researcher-quits-over-... ("Anthropic Researcher Quits Over ‘Out-of-Control’ AI Fears")
by peri-cl - Shows that no one is immune from the marketing BS of these companies.by Insanity
- More doomerism. Try to implement a deterministic workflow using agents with the latest models and no humans-in-the-loop, and you will realize what they are really capable of. There is too much unnecessary fear-mongering. All of this is only coming from the 2 AI labs trying to IPO. Not from anyone else.by bwfan123
- by ncr100
- Every news headline or public statement these days is a gut punch. Only bad news, and nothing we (as "the general public") can do about it.
Take this one. Ok, AI is going to ruin us all. But let's say we do our civic duty: we protest, vote in candidates with good views on AI etc. and somehow convince or regulate OpenAI and Anthropic into stopping their arms race... Then what about China? It would be a great opportunity for them if their major competitor were out of the arms race.
So basically we have no choice or influence; and even if we did, we'd be choosing from two terrible outcomes.
Same for geopolitics, climate, economy...just bad bad bad all around.
by leokennis - Indeed - we can't just pause some AI in one country, we have to stop it all globally, with an international treaty and monitoring.
Yes, that's hard. It's a challenge. You can set your brain on the challenge! It's a complex and interesting geopolitical and technical challenge.
A good starting point is to read Plan A https://ai-2040.com/
Look at all the things needed for that to happen, and think, how can we step towards making that happen?
by frabcus - It is just plain manipulation. So plain that I ask myself everyday who they fired that did it before, because at least that guy was not as plain and boring.
- The Chinese are humans too. Why would they want to be extinct either?by rahulyc
- If our government cared enough to regulate, they could also directly negotiate with China. There are proposed agreements that don't require either party to trust each other, and even if China doesn't want out of this race, we could pressure them other things like trade.
Examples: FlexHEG from Bengio (proposes on-chip mechanisms that allow workload verification without trust), large bilateral investments into joint AI interpretability or alignment efforts, or invasive audits & inspections (data centers for training these models have a large footprint + this can combine with on-chip mechanisms since producing chips is even more complicated).
And there are probably better proposals available for people to find, if we actually prioritized this.
by uselessTA - It's a bit depressing. I personally try to enjoy the present with my loved ones as much as I can, the future is a bit impossible to forecast right now. I'm focusing on staying alive, mortgage payments, etc.
I share your concern...
by igleria - "No other human activity poses this level of danger." I know this is a bit over the top but let us look at some of the most pressing issues in the world today: global warming, nuclear weapons, wealth inequality, war, technology dependence. Wouldn't a much more capable LLM in the wrong hands make these accelerate faster in the wrong direction ? Many of us , me included, have this mental model of a rogue Terminator-like AI but what I am most worried about is these LLMs in the hands of people. Look at what we have done to this Planet with the tools that we have so far, we have repeatedly tried to subjugate , enslave and kill each other because of things as peety as skin colour , tribe , religion and who owns which patch of land .
I don't have faith in the human race as is to do the right thing when handed these tools , you can already see the nonsense like the fake nudes , fake news and all that other filth being pushed on social media by people with access to relatively daft models. What happens when we can package a Mythos 5 level model in a box , when any war lord , supremacist or religous zealot can access these ? , You now you have your personal bio-chemist and nuclear physicist in a box.
The tool in itself is not what I am worried about , it is the human in the loop. I don't have an answer to this but i believe it is something we should all take a moment to think about , just think what a different world we would be living if any random person could buy a nuclear weapon off the shelf ? There are extremes to both ends , you can either worry way too much or don't care at all, I believe the best place is the middle ground were we actively think this through instead of using our usually "Move fast and break things" mode..
by madradavid - This is increasingly the consensus I see also on the academic side of AI/safety research. Specifically that AI poses an existential risk to humanity.
This was a fringe belief until recently, but the progress of AI in research is impossible to ignore. Epecially in math, where not only has AI outstripped humans in generative ability, but is able to create scientific knowledge which is beyond the capacity of human comprehension.
There's clearly no intelligence task that AIs can't do due to some magic fundamental constraint. And it's hard to imagine a world where current limitations like poor sample efficiency or lack of continual learning won't eventually be solved.
Total AI compute is estimated to grow somewhere in the 1-10 million-fold range in the next decade. Please don't underestimate the phase change that's still coming.
Sure, maybe there's some plateau due to RL being fundamentally limited in some surprising way, but this is nothing but a hope.
by x312 - I really hate how people who have always thought AI research to be an existential risk for humanity, now are apparently bundled to be on the same side of the Sam Altmans and Dario Amodeis that are using the existential risk as a sneaky form of marketing for their products.
You cannot discuss existential risks of AI without being seen as a booster, and that is very unhealthy for the discourse around this tech. I hate how AI ‘doomer’ is now used to indicate pro-AI sentiment. The “moderate” person now is the one that just shrugs and scoffs at the deep societal changes this tech will bring, head deep in the sand.
by sph - > There's clearly no intelligence task that AIs can't do due to some magic fundamental constraint.
Yes there is: write an English paragraph that doesn't make me want to claw my eyes out. LLMs are not better than human mathematicians (or security researchers) in all respects, just some specific ways (e.g. not having to take a lunch break) that make them good at exhaustively searching for an answer, given the right constraints.
by platinumrad - I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need some magical AGI breakthrough first. The dangerous part may come from combining models that are already good enough with an extremely capable harness and enough access.by sreekanth850
- ASI is just Claude in a while loop:by andai
- > an extremely capable harness and enough access.
Give enough access to a fuzzer and it's exactly as dangerous as an LLM. LLMs don't even have a moat in this domain.
by copperx - Doesn't this just move the need to be smarter from the model to the harness - if a human sometimes can't tell whether a model has produced something correct or just mostly correct-looking BS, how can an automated harness do it?
OTOH, if the goal is simple ("break into a protected system") rather than more complex ("write an application that satisfies all requirements on all supported devices/screen resolutions etc."), that's of course more suitable for a harness.
by rob74 - People are underestimating the costs in terms of money and energy.
The third law of thermodynamics is an essential barrier in all engineering.
by contubernio - It's as if two private companies are each building increasingly large nuclear bombs, both saying they'd love to stop but it would be unsafe to let any one company be in control of the nukes.by Mizza
- While simultaneously trying to own all economic activity derivable from labor... it's hard to see the argument as anything but disengenuous.by lumost
- That's basically the reasoning behind MAD, and it checks out? See what happened to any nation that ever gave away their nukes.by Aeolun
- The most optimistic outcome of generative AI leaves us with a technology that warps our perception of reality and crushes labor. The most pessimistic destroys all of humanity.
Our CEOs not only insist we genuflect before these machines but measure our sacrifice and shame our reluctance.
by chewbacha - The children yearn for the mines.by andai