Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- > (I used Mistral as reviewer here. I typed every word myself.)
What an irony. This is absolutely hilarious.
- I agree with everything you wrote but I would rather choose to be an optimist here because it will unlock an age of discovery where humans are going to live at 10x of their potential eventually and cure cancer and become a society bold and technologically adept enough to cut through the universe.by kkkamur
- There so much blind faith encapsulated by the assumptions that A) people will live 10x longer and B) that this will be an inherent and obviously good thing. There are obviously major complexities to such a scenario, not to mention a sort of narcissistic hubris.
But yeah, let’s cure cancer. If only there was even a blip of increasing evidence for that supposed eventuality coming from AI. Maybe I’m just not seeing it? I’d love some citations to correct the record.
by talon8635 - There’s actually a potential dark future where even this 10x ascension itself would mark the end of humanity. Humans would be the Homo sapiens who remain at 1x, and those who can access and afford to be 10x eventually split off into a new species. And historically there isn’t room for two species at the top…by cush
- What about the best-case optimistic outcomes for AI?
Here are some amazing positive outcomes which I think AI may bring us:
- the breakup of tech monoculture and tech monopolies as it becomes impossible to secure bloated overcomplicated operating systems and software due to never-ending vulnerabilities. Once Mythos/Cyber capabilities become mainstream I think the future of software is small, focused teams building high-quality, securable software in multiple jurisdictions. Monoculture software becomes inherently dangerous. Big bloated tech monopolies like Microsoft will collapse under their own weight. Like what is happening to GitHub right now.
- the ability for small teams to compete with multi-billion-dollar behemoths, and that is already happening right now.
- curing cancer and other scientific breakthroughs like Terence Tao’s use of ChatGPT
- a return to local trust-based networks because digital content can no longer be believed, and that is already happening. Any video can be disqualified as AI, which means people are going back to in-person networks to validate trust.
- you can just do things and learn anything. It's so much easier for me to learn almost any topic with AI-assisted content organization, retrieval, etc.
by znnajdla - With each passing day this seems more fantastical to me than sever negative outcome.
We have major black hat cyber security breakthroughs today. Where are the possitive medical breakthroughs?
by talon8635 - The author having mistral review the seven paragraphs in their short article is wild to me. It’s not like it’s a doctoral thesisby oinoom
- Totally agree. People have become totally dependent on AI even if they don't really need it.
As an analogy, I think about my dependency on Google Maps. Salt Lake City is probably the easiest city in the world to navigate because the streets are laid out in a Cartesian grid, and addresses are just literally those Cartesian coordinates (i.e. 500 South 450 East means 5 blocks south and 4 and a half blocks east of the center point, which is the SLC Mormon temple). It's trivial to know how to get to any address, but I reflexively enter in to Google Maps whenever I drive.
- by mag7269
- underrated comment :)
- "I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw. Yet, nobody seriously tries to sandbox AIs because they are too useful with access."
No one takes Yudowsky's claims that AI will destroy humanity seriously. There are many people who are seriously trying to sandbox AIs.
by kittikitti - "Nobody" if you're democratically-minded and think in terms of number of people.
But the handful of tech billionaires at the helm of the world economy absolutely do believe it. Or at least pretend to believe it. Or maybe they can't even tell the difference anymore as they strip-mine science fiction for any vision of a future that doesn't just look like an incrementally worse version of the present.
by jackb4040 - > No one takes Yudowsky's claims that AI will destroy humanity seriously. There are many people who are seriously trying to sandbox AIs.
I know some people at AI labs, there's definitely people working in this field who take it seriously.
This does not stop them also working on e.g. better sandboxes.
by ben_w - HN automatically stripping away the "How" makes the title sound funny.by chistev
- > Like a locust plague, they descend on any open wiki and forum and overwhelm them.
That’s the core pattern of unsupervised agents, the locust plague metaphor is apt. We do and will see that in every single system AI can interact with, be it human systems, software systems, etc. Relentlessly search for an entry point, flood in, consume the whole thing from the inside until there is no value for humans left.
You built a new, innovative software company? Thousands of agents will be working replicating the whole thing in no time. You publish your writing? Exact same thing, as soon as you get some traction your work is replicated in no time by thousands of agents. Same for videos (the whole “faceless YouTube channels” pushed by ElevenLabs and similar). Same for online courses. Same for any website with moderate value. Same for music or other digital art form. Any administrative service available online getting flooded by submissions.
by dgellow - As someone with a hobby interest in theology:
Arguments about super-intelligent AI have all the hallmarks of the philosophical "proofs of god's existence": they start from seemingly innocuous premises, and conclude in an apparently airtight way that a god exists.
There is a tendency among rational minded folks to look at such proofs of god and exclaim "you can't DO that" then turn right around and do the same thing about superintelligent AI.
The key message I want to deliver to people is:
A) Philosophy is not such a trivial thing that you can just wander in, "be a smart guy" and find flaws in established philosophical arguments.
B) AI has real theological implications, and everyone is tiptoeing around it. More than one public intellectuals are trying to smuggle their own metaphysical positions into the public consciousness via discussion of AI.
by bloaf - Interesting points. Can you share some thoughts about the theological implications of AI? Interested to hear.by edg5000
- >a superintelligent AI will inevitably destroy humanity
Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
Perhaps an agent could take all internet connected services offline, but that's not destroying humanity. An AI agent can only see and interact with the world through digital things. Just find the server it's running on and pull out the Ethernet cable. Just cut the power line to the data center. Turn off the whole power grid if it really comes to it.
by Crabber - I encourage you to read the forecast report "AI 2027" - it goes in detail on this. You can disagree with some of its points and conclusions, but it's pretty inevitable that AI will increasingly start interfacing with the real physical world through drones and robotics.
Humans are already putting AI into drones that kill people. A Russian drone killed 3 civilians in Ukraine and the targeting was done completely using onboard AI (no radio connection) using an Nvidia chip.
- By convincing people to do its bidding in the physical world. Language is powerful, and even now, there are some people who have fallen in love with current LLMs. If we imagine a truly super intelligent LLM, it isn't difficult to imagine that it could have superhuman capabilities to deceive, convince, and maybe offer Faustian bets. Even if it doesn't have a body, if it can convince enough people that do have them (and with power to carry out destructive actions) it can do a lot of stuff. Self-preservation and self-replication wouldn't be much of an issue if it had people at its side to help.
With this I'm not saying ASI will happen, but definitely if it does happen and it's not aligned, I don't find it much of a stretch to imagine that it could destroy humanity.
by Al-Khwarizmi - That's what the Mayans thought - what harm can a couple of people on a boat do?
> how is an AI going to affect anything in the real world?
At least two ways:
* Actuators, such as robots, industrial control systems, etc.
* By influencing humans: bribery (yay for crypto), blackmail, election interference, interfering with sensors (e.g. making it appear as if a nuclear attack was under way), and many other ways.
Either way, creating a biological agent that eliminates most humans (or food supply) seems quite feasible.
by FabHK - It doesn't really have to go nearly that far, something like replacing most jobs and collapsing the global economy while making people into mindless idiots from constant reliance on it would do it already. The interpretation of 'destroy' is in the eye of the beholder.by moffkalast
- > Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
It's a great question. I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.
by mjr00 - The list of things that can be done digitally includes:
Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers.
(And of course, building robots, but let's set that aside).
So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?
- > how is an AI going to affect anything in the real world?
Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but some people are so blinded by the hype that they don't stop a minute to think about what they are doing. Those sorts of people are very likely to let an actual AI loose on the world by hooking it up to physical infrastructure.
by bigstrat2003 - This is some Terminator version of destroying humanity, but a super intelligent AI could be much more insidious, and simple.
It looks a lot more like social engineering.
One easy, obvious example that has also been explored a thousand times: it could convince all nuclear countries that they are under attack by another nuclear country. Everyone nukes each other and Earth enters nuclear winter.
Maybe not every single human dies, but humanity is effectively destroyed, by our own hands!
> Perhaps an agent could take all internet connected services offline
And there wouldn't even be Spotify!
The most likely humanity-destroying outcome is the one we can't imagine, because a super-intelligent AI is smarter than all of us combined.
by mvkel - This is a bit of tangent and more of a rant. (and please don't suggest intervention of any kind)
> superintelligent AI will inevitably destroy humanity seems to have no flaw
For a section of population who were in a certain age-range when the pandemic hit and that derailed their certain kinds of plans/hopes/dreams in personal life and then this LLM/agentic world moved in, especially if they are in an industry directly and most affected this, this has been a surreal and slow Kafkaesque nightmare for a few years. The thing is with time and age passing hope keeps getting lost inch by inch and every lost bit of hope aids in killing the next bit. Then comes the sense of dread - the unchangeable reality that age passed passing means there are certain experiences and milestones that are essentially out of reach because there's no going back of this clock.
So I feel a strange kind of relief about this possibility of doom. I am just sharing what I feel, and I don't feel good about feeling this sense of relief that after all it might end for us all and not just an unlucky few. This feeling and the calmness about it saddens me.
- > * For a section of population who were in a certain age-range when the pandemic hit and that derailed their certain kinds of plans/hopes/dreams in personal life*
...
> So I feel a strange kind of relief about this possibility of doom.
So let me make sure I understand what you're saying. The pandemic disrupted your life plans for a few years and threw you off the track you planned? (Which does suck.) And now the possibility of actual human extinction makes you feel a strange relief? Am I understanding this correctly?
If so, this is a remarkable level of something. Depression? Nihilism? Jealousy? Something else?
Imagining total human extinction and feeling relief is not a healthy state of mind. And, yeah, you seem to know this. But I kind of wish we could reach some sort of broad public agreement that human extinction would be bad. And if it ever looks like it might happen, we should fight for human survival.
by ekidd - I mean clearly what you are talking about is a personal feeling here. You mention it saddens you and that is the concerning part imo but you also said no intervention bs so I will just share my counter factual: I am also kind of okay if the world ends tomorrow, to me it never mattered in the first place. But that feeling doesn't sadden me. I feel it frees me because I don't think of life as a very very serious thing and I feel there is a certain kind of lightness you can afford when you fundamentally don't care if you/the entire world ends tomorrow. I still love people and I still love my life and I don't want any harm to anyone. So I do feel the same calmness that you kind of talk about (for life in general aside from just AI killing us all) but my interpretation of it doesn't sadden me. If it did, I would really investigate why I interpret it that way, it might help you discover yourself. There is no shame in being okay with the world ending imo but there is sin in willing it or contributing to it because one thinks its inevitable. Furthermore, I also believe that we only get 1 life which is temporal and fundamentally doesn't have any meaning. How you live every day is how you live every day. We can suffer, we can cry, we can feel sad and for very good reasons but our relationship with ourself shapes our entire lives. If something constantly makes us sad, we should examine it to understand where that really comes from.by altmanaltman
- I have similar feelings, you’re not alone (nor alone in being saddened by it). Things seem to me to be more or less playing out the way I interpreted existential risk researchers saying it would, so there’s a strange relief in knowing the general shape of what is to come.by fwlr