Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Unintentional consequences or accidents enabled by technological/AI ubiquity is the more probable existential threat. We might accidentally poison ourselves, harm infrastructure or supply chain(s) in some very damaging manner, or do something else unwise that isn't readily apparent due to increasing system complexity.
- I agree.
It'll probably just hack and disable our water works, power stations and supply chains and let us all starve.
by nprateem - The author doesn't have a security mindset.by bananaflag
- Ye domesday weapon will not be a disenchanted teenager whipping up COVID-19 variants (whatever that means) and spraying them on people.
The real deal needs a lab messing around with viruses and whatever happened a few years ago ... and it did not destroy humanity.
by gerdesj - > The real deal needs a lab messing around with viruses and whatever happened a few years ago ... and it did not destroy humanity.
That observation maybe holds if the lab from a few years ago correctly selected the virus for the greatest overall lethality (why not MERS or Ebola or something else?) and made its best effort to turn it into a bioweapon. I don't know about you, but I don't think that's quite what happened...
by Scramblejams - "The problems with which the universe can confront any society are, for practical purposes, infinite in number and endless in variety." - Joseph Tainter
AI will really aid the universe in generating more problems. Maybe it's not a supervirus, but hell, maybe it is.
by 01284a7e - That quote is very interesting, because I think I interpreted it the opposite of you:
"we could be attacked by an infinity of problems, which means that worrying about all of them is statistically unpractical"
by ghurtado - Oh my God THANK YOU.
I do not have the words how tired I am of hearing tech people and economists talk about biology.
by colingauvin - Do you need to be a physicist to realize an asteroid is going to kill you ?by mrits
- As a biologist whose PhD research was on the genetics of microbial pathogens, I am far more concerned than the author.
I think he trots out a lot of straw-man arguments (e.g., teenager making the weapon, "super"virus, etc.).
Those scenarios are purposely far-fetched so that he can ridicule them and thereby leave the impression that all the AI/bioweapon concerns are similarly far-fetched.
He spends a lot of "tokens" discussing AI-generated viruses. But bioweapons can be much simpler than a virus--they can be small molecules that inactivate essential molecular targets in humans. This is something that was discussed at length when I was in grad school and it wasn't dismissed as hyperbolic hokum.
So, is AI going to develop a supervirus? Probably not. But it doesn't need to.
Instead, I speculate that AI can "help" in making bioweapons ...
a. more potent
b. less detectable
c. easier to disseminate
You still need humans in the lab to run the experiments, but I think it's foolish to dismiss the worries about AI in such a condescending way.
by busyant - Just because the author can't think of a way to cause mass death with an engineered virus, doesn't mean an AI won't be able to think of one.
(There are several biological threats the author hasn't considered. I'll just leave it at that.)
by Metacelsus - The author can't think of what a superintelligence will do, because they are not a superintelligence. But that is not really what people are concerned about, so I think the objections here are quite meaningful.
If there is a fully general superintelligence - in the current state of alignment - we will all die for some reason. It doesn't really matter what the reason is.
by jeremyjh - I am not sure what you mean here. There exist plenty high-concern biological threats that dont need any AI help. Human oncovirus design is low on my concern list (immunity is diverse), and in any case it does not need AI—rather labspace. Disgruntled high schoolers or undergrad chemists can do way more damage from readily available materials without AI and without research delay. As can nature (or amateur biologists without AI) by mixing bats with their animal of choice and waiting a while. If/when any scary global events like covid happen again, I sure hope we have true superintelligence to help us navigate it quickly.by pama
- I know what you’re thinking. A modified HIV virus that also makes everyone hot and horny.by phoghed
- I was a biophysicist and experimental biologist for decades, I designed viral vectors and cell therapies, etc. What protects us here isn't our ignorance vs an "ASI".
It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "superviruses" would require a lot of iterative experimental development in humans.
(And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)
by alevskaya - Everyone is gathering around patting themselves on the back, but smallpox depopulated an entire continent. Clearly it is possible for a highly transmittable deadly virus to exist.by Footnote7341
- The author seems to be arguing that it is biologically impossible or something to have a virus with the same transmission properties as COVID but higher virulence. I don't understand what principle he is invoking to make this claim.by woopwoop
- The cope principleby mrits
- Because fatality and virulence stand in fundamental tension with each other when it comes to epidemiology. You can't magic that tension away or just invent other incompatible properties of a hypothetical virus (stealth, latency, virulence, fatality, etc etc etc) just because you want to tell a ghost story. Biology: (immunology, cell biology, virology) impose constraints!by alevskaya
- Native americans had like a 95% population mortality rate from infectious diseases in the first generation of European settlement, no? Measles, smallpox, flu, etc.
The mortality rate from those diseases was higher than today because the population had no immunity at all, leading to both higher transmission rates and higher mortality rates. This would be true of an engineered virus as well.
And it wasn't just the one virus, it was an entire continent's infectious disease arsenal landing in overlapping waves. Again, something you'd do if you were trying to maximize damage.
So I don't find the bulk of the article about it being impossible for a virus to have this effect all that compelling.
by jsnell - We might even get to relive history the way things are going…by BobbyTables2
- > So I don't find the bulk of the article about it being impossible for a virus to have this effect all that compelling.
I think the other flaw the article has it that it assumes the bad actor is doing targeted design. That they or the AI has some specific virus in mind meeting a set of predefined criteria and they are fighting the biology to hit that predefined target.
But what if the AI-enabled bad actor is a true chaos agent and has no defined goal and they are using tooling that AI helped them construct to perform genetic reassortment pretty much at random to see what happens?
Most of what they create won't really do anything. But some of it will. And there's a low but non-zero probability of Something Really, Really Bad.
by georgemcbay - This number gets cited a lot but it's actually the excess mortality of post-European contact, and it's not content wide, it's specific to Mexico. It also includes climate change, famine, war, relocation, etc.by colingauvin
- > When people believe that AI could figure this out on its own, without first running extensive and expensive basic science experiments, they have left the realm of science and have entered what can only be described as religion.
I strongly agree with this. Too many people have the Arthur C Clarke quote infecting their brains.
So much of science is contingent on the physical world and cannot be determined from first principles i.e. by thinking hard about it. (Sorry Plato, you can’t think your way to the moon.) You need increasingly precise measurements, which means increasingly complex tools which means…supply chains. Science requires supply chains. Magic doesn’t.
by janalsncm - Is it so hard to think that super intelligence could piece things together that humans can’t?by mrits
- We're doing that science right now. We're using AI to do it. By the time AGI truly spreads, we'll have a lot of the knowledge you say will be needed. This is simply not as far fetched as you seem to think. Next few years, unlikely. 10-15 years? Increasingly possible. 20+ years, I'd hazard even likely.by naasking
- How about just take an existing design like
>Scientists condemn 'crazy, dangerous' creation of deadly airborne flu virus https://www.theguardian.com/science/2014/jun/11/crazy-danger...
and either produce that or modify it a bit?
by tim333 - A morally bankrupt scientist would give AI the hands it needs. Or a military general who thinks he must use AI and any means possible to win, because his enemy may use AI first.by akomtu
- And if an agent were to amass a large sum of Bitcoin you don't think people wouldn't do things for money? Especially if they don't realize they are doing part 1 of 160 parts that get it to the end goal of a virus?by ewild
- This I read yesterday on LinkedIn, so a whole lot is already possible.
---
Markus J. Buehler McAfee Professor of Engineering at MIT; Co-Founder & CTO at Unreasonable Labs; AI-Driven Scientific Discovery
Astra is an incredible world builder and explorer, and can invent its own scientific instruments - crossing a game-changing threshold: it turned a few images of a biological microstructure into a full-blown metamaterials lab, then used its own creation to discover a design with ~2.1x the reference work-to-failure/peak-strength ratio. Metamaterials are some of the most complex materials we can engineer. Their properties come from deep architectural complexity - struts, cells, disorder, and hierarchy arranged across multiple length scales determine how the material deforms, absorbs energy, and fails. Designing them means searching a geometric space far too large for "intuition" alone. In this experiment we handed an AI agent that entire problem end-to-end, from image, to simulating the physics, to fabrication-ready geometry.
We started with an image of hierarchical, biological architecture; Astra then built a complete 3D metamaterial studio: editable geometries, multiple levels of hierarchy, disorder, gradients, a complete physics simulator featuring linear and nonlinear material responses, deformation and fracture experiments (with replay), and STL export for 3D printing.
Then we asked the agent to use the app it created to search for a high work-to-failure/peak-strength ratio. Among the candidates, the leading design reached approximately 2.1x the reference ratio.
The movie follows the structures through deformation and fracture, connects their designs to the property map, and shows the finalist assembled geometrically into a connected multi-scale material.
by fsiefken