Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- "AI is progressing at a rate that our society might not be ready for. Frontier labs are very close to AI that can exceed even the best people on almost every metric of intelligence. This will lead to unprecedented social and safety risks. To ensure a positive future we need to develop AI in a way that is driven by responsibility and thoughtfulness."
Responsibility and thoughtfulness? In this economy?
by ndom91 - We know that at the end of the day AI intelligence is closely tied to invested compute. The fact that automated AI research might spark some kind of exponencial increase in capabilities seems extremely unlikely to me. Of course we know that some got there faster and cheaper than others, but the fact that it requires immense computing power has not changed.
We also have already witnessed a slowdown, the raw intelligence of models has been relatively stable since january, but their use of tools and what we call agent capabilities have allowed us to tackle significalty harder problems with what we have already. This is becoming blatently obvious how smaller models such as gpt 5.6 luna are capable of doing similar things to sol while presumeably being fraction of the side and compute investment. This is also the likely reason why chinese models are able to catch up as intelligence strictly requires more compute, agentic capabilities are more of a mixed bag of RLHF and general post training.
I believe automated AI development will simply lead to cheaper intelligence, but it will not necessarily produce something more intelligent. There is a sentance from a book that I don't remember where I read it from, but has stuck with me which boils down to this: "We are very unlikely to make something smarter than a human brain.". If you apply this concept more broadly an AI system will not be able to produce something more capable than itself, it might be specialize in different areas via trial and error (i.e. solution discovery), but it will not have better intuition than that which was trained from the fruits of human labor. I also strongly believe that model performance is ultimately governed by the advancements brought into the world by humans.
AI of course can find improvements by simply trying a bunch of random things, emphasis on the word FIND. Until we can prove that AI systems are able to come up with unique solutions that have yet to exist even with simplier problems such as creating any kind of new AI architecture that has not been already documented, conceptiualized or explored by humans already -- and it doesn't need to be better than what we have already, just something new -- I believe it's too early to sound alarms until proven otherwise.
--
Something I've realized as I was fact checking myself: We could presumeably have self-improving intelligence if we used enough properly engineered agentic workers working towards more intelligent AI, but at that point we are back to the same problem where we're spending mass amounts of compute to increase the raw intelligence of a model.
by himata4113 - > We know that at the end of the day AI intelligence
Stopped reading at "intelligence".
Try capability.
(unless your target audience is those already gulled.)
by chrisjj - > We request that the U.S. government ...
Too bad these researcher's organizations spent their efforts supporting the dumbest, ficklest possible version of the U.S. government to guide us through the gravity of this moment.
by yks - Can someone help clarify what it means, concretely, to make binding agreements to proceed "safely and slower" in the context of AI research (whether or not at frontier labs)? Are there specific policies and actionable guides being proposed?by girfan
- There's a decent amount of stuff in Plan A. Especially if you read the supplements.by frabcus
- What these people want is same thing people in every other field want too, which is being able to use AI as a tool and nothing more. Off course, some of the people that signed this might not recognize this, and the people who didn't sign it are actively working against such an equilibrium.by ece
- Something people don't seem to see, and not that it's placed directly in front of you, is that pacing the release and adoption of AI is already happening.
Cost-per-token determines the economic viability of tasks that AI gets applied to. How many tokens would it take to automate all of the responsibilities of a gas station clerk? Probably more than their hourly wage. Job safe.
A slightly larger stretch, that most people attribute to profit margins (labs do not have profit margins, they have unlimited invested capital and no expectation of short-term returns), is the flat-rate billing plans of most code products. Why doesn't Anthropic have a 50x or 100x Max plan? Not that it's impossible, that it releases tokens into the workforce at a greater rate (note that their are hour, day, week limits across most of these services as well).
Hypothetical, but reasoned, and I do not expect most agree with me. If compute as a variable is inferred, this supports the incorporation a deliberate (redundant) one as well.
by tcdent - I agree with the substance of your comment that token economics pace the diffusion of AI into the labor market. But I think they have a much smaller influence over the pace of the frontier, right?by alach11
- It's an admirable idea, but I don't see how you can slow this down in practice. We certainly know China will not, and they are already at the frontier.
RSI is too important and too tangible at this point for everyone to slow down. It's the equivalent of telling the world to stop developing nuclear weapons after Hiroshima. Just not going to happen, and in the meantime dramatically increases the risk that the power of AGI lands in the hands of the few (read: the political class) rather than empowers the people.
by biffles - You need AI capabilities to the point where the US and Chinese leaders get scared enough that they do a deal.
There are detailed specific scenarios in Plan A https://ai-2040.com/
by frabcus - This. Will. Never. Work.
This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will always be defections. OpenAI defected WHILE THEY WERE pushing to slow.
The cost to the US of deciding to slow is so massively harmful that it is wrong to even put this idea out in the public zeitgeist; it may somehow become a thing that US legislators believe they can accomplish. They CANNOT accomplish this, because the fundamental economics and possible outcomes make it mandatory for any strategic adversary to work as hard as possible to defect.
The proposals here imagine a global panopticon surveillance state overseeing all research and chips as a 'reasonable' solution. EVEN IN THAT (terrible) state for the world, we will have effective defection.
Like it or not, the world is on this train, and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology.
Variants of this idea in AI 2030 included burning some of the US lead to get some nebulous safety metrics improved - also insane. Any slowing allows strategic adversaries time to catch up with very different alignment goals.
Do I like this situation? No. Are the people signing these things just fully in denial about the game theory and the stakes and the behaviors of strategic adversaries? Yes, yes they are.
by vessenes - Some reason this kind of argument always by default assumes the US will be the good guys.by lifeformed
- "and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology."
These research groups are almost unanimously advocating for a pause/slowdown as our best chance.
by frotaur - The only thing it might accomplish is allowing OpenAI / Anthropic to squeak out an IPO before the market realizes AI will become a commodity and the massive profits these companies are promising will never materialize. They aren't in denial about this open letter's futility in terms of actually reducing the pace of AI, that's just not their goal in the first place.
- Why not? Slowing down / halting biological weapons research mostly worked. Yes there have probably been modest sized defections here and there, but the pace of bioweapon development is a crawl compared with (a) what is possible with science already, and also with (b) the pace of bioweapons development from 1910 to 1970.by levocardia
- The historic analog most befitting AI are weapons. It provides an advantage to companies and governments in creating weapons, in surveilling people and thus gaining leverage over individuals, in controlling what individuals can do or cannot stop doing. It thus follows the laws of weapons control - the more restrictions are threatened to be imposed, the more even the smallest actors will fight to get more of it. They may not understand or know what they are getting, but they will want more of it because their potential rivals have it. Governments are the ultimate customer here - they have the money, the energy, and the appetite to consume AI infinitely in battlement against opponents - foreign and domestic. The only approximate solution is to prevent any restrictions on the use of it - a-la-2nd amendment (the upside of that is it advances the AI for future use and is thus in the interest of governments as well). Modern large models seem to be compressible and thus we may see more and more tiny sub-specialized varieties. But the push to slow the frontier in weaponry always leads to the reverse.by dzink
- > only approximate solution is to prevent any restrictions on the use of it - a-la-2nd amendment
Not even in 'Murica do we have anything close to no restrictions.
You can't buy your own EMP or nuke, and even warring governments have been surprisingly reserved in building (at least to public knowledge) biological weapons.
by paytonjjones - I think we see things similarly about the negative potential but please be careful with the framing.
AI is a printing press or a steam engine. It's not a weapon.
The framing is important. Much like 'distillation attack'. These framings are an attack on the narrative.
Printing presses at the time were seen as extremely dangerous too. For many of the same reasons cited about AI.
In both cases, the technologies were used to destablize existing power structures. And, after some hardship, make a better world for most people.
Even calling these things 'dual use' is a weird framing. Books are not 'dual use technologies' unless you want to sell fear.
- Maybe I am too cynical, but this just sounds like a plea for the government to step in and save them from their investors.by fwlr
- "Oh no! The AI we developed for years, and whose stratospheric valuation is predicated on replacing human jobs, is coming for our jobs, the government should do something!"
Signed,
AI researchers and engineers.
by overfeed - Investors want to see a return on their investment ASAP. The Economist recently published an article:
"AI revenues are growing fast, but not fast enough -- The returns on trillions of dollars of spending are deeply uncertain"
https://www.economist.com/finance-and-economics/2026/07/28/a...
As OP states: "each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration."
Let's not overwrite human civilization for the sake of the shareholders.
by 0xDEAFBEAD - Anyone serious about this should participate in pacing the frontier by quitting their job so as to not potentially contribute to the problematic progress. Be the change you want to see!
Curious how that isn't what happens.
by fidotron - “It is difficult to get a man to understand something, when his salary depends on his not understanding it”by brap
- FWIW, here's an account from a guy who quit his job at Google Deepmind recently:by 0xDEAFBEAD
- Some of us considered seeking one of these jobs and decided not to. Some have quit (e.g. Hinton, OpenAI people after OpenAI reneged on safety commitments). Now you're getting this message from people who passed the previous filters. If you think a brake is indicated and not just easing up on the accelerator, you need collective action as the statement said.by abecedarius
- That's an application of the "Tragedy of the commons": One person stopping to push the frontier just means that this person will lose their job, the frontier will be pushed by the others. The solution is to push for a binding rule instead that will make everyone slow down in the racing towards artificial superintelligence.by mofeien
- The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive.
I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing? We should want the AI Labs to make binding agreements to proceed safely and slower, rather than being stuck in a suboptimal game theoretical race to advance unsafely. We should be hoping that nations can make binding agreements. Our discourse here should be about the substance of what these agreements could be and the practicality of enforcement mechanisms.
by alach11 - > It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
It's not a shallow dismissal if you have been following their - let's charitably call it "evolving" - public posture on AI legislation.
It wasn't long ago the same people/companies successfully lobbied the Trump admin to quash state laws that threatened to "Pace" AI, and set ground-rules for safety. Back then, open source models were very far behind the frontier, and their spiel then was it was imperative not to impinge on AI progress with red-tape, lest America's adversaries catch up.
I challenge you to explain what triggered the about-face. The simplest explanation is that open source/Chinese models have caught up with the frontier, and the change in opinion is tactical (and obviously self-interested), rather than a deeply held belief.
by overfeed - Are they lazy dismissals? Or a simple observation that LLM scaling is hitting diminishing returns way way short of AGI.
If the claim is (1) AGI coming and (2) we should so something about AGI, wouldn't it be smart to confirm (1) before worrying about (2)? Especially if certain companies have trillions of dollars of valuation that depend on society as a whole believing (1)?