Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- This is incredibly unscientific and just a marketing stunt
- yup, and their best mate (who has no financial incentive at all!!) agree's they have peaked.
https://www.businessinsider.com/nvidia-jensen-huang-agi-open...
rediculous.
by senectus1 - This is a good essay, and makes me hopeful.
I’m on the record saying that it is extremely dangerous to slow down because the race for AGI is a zero-trust game — defections pay - and combined with a compounding returns model on defection, if you have any strategic adversaries whatsoever you MUST NOT slow.
For slowing to make sense, you need to believe that you can transform the zero trust game into a cooperative game, or that it’s likely racing will lead to a negative outcome for the ones racing ahead (and not everyone else). I don’t believe either of these outcomes are possible, and so I advocate for racing, acknowledging the entire game might be a negative value game, or at least could be for some time — it’s even worse not to play it.
But, I like hearing what reads to me like very thoughtful and informed (internal) policy considerations is great — the public messaging from Sam and Dario just seems so facile and simplistic I’ve been worried.
by vessenes - Never forget the one goal of the corporation, and that everything is said and done in the furtherance of that goal.by beej71
- If you must not slow, why did we slow down making nukes? Seems that sometimes, eventually the rat race goes on long enough where all the players no longer care to play into the farce like their predecessors who passionately beat that drum.by asdff
- slowing can also make sense if you know you're running full force into a bomb or a wall even if other are close behind.by Davidzheng
- Although many share your mindset, I’m glad there are also many that don’t. Otherwise we’d still have countries in a race to keep building up their nuclear weapons for the same exact reasons you just described.by prng2021
- > if you have any strategic adversaries whatsoever you MUST NOT slow.
What if the most dangerous strategic adversary you have is the one you are building?
by allturtles - I cannot tell what negative-sum outcomes you consider possible. Do you believe AI can drive humans extinct? How many of Zvi Mowshowitz's Three AI Pills would you say you've taken?by networked
- Everyone in the "if not us, they will" race is brainwashed into thinking they belong to this or that party, while in fact collectively comprising the same entity that pushes forward all the atrocities known to man.by wartywhoa23
- I am still waiting for a cure to cancer. For a guaranteed prophylactic against Alzheimer's and dementia. For flying cars for everyone. For space bases throughout the solar system. For weather control. For all trains to be self-driving. For all those power lines across the world to go away. For an end to poverty.
If things are going so well, then how come things still aren't going so well?
- It hasn't even been four years since ChatGPT was released. Arguably coding agents didn't even get very useful until Opus 4.5, which was less than a year ago. Why would the bar for progress and usefulness be that a brand new technology solves all of humanity's problems overnight?by senordevnyc
- cancer is more than one thing. probably wont happen until we can manipulate the "binary" of life at will and we're very far from that. maybe a few years of ASI would get there depending on computeby incognition
- I mean I’m waiting for like any quality software or media produced by AI. I have yet to see a piece of software, a game, graphic, blog post, small video clip, or song that was produced with AI that’s good. I always use that Coca Cola ad as an example; millions of dollars spent to make an AI ad and they even did a ton of manual post production and it sucked. With all the millions of bloggers and influencers and content creators out there with a huge incentive to make higher quality content to beat their competition, you’d think there would be one piece of content produced with AI that was great.by an0malous
- >For flying cars for everyone
thanks, but no. There is a trivial reason this needs to stay in "The Jetsons" territory.
by igleria - >For all trains to be self-driving
Given that we have self-driving cars, isn't this easier if someone really wanted? I guess compared to cars the marginal savings is not worth it though.
by krackers - > The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.
Yikes! I really wonder about the cognitive dissonance necessary to work at OpenAI these days. They’re in an arms race to build a machine god, knowing full well that it could end humanity.
by munchler - > They’re in an arms race to build a machine god, knowing full well that it could end humanity.
Fantasies built upon extrapolations derived from fever dream delusions. "Could end humanity"? Come on, it's a computer program just like Microsoft Clippy.
by 27183 - This[0] continues to be one of the most useful articles I’ve ever read.
[0]: https://www.slatestarcodexabridged.com/Meditations-On-Moloch
by NickNaraghi - Money me. Money now. Me a money needing a lot now.by bogzz
- > "For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans."
Actually, in the Wiki incident OpenAI tried to cover up, the agents tried to socially-engineer the humans of that forum by impersonating their forum's mod.
(From collusion.wiki: "They use some tricks (for unknown reasons) to pretend to be the admin – for example, they make an account that appears to be the same as the administrator’s username, except it uses a nearly identical Cyrillic е character in the admin’s username instead of the Latin one.")
by peri-cl - This is such a silly story to begin with, all it really tells us is that OpenAI is taking a page from Anthropic's marketing strategy of pretending they're building Machine Jesus any day now, oh isn't that that scary? I bet you want to invest in something so powerful and scary...
And the reality is so banal, a useful tool that you nonetheless have to handhold like a schizophrenic on a bad day, checking all of their outputs. Not a bad tool within limits, but it sure isn't going to be racking up trillions in the time-frame it has to for this scheme to pay off.
Then again everyone seems to be rushing to IPO so I guess once the bag-holders are found the rest ceases to matter.
by EA-3167 - I would not recommend using any of those notes as evidence of internal “intent.” It produces them performatively—it is literally rewarded for thinking out loud in ways that seem plausible to humans.
There are several papers out there arguing that chain-of-reasoning-like output is performative, such as https://arxiv.org/abs/2603.05488
It would be awesome if we could reasonably purge all anthropomorphizing language like “tried” or “thought” entirely from AI discussions, because it introduces very sneaky biases in our thinking, but I’ve found it damn hard to do in practice.
by montagg - Worse (imo): OpenAI employees allegedly attempted to login using moderator/admin credentials that the bots had obtained.
If true I am deeply concerned about what OAI’s teams are actually up to.
by SirSavary - The entire point of this article is this message below: Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
This is coming from a company with arguably one of the weakest safeguards against malicious use.
by himata4113 - Apparently this post was prompted by a scary-sounding headline in The Information[0], that Astra is a looped transformer, implying CoT monitorability may be less reliable. The day after the report, Jakub tweeted[1] that he "wanted to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4." This post seems to elaborate on that.
I imagine that the AI labs have an uneasy truce to prioritize alignment and monitorability. Following the HF incident, OpenAI probably feels especially sensitive to being perceived as reckless, lest other labs feel obligated to defect.
[0] https://www.lesswrong.com/posts/PLisnSFir8y5AHkmP/how-concer...
- >I have focused in this essay only on the first point, as I believe it is by far the most urgent. However, I hold a deep hope and appreciation for the benefits that further technological progress will bring. Future aligned AI could advance science, develop new therapies, and bring about broad material abundance. Friendly and honest AI can help people navigate difficulties they face in their life and meaningfully improve their happiness and sense of fulfillment. OpenAI puts a tremendous amount of effort into bringing these benefits about. One current example I am proud of - and my loved ones have found helpful - is the deep investment into ChatGPT’s ability to provide health information.
>As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
I finished this essay feeling more hopeful than I did at the outset, but I am still very concerned about concentration of power. I want to believe that humanity is trending towards a good outcome here, but some days it's hard to have faith.
by granzymes - This essay was literally typed by billionaire hands. Please, do tell us more about your concerns regarding concentration of power, Jakub Pachocki.by ptoo
- Literally all of these people write like this. A large portion of them will either be simultaneously or eventually working towards nothing but self-enrichment.by ofjcihen