Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- LLMs are basically multi-dimensional magic mirrors.
Depending on where you point them, they can be incredibly useful.
They can even be useful when you point them at each other (though increasingly difficult to get good results).
I'm excited for the promise of RSI and a future where models have inherently "live" weights, but it's not clear to me that the transformer is more than a useful tool to help us get there.
by jumploops - > those who need done a small set of narrowly defined tasks with existing clear guardrails: repetitive physical labor in a controlled environment, call center and customer service chat work, etc.
I have no idea how people can so confidently say that call center work is a “controlled environment” or “repetitive”. It’s almost by definition not repetitive or controlled. Customer support is what I go to when the controlled environment has failed
by yunwal - came here to say exactly this. in fact, this is probably why we are not seeing a lot of AI application on customer service use case, and when we see one, it's almost always frustrating.by fhe
- Depends on what customer support means.
Typically it means knowledge retrieval from a KB or manipulating a control surface not visible to you.
by vachina - One the main theses is
> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on
Unless frontier labs have surprisingly trained their models in the exact tasks my team works on, this is patently false. We are getting very good results on automation and I'm bullish we will be able to mostly remove humans in the loop for most of our infra tasks by the end of the year.
I have no opinion on the other theses, but given that OP doesn't back up these claims in any way, I have my doubts about the conclusions of this article.
by angarg12 - > the best alternative to rigorous specification is human review. human review doesn't scale well to the volumes of output produced by language models. to make matters worse
when the business model is selling more tokens you get such per serve ice times that lead to “more” thinking, engagement baiting, fluffy narratives, and straight up dark patterns
by ausbah - Short and to the point! Open and cheap models will undercut the big labs continuously. The blast radius won't be pretty once spending commitments knock the door.by knuppar
- You said it like labs like open AI doesn’t know and don’t constantly make moves to prevent that undercuttingby m3kw9
- I agree with you but I'm still worried about the safety of open weight models as well. Both aligned and unaligned models.by pvab3
- Open models wont be open for long. No one is going to release an open model capable of chaining zero-days. Even the Chinese aren't that reckless because it will just be turned around and used against them.by woeirua
- "Language Models are Few-Shot Learners" - 2020, the GPT-3 paper.
It have been demonstrated that in-context learning is a very powerful mechanism. There's no evidence that models of the size of GPT-6 are bad at in-context learning. In fact, ARC-AGI-3 score might indicate they are good at it.
There's no evidence that a bespoke RL environment is required for each new skill - quite likely a good demonstration is sufficient.
by killerstorm - I think bearish on LLMs for automation, and bullish for LLM+human experts in specific fields, is about the right expectation for current architectures.
Apart from issues with task generalization, or perhaps related to it, is the fact that LLMs have real trouble with timekeeping, and cannot estimate the real world time it will take them to do things very well. This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.
In narrow domains with more deterministic outputs though, this is less of an issue, and we see multiple agents succeed much better.
The fusion of that capacity, with humans in the loop able to better direct such agents and act as their temporal tethers, is where I think the real action will be for a while at least.
- > This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.
Any reason why that can't be solved through context management and keep-forward scaffolding?
- I think automation is coming but it will be way more gnarly than frontier labs want public to believe. Value is just too big, when you can automate most of eg customer support it will create huge savings and same time customer satisfaction will get better.by antupis
- My belief is that LLMs will fundamentally change how we approach domain expertise. From what I've seen, SDEs tend to be over-specialized compared to what the company actually needs to implement due to the need to understand enough of the domain to pick a best path. If an LLM can see the domain enough so that someone in an adjacent field can be confident in their approach and quickly change course then you don't need as many niche SDEs
- "are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers, "
No, they're really not.
They're priced in a way that would imply AI will be universal form of compute, alongside traditional deterministic systems - which it will be.
And that they will capture most of that ... which they won't.
The Frontier Labs are a very bad buy at a high price, but that partly has to do with wacky pricing, but actually mostly has to do with their relatively weak place in the value chain.
The money is going to Nvidia, who have the most powerful position.
A bit like how a retailer can take all the margins of some innovative product, if they own the channel.
AI is over-hyped, the Frontier Labs are over priced - but AI is here to stay, and will grow. Not like Skynet, but like a new form of compute. And it will take it's time, and the profits will be reaped by those with the power.
by bluegatty - ai has a >10% chance of causing human extinction, according to anthropic big heads.
if that's true, you are wrong.
if that's false, anthropic is dishonest. why trust a dishonest company to be worth anything?
by lukewarm707 - They are pricing as a first mover AGI monopoly that makes no sense.
I think the core mistake is this partial-equilibrium reasoning. Take the new technology and then hold everything else fixed.
$40 trillion of knowledge work routed unchanged through a new toll booth. Profit. This has nothing to do with reality.
Nvidia on the other hand does have the CUDA monopoly so their toll booth is printing money but that will get routed around or broken at some point.
by webern777 - Traditional non-chaotic systems*
LLMs are deterministic. They are chaotic, which people confuse for non-deterministic.
by Zambyte - The premise in the very first point seems off:
> the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers...
Even assuming this is how the AI companies are being valued (they're not), the numbers are off.
The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually. It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that, the entire AI industry would be valued at double-digit trillions at the least.
Yet cumulatively the industry (the frontier labs + the SWAG estimate of the AI parts of all the other players) are valued at, say, ~6 - 7 trillion? Which seems like a fair approximation of how much knowledge work they can currently automate.
by keeda - “Bearing” is not how products are priced.
You need to think in terms of supply and demand.
The demand is there, but the supply is also going to skyrocket. Free open weights models will contribute to supply too.
There will be a new equilibrium that’s hard to predict.
by randyrand - > The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually. It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
I don't think that AI companies can charge the same. The human workforce can charge these costs, because of scarcity. But AI systems won't be scarce, it's just a matter of who can run inference cheapest. Plus you still have the human workforce, which might be forced to offer their time for less money.
by _ink_ - There's a leverage issue.
In one case (financial services) it's thought that expertise is valuable at V=S^2/b4 where V is value, S is skill and b capacity (the leverage available to the manager/expert. b erodes as it becomes harder to find examples of things that are not done well, so if you manage $1bn you might find lots of miss allocations that you can exploit with just that $1bn really effectively, but if you manage $10bn it's much harder to find good places for the extra $9bn. A low hanging fruit effect.
Anyway, that double hit - raw skill and the amount of times you can supply the skill makes the value of skill (V) convex, and it means that in a perfect market (heh heh heh) someone running $100bn is worth 1000's or maybe 10,000's of an average joe expert.
Now, if AI is trusted to run the top 0.1% of everything and has the skill to do it at human top level expertise, then your calc holds. If it's the case that it isn't then more than half of that value disappears. If it's not even top 1% then chop out another 25%.
That implies that we need a lot of trust and a lot of AI capability before these valuations stack up, and it also implies that all other competitors and incumbants are going away. I do not think that Citidal or Bridgewater are going to let Anthropic or OAI take them without a fight. They might lose - but there is a decent bet that they don't. I don't think that many professions like Lawyers or Doctors are just going to roll over and cede their monopoly rights to OAI or Anthropic either.
by sgt101 - I would add to the other comments that are replying below the following thought : replacing workers with AI means less income/spending, which drives the value of those companies down at the same time.by AureliusMA
- > It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing
This assumes you don't change the market, but at the scale of (checks notes...) "all knowledge work", that just doesn't hold.
For example if you put 1bn people out of work, you now need some sort of safety net to bail out much of that workforce, a truly unprecedented change. You also lose tens of trillions of dollars of tax revenue.
One solution might be to recoup that cost and lost tax revenue from businesses by raising corporation tax. If corporation tax went from low tens of percent to high tens of percent, would those businesses be able to afford all that AI? No. Same order of magnitude? I doubt it.
There are many possible futures there, but the simplification made in the parent comment is completely unrealistic. The article is right in calling out the valuations as crazy.
by danpalmer - The real issue IMO is that is not really what Anthropic and OpenAI are operating on.
That is the after the fact justification of the AGI dollar auction. Each round is kind of 3x the previous cost and neither can really stop because second place in the dollar auction is so much worse than winning.
The only way to stop the auction is one bidder hits a hard budget constraint, both agree to stop, or an outside party breaks the auction.
IMO this is why they want to slow down or have regulation. I think this is also why we see some claims of already reaching "AGI".
The TAM of global knowledge work is just a narrative tacked on after the fact to justify the AGI dollar auction.
The economic fallacy here with the actual valuation is akin to pricing the electric utilities 120+ years ago as some % of the future cash flow of global food production. Take the TAM of global food production and then work back to what % will the electric utilities capture from the advances in the automation of farming? It is nonsense.
The only narrative that actually justifies the capex spend that I can figure out is a first mover AGI monopoly. Even the oligopoly case is hard to justify the capex spend IMO. There is this enormous mismatch between the AGI monopoly and the actual rolling 12-month window of pricing power.
Even the rolling 12-month window of pricing power is going to saturate well before AGI too so it is hard to see how any of this makes economic sense.
by webern777 - When thinking about these valuations, shouldn’t we try to quantify how much knowledge work becomes obsolete if other knowledge workers are automated? I.e. there are a huge amount of knowledge workers employed in businesses that create tools for other knowledge workers. AI won’t automate their work, those businesses will just cease to exist.
And then there’s the second order effect: if all the knowledge workers get automated, who is going to buy the stuff that’s produced?
- > The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually.
What do you mean? The sum of ALL US salaries is $13.4 Trillion per year. According to google $65T is the sum of ALL salaries Globally (not just knowledge workers). It's not reasonable to assume AI is a drop-in-replacement for any job yet (perhaps bottom tier customer support from oversees?).
> So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that
So you're sort of premising here than more than 16% or 1/6 of all the world's jobs get replaced by AI. Hopefully you can understand that's both not the current AI capability and also would be a terrible (unprecedented?) economic shock.
by zug_zug - Great article, but the lack of sentence capitalization makes it unnecessarily difficult to read.
Apologies if this comment is off-topic, but it really is quite egregious, and since the article was submitted by the author I presume they are open to the feedback.
by yshklarov - Counter-point: Doesn't make it hard to read at all, definitely not "egregious" as it's a very common style to find for informal writing all over the internet.
Author's own style is certainly refreshing and welcome over LLM slop that dominates most HN posts now.
by mattacular