Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I don't think they are toast, I mean they will be in some trouble because all of them have fallen victim to fomo and started building out with so much debt for capacity that may or may not be needed nor achieve the returns that they want. I think there's a future where "personal software" meaning highly custom apps generated by an agent is a thing that doesn't mean everything will become that, same for local LLMs but all of this is still too far. The main issue is that hyperscalers or big tech in general have become too powerful they can just buy their way in and out of legislation as they please, sorry I mean lobby ... funny how if you rename something it becomes legal or illegalby aslkalska
- I believe it is true, and likely there exists a class of even smaller models than what they call "small".
You can imagine a reasoning model as a huge set of rules that generate the next statement from previous statements (written in context). In that sense, a reasoning model can be compared to a logical theory - you have certain deduction rules which can generate new judgments.
Often, logical theories are structured that the rules are remade into axioms, and the deduction rule is only modus ponens (which corresponds to function application and is a building block of program execution).
In the case of an LLM, the set of rules (or axioms) they have in the theory is quite large, but most likely semantically unsound (with respect to their their own representation of truth) - that's why LLM's make mistakes.
It would be desirable to break the logical theory represented by LLM into a smaller set of axioms, which would:
a) remove rules easily deductible from the smaller core of axioms (for example, LLM doesn't need to remember "Socrates is mortal", as it can derive it from "Socrates is a man" and "all men are mortal")
b) remove rules that have low value (facts that aren't used often or have weak validity) which cause ruleset to become unsound
I suspect that's what SLM distillation is doing, to some extent.
The question is, how far this process can go? I personally believe there is a useful logic for commonsense reasoning that has less than thousand rules (still several orders more than your typical mathematical logic, but orders less than SLMs). These axioms do not contain much facts about the world, but that could be added.
So I believe there is a sweet spot (deductive core, encyclopedic shell) which we have not yet found (it's a little bit more formal language than natural language) but is very efficient for general reasoning.
by js8 - The paper underlying this blog post is fundamentally flawed because of benchmark ceilings. If we define only simple tasks like asking what is the capital of France, all models will converge to 100%, obviously. But as bigger models get more capable we want them to replace more and more complex tasks, in as short time as possible.
Then of course there is the economics of it. Do people prefer to spend $5000 upfront to get things done 5x slower, or would they rather pay $20 a month for that?
by pu_pe - One of the big things to think about is whether local LLMs will be things companies want to deploy.
If you think of for e.g. some proprietary piece of software that wants to embed an LLM they've fine tuned or trained, they will want to make back some of their research cost right. So they are not going to want to put this on-device even if the hardware is there, unless there's some way of locking it down. I suspect we'll need on-hardware validation/verification and a way of preventing extraction of weights for this move to happen for many use cases.
by physicsguy - Haven't the SLMs been distilled using the LLMs?
If this is correct I see a future where the hyperscalers are funded by the businesses integrating siloed SLMs in their software.
Also the defence/intelligence industry will always want to keep an edge so don't be surprised if they stick around and we see favourable regulations for them similarly to how the government turns a blind eye to social media platforms because they increase the footprint of mass surveillance.
I wouldn't be surprised if the hyperscalers became software auditors and any piece of critical software was required to have a regulated security audit before it could enter production. Selling the poison and the cure is a great business model.
- "If", sure.
How many developers here don't see a difference between the latest LLMs and SLMs they can run on their own computer? I tried running a smaller model locally, and it's not usable for me.
I know people like to "predict" things, so that if they happen they can then say "I am a visionary, I predicted it" and start their blog posts with "as I predicted long ago (because I am a visionary), ...".
> The research report estimates that the addressable market in the US for SLMs has grown to about $10tn or one-third of the entire US GDP of $30tn. There isn’t much left for LLMs to thrive in, and every year, their advantage over SLMs is shrinking.
I stopped counting the number of times "estimates" said that a market would absolutely explode, and it absolutely didn't. Those are in the business of being a broken clock.
If something better comes, it will be better. Sure. And we would like to have something better, because it would be better.
by palata - From what’s presented this seems to be the lower end of Q&A and reasoning tasks and not long horizon agentic work. I agree that the search engine replacement AI usage is something that can run anywhere (though it’s still better run in the cloud for speed, context length, sandboxing and convenience) but this isn’t the engine of AI growth.
Also, the average consumer is not going to be running a local model until they are built into the hardware they already buy and when they are, who is supplying the weights? They’ll likely be shipped as an ASIC (or MSIC) at that point anyways. Those will use a licensed model from the current leaders. The whole argument sounds like saying that cloud services shouldn’t be profitable because everyone has a computer at home or to meme “we have AI at home”.
- This logic seems mad. If people only need SLMs then hyperscalers can also centrally host higher-efficiency models, and still gain efficiencies of scale and convenience over hosting locally.