Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- LLMs do not reason. They also have a very different moral compass than humans. Maybe stop hurling trash articles at everyone
- The article doesn’t argue that they do reason.by jrmg
- If you define reason as: "Incrementally refine output vectors to converge at the correct output" then one could argue that they do reason.
Their reasoning tokens are refining their eventual output
- The idea that human-readable explanations emitted by a language model don't necessarily correspond to the model's actual internal process of reaching a conclusion reminds me of parallel construction [1], a (fraudulent) law enforcement strategy of obtaining evidence of a crime through usually illegal means and claiming that the evidence was obtained legally through some other means.
[1] https://www.hrw.org/report/2018/01/09/dark-side/secret-origi...
by hn_acker - If anyone is interested in visualizing AI reasoning, I made flame graphs of Sonnet thinking output tokens which are colored and organized by purpose, for example, verification reasoning is purple and error correction reasoning is purple. [0] I asked the model to solve the same problem with the same prompt 5 times so you can see the differences in reasoning granted the coding agent sets the model temperature very high.
I won't get into the metaphysics of reasoning, however, the Sonnet is using an OODA loop. The difference which hasn't been gapped is that human reason and imagination (in the sense of Mr. Rogers' Neighborhood) can predict the consequences of the actions we take.
This ability to loop is much, much wider in Opus 5 than Opus 4.. I had to strain to get Opus 4. to do the wider OODA loop but Opus 5 does it out of the box. I needed to throw out all existing instructions, skills, guidance, moving from 4-* to 5.
by dataviz1000 - The article is arguing that your color coding is misleading because the ‘purpose’ of the tokens doesn’t seem to be what a plain English reading of them would suggest. They’re not a representation of ‘why’ the process ends up at a correct answer.by jrmg
- The more I read about LLMs and more complex ML in general, the more I realize nobody really knows what is going on.by AsyncBanana
- That's pretty much a given when it comes to neural networks.by Sharlin
- It's been this way for a long time, basically since deep learning became the "default" for ML. I remember back in 2018 taking a "Deep Learning" course and one of the most emphasized aspects of the approach is how much of a "black box" it is and how difficult (basically impossible at any non-trivial scale) it is to "understand" the outputs of a deep neural network compared to more classical methods like decisions trees or basic regression. This has only gotten more extreme as things have gotten more complex, abstract, and large.by nater5000
- I took a "Deep Learning" CS class in college back when it was in its early stages. I doubt the field is still called that now, but it was the subset of ML that has been rebranded as AI; includes LLMs, image generation, image recognition, etc.
Like any class, it was confusing at first, but when I eventually grasped the math behind what we were doing, and of course the visual representations of different elements to show lots of iterations of this math, it grounded the science for me, and I would hardly say people don't know what is going on. It only began to feel that way when it got a ton of hype and people jumping on the bandwagon who truly didn't understand it were trying to explain it to others, not to mention all the SOTA models put great effort into ensuring their methodologies stay trade secrets, going as far as effectively trying to ban people from learning the math by lobbying for the outlaw of open models.
Granted, "AI" has gotten way better than it was when I took that class, but the principles are the same, with different tooling and additional filters and algorithms thrown in there, as well as letting it determine the most appropriate statistically viable path forward for a particular prompt.
by eks391 - > This is how I make sense of AI reasoning. LRMs, chains of thought, thinking tokens: It’s wishful mnemonics all the way down — a heady mix of shorthand and suspended disbelief, like Oprah-style “manifesting” (opens a new tab) with a computer science spin. This isn’t necessarily a dig; all novel research likely requires some version of this mindset just to get off the ground. It certainly doesn’t mean AI reasoning can’t or doesn’t work. But the “wishful” part seems to be as powerful as ever.
“We react to language in a way that is very anthropomorphizing. That’s just the way that we humans work,” Mitchell told me.
I can definitely confirm the last part. Every time I read the output of an LLM, I picture a person talking to me.
by baxtr - I agree that these concepts seem a little vague and hand-wavy, but this is a) no substitute and b) far vaguer and unsubstantiated.
- I think sensible legislation might require that commercial AI providers discourage anthropomorphisation by avoiding personal pronouns from chatbot interfaces.
"Hey, customer service chatbot, can you help me get a refund for my order?"
BAD: "Sure thing, I'll be happy to help you with that, I just need your order details..."
GOOD: "Yes, this computer system can start the refund process. Please enter your order number."
- An intuitive explanation for why reasoning tokens help is to remember that LLMs are just mathmatical functions f() that take in an input sequence x and produces the next token f(x). Without reasoning tokens, you require the function f() to immediately take you from x to the start of an output sequence that is a correct answer. With reasoning tokens, this is much relaxed, allowing for many repeated applications of f() to gradually steer you from the input sequence to the start of the correct output sequence.
It seems intuitive that continuing a correct output sequence is easier than the "discontinuity" of jumping from the input prompt to the output sequence.
by TGower - The article is about existing research suggesting that this intuition is incorrect, though.by calf
- Obviously likely corollary: text diffusion models don't need reasoning tokensby inigyou
- Indeed, and maybe that's all there is to it. Still, I'd hope we will eventually better understand what's exactly happening in the wake of many repeated applications of f().
- CoT evidently helps but a bias towards both correctness and innovation has to come from somewhere. The article implies OpenAI's proofs may be supported by Lean but regardless, who knows how many people are trying to disprove the next conjecture each day throwing away nonanswers. Fundamentally these systems are more powerful with better training and sampling methods, or better prompting. Tokens matter but you can rewrite many prompts to get a much better, faster answer using fewer tokens vs allowing "thinking" to go on and on.by woopsn
- The way I think about it is that it's unreasonable for a compute graph with a static number of operations to be able to answer both y=a*10 and something like y=((((x+x)*(x+1))/((2*x)+2))+((x*(x+3))/(x+3))-((x*x)/(x+1))+((x*x)/(x+1))-((x*(x+3))/(x+3))) in a single forward pass. Tokens are essentially a unit of work and can also be used for intermediate steps, not just final results.by nodja
- Not only that! LLM can also learn how to repeatedly apply f() for certain arguments, and run the shortcut. Sometimes, the shortcut learned is not actually repeated application of f(), which breaks semantic soundness of LLM's reasoning chain.
These shortcuts can greatly help reasoning, but they are not guaranteed to be sound. So yeah, in that case, LLMs are reasoning right (using shortcuts) for the wrong reasons (learning them from data instead by running actual repeated application and memoizing the resulting rule).
by js8 - There is a long history of bad naming conventions in the field of AI, including “artificial intelligence” itself imo. (What is “intelligence” here? It’s more like “automation” or “automated problem solving”.)
What really happens is that we figure out something that works, sometimes inspired by some biological thing or neuroscience thing.
Examples: neural network, attention, reasoning, hallucination, agents, experts in “mixture of experts”.
And then we go to name it, and rather than reaching for some three letter acronym we sometimes borrow a more catchy term.
I almost never means the original research was confused about what is going on. And in some cases we eventually strip away things from the original, like in neural nets which used to have a more biologically inspired activation function but we found out that ReLU works just as well because the important thing was the non linearity not the sigmoid.
by janalsncm - AI = cognitive automation?by storus
- > There is a long history of bad naming conventions in the field of AI, including “artificial intelligence” itself imo.
It's all marketing. AI companies repurpose the terms of AI researchers to exaggerate their products. The public has an idea of what AI means and the companies exploit it. AI, in public perception, is what the Terminator movies are about.
It's exactly like all the "hover boards" you can buy.
by parineum - Automated problem solving... of specifically the kind of problems we'd normally need human intelligence to solve (coding, translation, etc.). So "automated intelligence" would make sense to me.by esperent
- There are two things occurring here. One is the AI effect, once something mysterious becomes creatable it is not intelligent at all.
The other isn't a problem of the field of AI, but the total fields of intelligence together. There is no formal agreed upon definition of what intelligence actually is across disciplines. It would be slightly hypocritical to point out the field of AI on this, when biology and psychology are just as guilty.
The more we look, the more lumpy intelligence seems to get. We find some species have min/maxxed portions of intelligence that we have to question what general intelligence actually is. It also should give us pause that some less generally intelligent agent can cause massive problems by maxing out the right stats.
by pixl97 - Transformers lack recursion and are limited by the network's fixed depth, so "reasoning", IMHO, is basically a way to emulate deeper recursion. As we go through the layers, concepts are pattern-matched and refined, but at some point we have to stop and cannot refine them any further (no more layers). Usually, this refinement continues during the generation of the next token (the previous intermediate results needed to continue the refinement are still in the KV cache).
But some problems require a substantial number of pattern-matching and refinement steps. The problem is, we also have interference from the fact that the model is trained to model language using mostly non-reasoning data of varying semantic lengths. Because of that, it may stop generating text before the abstract refinements are fully completed, simply because the pretraining data tells it to.
So we have to additionally train models to produce "reasoning traces" so that the emulated recursion continues for longer than what is typically found in pretraining data, allowing the model to build richer and more complex abstractions and surface more concepts. The ability to split problems into steps and logically connect concepts is already present in non-reasoning models (the original CoT trick), because some of it exists in the pretraining data, but not enough to support much longer recursion (hence the premature stops).
As for whether it is "true reasoning" or not, I think that is just arguing semantics for the sake of it. LLMs can demonstrably solve various complex problems. Yes, they often make stupid mistakes, but don't we have the saying, errare humanum est? Don't humans make mistakes too? Don't we also have around 200 cognitive biases showing that we "simply pattern-match" too? I think we still cannot get rid of the Great Chain of Being idea.
by kgeist
Never understood this argument.Don't humans make mistakes too?Humans get simple multiplication wrong often, so ok for computers to make mistakes multiplying numbers?
- LLMs lack qualia, among other things.
If I ask an LLM "what is an apple?" it tells me:
> An apple is the edible fruit of the apple tree, scientifically known as Malus domestica. It is one of the world's most widely grown fruits and is eaten fresh or used in many foods and drinks.
If I ask an LLM "what is a mundu fruit?" it tells me:
> Mundu is a tropical fruit native to Southeast Asia, especially found in Indonesia, Malaysia, Thailand, and Cambodia. It comes from a small evergreen tree in the same genus as mangosteen.
I've never eaten a mundu fruit. To me, an apple and a mundu fruit are categorically different. An apple is a fruit that I've held, touched, tasted, eaten, enjoyed, cooked with. A mundu fruit is an abstract experience: text, images, only slightly more real than a fictional fruit. I'm aware that mundu fruit exist, just as the LLM is aware the apples exist, but that doesn't make them exist for me.
"Existing in an abstract way" is how an LLM experiences everything. To an LLM, an apple and a mundu fruit are in the same category. The LLM has been trained on text about both fruit, it's seen images of both fruit, it knows everything that has been recorded about both fruit ...except everything that's important to know about a fruit.
Many of our issues with LLMs arise because from the LLM's perspective, nothing exists. If Claude accidentally deletes your production database, it may well apologize afterward, but only because an apology is statistically likely. It doesn't feel guilt like a human would, and the lack of consequences makes any action an LLM takes inherently frivolous. We want them to understand what's real and what's not, but without any lived experience perhaps that's an unreasonable expectation.
by zarzavat - Humans also lack qualia.by inigyou
- > We want them to understand what's
We want them to assess what's true and what's not.
by mdp2021