Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- First author here, happy to answer questionsby flyingpumba
- "Study: Communing With The Gods of Mount Olympus Via the Oracle at Delphi Is Not Always Faithful"by kibwen
- This paper made me wonder not whether the chain we can read is actual “thought,” but what conclusions we can draw by observing it. It is an output channel, but not direct access to the black box that creates it. The question pairs present an interesting experiment, but they also got me thinking about semantics: the same underlying relation can have many valid representations, and how models reason across those representations can tell us more about their stability and correctness. Basically, are models semantically consistent when given different representations of the same underlying relation? Do their answers transform as the relationship requires, and do their explanations remain consistent with that relationship?
- I thought this was already widely known?
From March last year: https://transformer-circuits.pub/2025/attribution-graphs/bio...
There's no reason to believe the model's self-reported "thinking" bears any relation to the mechanics by which it arrived at some output.
- that sounds testable - if you skip the reasoning tokens, do you get the same result?
if not, then there's certainly some bearing, but not necessarily in how we read the tokens as text
by 8note - It's... complicated. Yes, RL reward hacking makes it learn "bird language" and yes, reasoning traces can be misleading. However they also pretty clearly steer the final reply and not simply justify it, and can stay somewhat coherent and relevant with readability SFT and rewards. All these phenomenas coexist, they aren't mutually exclusive. Reasoning traces are still useful for debugging.
- I see this when just using some chat bot that shows the “reasoning” steps (ad-hoc observation of course, it’s really cool that people are actually studying it).
It is annoying when the bot seems be “reasoning” correctly and then makes an obvious mistake at the end. And perplexing when it seems to be completely wrong and then pull the right answer out of a magic hat at the end.
I guess it makes sense; the “reasoning” steps aren’t actually doing logic, just adding more context to influence the final generation, right? But it is weird to see.
by bee_rider - This paper puts words to something I’ve noticed repeatedly with LLMs, particularly Qwen3.6. When I read its reasoning, it appears to recognise the mistake and then carry on as if it hadn’t noticed it at all.
> models often determine their answers based on implicit biases tied to question templates, then construct reasoning chains to justify their predetermined conclusions > its reasoning was correct right until the final step (Yes/No answer)
- Natural intelligences do this too
- First author here, surprised to see the paper in HN! :)
When doing the paper we noticed that models are very good at generating post-hoc plausible CoT, which to me knowledge can happen quite often with relatively easy tasks.
You might be interested in reading this other paper that came out after ours: https://arxiv.org/abs/2507.05246
by flyingpumba - Of course not. Because the article uses the words 'thought' and 'reasoning' and even 'faithful' to mean something other than their normal meanings, but then expects them to behave exactly the same.
Every field has terms of art, and 'reasoning' is one for LLMs. But that doesn't mean it has the same properties as 'reasoning' in other contexts, because you're not referring to the same thing.
Why doesn't my asteroid belt buckle?
by Planktonne - Over time, I've learned to accept that many people -- even very clever ones -- are incapable of holding a metaphor at arm's length. Once they accept the words of a metaphor as applicable at all, the metaphor collapses entirely into literalism for them. They can no longer see that the metaphor was just a tool with inherent limitatation and boundaries.
Because the field of artificial "intelligence" is constructed around the idea of applying psychological metaphors to computational systems (a very powerful idea!) it's almost a worst case scenario for these people.
Suddenly, they're reversing the metaphors and applying computational schema to psychological processes ("aren't we really just stochastic parrots ourselves?!"); or, like here, they find themselves surprised and confused when they stumble across the natural boundaries of the metaphor experimentally.
It's because they never had sight of the boundaries in the first place and maybe never can quite see them. The words only make sense to them as literal equivalence, and so their surprise when they run into stuff like this is earnest and deep.
by swatcoder - Related: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces![1]
> Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks.
by stymaar