Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- LLMs work by generating the most likely continuation to a prompt. But they can also generate multiple likely continuations. This create multiple branches which in turn can generate even more branches. The LLM can then evaluate the branches, prune the unpromising ones, and merge the best ones. More branches means more tokens, means more effort.by bjourne
- It's usually implemented as a keyword the model is trained to react to. In API-only models like Claude you don't have the access to it, as it's inserted into a masked area. The reason the cache might break is that it's inserted at the start, and if you change it the rest will be invalidated.
- I thought it's a loop really... I may be wrong. It's going through a process of thinking, evaluating, outputting, grading, and if grade was low, then loop from thinking again.by inthepond
- I suspect it's not possible (as an end user) to get a thinking trace from one of the models. But what happens with "thinking" is that the model has a conversation with itself in an attempt to home in on a better answer to the original prompt.
The "amount of thinking" is how long this internal conversation is allowed to progress. The longer it goes on the more it costs. It's all part of the token budget but, because this internal dialogue is hidden, it's not obvious to the end user.
by masfuerte - Different models do slight variants.
Usually it’s done in post training to enforce behavior based on prompt. Ie. System prompt with thinking:max or low or wtv.
Enforcement then goes via constrained decoding, checking for think token start and end with max lengths, or other variations
by aabdi - For Deepseek V4, The main effect of raising the reasoning effort is to add a little section[1] to the end of the system prompt that says "BTW make sure to think really hard! :)"
If Anthropic's models work the same way, then changing reasoning effort would break the cache because the API has to modify the system prompt given at the very start of the context and rerun the whole thing through the inference server.
This kind of limitation is one reason Opus 4.8's mid-conversation system messages[2] are actually a pretty big deal (if they actually work).
[1] https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/ma...
[2] https://platform.claude.com/docs/en/build-with-claude/mid-co...
by Centigonal - Take a look at the harmony repo which specifies the internal OpenAI format - the effort level is specified in the context after the <|start|> tag - https://github.com/openai/harmony
Note that inference libs also have parsers that put hard limits on reasoning tokens with separate counters (similar to how you can put a limit on token generation per completion versus waiting for an <eos>). For that, take a look at vllm reasoning docs.
by pyentropy - Don't work at a lab but I think they might be warping the probability distribution in the decoding step, at least to generate RL examples for training and maybe in production too.
There aren't other comments discussing this possibility at the moment, but you don't have to take the token predicted as most likely (greedy decoding). Most decoding strategies do something else which is where settings like temperature come in. So if you want the model to "think harder" you can track whether the current tokens are thinking or answer - in OpenAI's system that's called a channel - and then if you're in a thinking block you might get a model output whose top three predictions are:
Greedy decoding would stop thinking at this point and start answering, but you want the model to keep thinking so you skip that token and select the next most likely which is "Wait, ". The reasoning levels can map to the probability of skipping the channel change tokens.60% <|channel=answer|> 10% Wait, 5% . The [...]by mike_hearn