Comments
Hacker News
"Last" or "latest"? Those are rather different.
by nichohel
Obligatory "The Conspiracy Against High Temperature Sampling":
https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
by kouteiheika
It's not surprising since these are all reasoning models now. Token sampling/decoding isn't exposed because the reasoning models are trained around specific configurations. That's why it's been replaced with "reasoning effort" instead (low, medium, high "thinking" levels).
I'm curious: If someone wanted to serve models off hardware/silicon directly (like Cerebras or Taalas, and soon Google I think) rather than GPUs, would these parameters still be adjustable at request time? Or would they have to decide that before the model can even start serving and it would be locked in until they reload it (which would make it briefly unavailable)?
> To improve determinism, define a system instruction with explicit rules for your specific use case.
What if I want to do the other thing? When performing research with many sub agents, having a lot of diversity in the hypotheses is a big deal. If my 5 parallel sub agents all produce the same conclusion I might as well have only ran one.
The latest OAI models have done the same thing. I'm currently adding random variation to prompts to compensate for the lack of higher temperature sampling.
by bob1029
My guess is that RL training being done with particular generation parameters makes models much more brittle to changes in these parameters, and that's why we're seeing changes like this across model providers. But I don't really know.
by aesthesia
> To improve determinism, define a system instruction with explicit rules for your specific use case.
"Please be deterministic".
by franky47
Possible reasons:
- They might be dynamically adjusting these at inference time [1]. For example, start with a low temperature and generate samples with increasingly high temperatures until one of them passes some quality gate.
- They don't want you to fine-tune on high temperature completions (rejection fine-tuning). You could call this "rejection fine-tuning rejection".
[1] https://rlhfbook.com/c/09-rejection-sampling#related-best-of...
by salamo
Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- "Last" or "latest"? Those are rather different.by nichohel
- Obligatory "The Conspiracy Against High Temperature Sampling":
https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
by kouteiheika - It's not surprising since these are all reasoning models now. Token sampling/decoding isn't exposed because the reasoning models are trained around specific configurations. That's why it's been replaced with "reasoning effort" instead (low, medium, high "thinking" levels).
- I'm curious: If someone wanted to serve models off hardware/silicon directly (like Cerebras or Taalas, and soon Google I think) rather than GPUs, would these parameters still be adjustable at request time? Or would they have to decide that before the model can even start serving and it would be locked in until they reload it (which would make it briefly unavailable)?
- > To improve determinism, define a system instruction with explicit rules for your specific use case.
What if I want to do the other thing? When performing research with many sub agents, having a lot of diversity in the hypotheses is a big deal. If my 5 parallel sub agents all produce the same conclusion I might as well have only ran one.
The latest OAI models have done the same thing. I'm currently adding random variation to prompts to compensate for the lack of higher temperature sampling.
by bob1029 - My guess is that RL training being done with particular generation parameters makes models much more brittle to changes in these parameters, and that's why we're seeing changes like this across model providers. But I don't really know.by aesthesia
- > To improve determinism, define a system instruction with explicit rules for your specific use case.
"Please be deterministic".
by franky47 - Possible reasons:
- They might be dynamically adjusting these at inference time [1]. For example, start with a low temperature and generate samples with increasingly high temperatures until one of them passes some quality gate.
- They don't want you to fine-tune on high temperature completions (rejection fine-tuning). You could call this "rejection fine-tuning rejection".
[1] https://rlhfbook.com/c/09-rejection-sampling#related-best-of...
by salamo