Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • 1. Sampling parameter deprecation (temperature, top_p, top_k)

    temperature, top_p, and top_k are deprecated and ignored. In future model generations, supplying these parameters returns an HTTP 400 error. Remove these parameters from all requests.

    by greatgib
  • fwiw sonnet-5 also drops temperature (sonne-4 had it)
    by tough
  • The API endpoints are less useful if you can't even fine-tune these parameters. These endpoints used to be developer oriented and this sort of change would be a huge cause of concern. If you primarily vibe-code, this is not a problem but understand that they're making their product worse on purpose.
    by kittikitti
  • Same thing as the new APIs like assistant API that rip us of the control of the history of messages for the context that is now hidden server side.
    by greatgib
  • > To improve determinism, define a system instruction with explicit rules for your specific use case.

    Is this guaranteed to work any better than top_k or top_p? This just sounds like making a smaller version of a Agent.md doc.

    by tolugenius
  • It is guaranteed to work worse than top_k=1, that’s for sure.
    by janalsncm
  • "Last" or "latest"? Those are rather different.
    by nichohel
  • The OP does not appear to be a native English speaker. And, for example, languages like Spanish don't clearly differentiate between "last" and "latest".
    by david-gpu
  • hehe... fwiw, the un-editorialized title is "Using the *latest" Gemini models"

    I speculate OP wanted to put focus on their chosen detail in the title...

    by froh
  • Obligatory "The Conspiracy Against High Temperature Sampling":

    https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...

    by kouteiheika
  • where can one learn what top_k and top_p mean?
    by NooneAtAll3
  • wow I wish I knew this post before I installed various UIs with tons of temp/top-X sliders.

    They are extremely confusing.

    by est
  • It's not surprising since these are all reasoning models now. Token sampling/decoding isn't exposed because the reasoning models are trained around specific configurations. That's why it's been replaced with "reasoning effort" instead (low, medium, high "thinking" levels).
    by marcinjachymiak
  • I'm curious: If someone wanted to serve models off hardware/silicon directly (like Cerebras or Taalas, and soon Google I think) rather than GPUs, would these parameters still be adjustable at request time? Or would they have to decide that before the model can even start serving and it would be locked in until they reload it (which would make it briefly unavailable)?
    by herpderperator
  • I'd expect that sampling would happen in software. Probably the hardware system would output a vector of probabilities over the tokenspace, just as the nets do when run in software.

    I don't know though and am not aware of any docs going into detail here. That being said, sampling is really cheap. So implementing it in hardware wouldn't be worth it.

    by StrauXX
  • > To improve determinism, define a system instruction with explicit rules for your specific use case.

    What if I want to do the other thing? When performing research with many sub agents, having a lot of diversity in the hypotheses is a big deal. If my 5 parallel sub agents all produce the same conclusion I might as well have only ran one.

    The latest OAI models have done the same thing. I'm currently adding random variation to prompts to compensate for the lack of higher temperature sampling.

    by bob1029
  • Current post-trained models are stochastically and semantically collapsed to a large extent, you'll never get meaningful semantic diversity if your prompt stays the same. This makes the temperature brittle, the model becomes dumb much earlier than meaningfully diverse. There are fancy samplers that claim to overcome this (comments ITT mentioned them), but a) you need a local inference stack to use them and b) I've yet to see any definitive evidence they add any deep semantic diversity without dumbing the model down, most of what I've seen is pretty superficial, including distribution-aware samplers.

    >I'm currently adding random variation to prompts

    This is better than naive temperature tweaks, if done right.

    by orbital-decay
  • My guess is that RL training being done with particular generation parameters makes models much more brittle to changes in these parameters, and that's why we're seeing changes like this across model providers. But I don't really know.
    by aesthesia
  • I'm inclined to agree given how unstable Gemma 4 is when not using the "official" sampler settings
    by pixelmelt
  • > To improve determinism, define a system instruction with explicit rules for your specific use case.

    "Please be deterministic".

    by franky47
  • please oh god please, I don't know how to
    by rf15
  • In all seriousness, this seems like it could be fine if done well. You can just have a model do a pass over the system prompt and set reasonable parameters based on that. That's probably not what they're doing, but it could be.
    by Zambyte
  • > improve determinism

    Oh well. I might be to picky here, but how I see things, determinism cannot be improved or worsened, but achieved or not achieved. Or Partially archieved, when analyzing a system that has both components that are deterministic or non deterministic.

    There are times to think in absolutes, and when talking about deterministic behavior of technical systems, this is one of them. Join the sith side, we have cookies, and when we say we have cookies, we do.

    by dev_dan_2
  • Possible reasons:

    - They might be dynamically adjusting these at inference time [1]. For example, start with a low temperature and generate samples with increasingly high temperatures until one of them passes some quality gate.

    - They don't want you to fine-tune on high temperature completions (rejection fine-tuning). You could call this "rejection fine-tuning rejection".

    [1] https://rlhfbook.com/c/09-rejection-sampling#related-best-of...

    by salamo
  • Couple of other more businessy reasons:

    - SynthID hides the watermark in the sampling RNG. No randomness -> no watermark.

    - If you want to distil on the model outputs, you want temp=0 outputs. No temp=0 -> worse distillation.

    by regularfry