Birbla

Comments

Hacker News

I don't really get why you need handoff if your score is accurate. If it is, and it is low for a given response, just let the harness re-run the prompt with a different seed until the score is high enough. If this approach doesn't work, your score is most likely garbage.

by maxgashkov

Have you benched this for coding tasks, with a fallback to a larger local model, for example Qwen-3.6-27B?

Or using it for sub-tasks, where a framework with a larger primary model dispatches simpler jobs ("summarize ...", etc.) to it?

by zdw

> So we did mechanistic studies on small models, Gemma 4 particularly, and found the hidden state for different layers carry meaningful self-awareness signal for various situations.

Neat! Just to make sure I understand - you trained your probe layer to take this hidden state and predict p(wrong)?

Curious to learn more. Any more info on your approach (esp the mechanistic study)?

by cacio-e-pepe

> the hidden state for different layers carry meaningful self-awareness signal for various situations.

Is it plausible to wonder if some developer judgement feels, like maybe "the code I just wrote is clean/crufty", or "things came together smoothly/janky", might have extractable signals in some models?

If so, might one create a shopping list of desired signals to check for in a model, as with activation steering concepts, where one checks whether and how hard each concept can usefully be nudged?

by mncharity

Cool idea, some feedback:

1. Consider using conformal prediction to calibrate the cutoff. Conformal prediction provides a distribution-free guarantee under exchangeability. This would let you turn your raw probe score into a threshold with a guaranteed bound on the rate of wrongly-kept on-device answers. Source: https://en.wikipedia.org/wiki/Conformal_prediction

2. The best indicators of confidence in ML come from multiple independent methods. What was the result if you combine the token entropy and verbal confidence reporting methods? Does this improve the result?

3. I noticed you didn't mention the assessment method of rerunning the model and judging whether outputs are consistent. How does that method compare in terms of AUROC?

by dmrivers

> "post-trained to know when it's wrong"

Is it also post-trained to know when it's wrong about when it's wrong?

> "Every response comes with a confidence score between 0 and 1"

How confident is it in its confidence?

Please, I'm sure that what you're doing is very neat and useful, but use other language to describe it. I beg you. You can't know when you're wrong. You can only know when you're unsure or inconsistent. You can be absolutely certain and still wrong and uncertain and still correct.

by BugsJustFindMe

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • I don't really get why you need handoff if your score is accurate. If it is, and it is low for a given response, just let the harness re-run the prompt with a different seed until the score is high enough. If this approach doesn't work, your score is most likely garbage.
    by maxgashkov
  • Have you benched this for coding tasks, with a fallback to a larger local model, for example Qwen-3.6-27B?

    Or using it for sub-tasks, where a framework with a larger primary model dispatches simpler jobs ("summarize ...", etc.) to it?

    by zdw
  • Was actually pulling on a similar thread as I saw announcement so I integrated it just for fun. Have been only running this on my Framework Desktop but should be runnable elsewhere https://github.com/olafura/gemma-4-mic-transcribe
    by olafura
  • > So we did mechanistic studies on small models, Gemma 4 particularly, and found the hidden state for different layers carry meaningful self-awareness signal for various situations.

    Neat! Just to make sure I understand - you trained your probe layer to take this hidden state and predict p(wrong)?

    Curious to learn more. Any more info on your approach (esp the mechanistic study)?

    by cacio-e-pepe
  • Is this in any way similar to Goodfire's work? https://www.goodfire.ai/research/rlfr#
    by astrobiased
  • > the hidden state for different layers carry meaningful self-awareness signal for various situations.

    Is it plausible to wonder if some developer judgement feels, like maybe "the code I just wrote is clean/crufty", or "things came together smoothly/janky", might have extractable signals in some models?

    If so, might one create a shopping list of desired signals to check for in a model, as with activation steering concepts, where one checks whether and how hard each concept can usefully be nudged?

    by mncharity
  • Cool idea, some feedback:

    1. Consider using conformal prediction to calibrate the cutoff. Conformal prediction provides a distribution-free guarantee under exchangeability. This would let you turn your raw probe score into a threshold with a guaranteed bound on the rate of wrongly-kept on-device answers. Source: https://en.wikipedia.org/wiki/Conformal_prediction

    2. The best indicators of confidence in ML come from multiple independent methods. What was the result if you combine the token entropy and verbal confidence reporting methods? Does this improve the result?

    3. I noticed you didn't mention the assessment method of rerunning the model and judging whether outputs are consistent. How does that method compare in terms of AUROC?

    by dmrivers
  • > "post-trained to know when it's wrong"

    Is it also post-trained to know when it's wrong about when it's wrong?

    > "Every response comes with a confidence score between 0 and 1"

    How confident is it in its confidence?

    Please, I'm sure that what you're doing is very neat and useful, but use other language to describe it. I beg you. You can't know when you're wrong. You can only know when you're unsure or inconsistent. You can be absolutely certain and still wrong and uncertain and still correct.

    by BugsJustFindMe