Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • I was hoping the writer would follow the interesting framing of the quality of an LLM's confidence score with some data on how good it is, rather than just proclaiming it's bad because of sycophancy tendencies. If someone collected the data on something measurable, I would be willing to bet the LLM's confidence number isn't too far off.
    by jrflo
  • Isn’t this what jev training is supposed to improve? They don’t train on predicting the next token but predicting the distribution of next tokens, or something like that.
    by m101
  • In my own words - you can't calibrate confidence of a model without a calibration set. Confidence is relative to some labeled dataset, it is established on a set of buckets or intervals, and is population based, something you can't check for a single prediction. Assign per case, verified per population.

    I thought the way Jev was presented was sneaky - you can't say "we calibrated the model for you" and skip having a task. specific calibration set. They couldn't have known all our future tasks.

    by visarga
  • Confidence scoring is fine when you provide the right context, like everything else LLM related. I use grading scripts for things that can be measured directly so the LLM just operates on it. For things less well-defined, I use a grading matrix with explicit criteria on what constitutes an A to F grade, usually with an assumption of an A grade that drops depending on specific findings.

    If you are asking about confidence and somehow it can’t be defined either mathematically or in a grading rubric, then you the implementer need to start asking yourself the questions, because then you got a target that you don’t even know how to measure. The LLM isn’t even the problem at this point because another human would also just be going off gut feeling and vibing a number that may or may not be accurate.

    by cityofdelusion
  • It has always seemed to me crazy (and maybe a bit lazy) that people will just blindly use self-generated confidence scores from LLMs without having some sort of validation mechanism.

    Also, as another poster said: decomposing your rating into as many dimension as possible is better way to generate some confidence score grounded in fact.

    by Otterly99
  • If you found this interesting, you should read section 5 of this article to learn about how LLMs can be trained so that they're good at confidence scoring: https://magazine.sebastianraschka.com/i/217935364/51-on-cali...

    OP says that LLMs were never scored against correctness. But that's an assumption about the optimization process.

    This is sort of right:

    - pretraining scores on next token, and says nothing about confidence

    - RLHF and RLVR scores are binary (1=preferred/correct, 0=worse/incorrect) and the feedback doesn't do anything to push the weights in a direction that improves calibration/confidence

    BUT you can ask the model to output its confidence, and then train it to be more correct. RLCR (mentioned in the article I linked above) adds a Brier penalty to the reward, so the model's performance isn't just measured on % correctness. (Presumably TypeSafeAI's RLCD is similar to RLCR, but they haven't published anything about it.)

    This eliminates the "always say 30%" loophole OP mentioned. The Brier penalty, (q-c)^2, punishes a forecaster who ignores instance-level information. For example if we have a model that's right 60% of the time, it could use one of these strategies to give its confidence:

      Strategy A, always say 0.6:
  When right (60% of the time): (0.6-1)^2 = 0.16
  When wrong (40% of the time): (0.6-0)^2 = 0.36
  Expected penalty: 0.6×0.16 + 0.4×0.36 = 0.24

  Strategy B: the model can tell its answers apart. Half are 'easy' (right 90% of the time) and half are 'hard' (right 30% of the time), which still averages to 60%. It says 0.9 on easy ones and 0.3 on hard ones:
  Easy: 0.9×(0.1)^2 + 0.1×(0.9)^2 = 0.09
  Hard: 0.3×(0.7)^2 + 0.7×(0.3)^2 = 0.21
  Expected penalty: (0.09+0.21)/2 = 0.15
    Both strategies are perfectly calibrated (60% is both the average confidence and thje average % of time it's right), but B gets a lower penalty. Of course these are not actual strategies used by LLMs. But I'm just trying to say why training can push the model toward optimizing instance-specific confidence.
    by rahimnathwani
  • I got a degree in statistics, and what I remember from those times is that we spent maybe 10% of our time learning how to build the model and 90% of the time correctly interpreting the model output. Of course, in the time I learned statistics it was almost exclusively applied in hard science, finance, and actuary. The wild west of machine learning hadn't gotten started. I wonder if they teach it differently now.

    When we start using big datasets where n went into the tens of millions we kinda stopped bothering because margins were so small it was effectively certain you found the mean of some range. However I think in that time we forget what confidence scoring actually is and how quickly multiplied alpha can balloon on you.

    by madrox
  • Of course an LLM does have a latent concept of its own confidence that is internally consistent, and is perfectly able to articulate it, the author just chose the worst possible way to ask. Using a 0-100 scale for anything is a bad idea with both LLMs and people. Instead of rating it 0-100, split your problem into many dimensions/axes and give each a small and meaningful enum or just a less granular numerical scale (ideally just a binary choice). Then apply weighted scoring to the result. You'll be surprised how much better it becomes at meaningfully expressing its confidence this way.

    YouTube removed the 1-5 scale for the same reason, people have no idea what to do with it.

    by orbital-decay

Explore Birbla archives