Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- This article does not make its case. He shows a graph of "random" data, and then just kind of keeps going. But that random data is the meat of the whole thing.
Cut out 60% of the useless text, and focus on explaining why random data should look like that.
by ars - Agreed. Also it seems like, if the actual test scores and perceived test scores were both sampled iid, the “perceived” line should just be flat, with everyone being at 50th percentile. The fact that the displayed graph deviates from 50 in a way that more resembles the empirical data makes it seem like a cherry-picked random sampleby mobeets
- Man sets out to write article on DK Effect, comes across one skeptical study, confidently writes article refuting DK Effect.
It writes itself.
- Weird take. The author corresponded with Dunning and reproduced the classic graph using random data.by sa46
- The author could have a promising career at The Onion.by pryelluw
- The strict academic definition hasn’t followed the colloquial usage for a long time. Maybe ever:
If a specific novice is over-confident and out of their depth, we say “Dunning-Kruger”
If a specific is under-confident and performing better than their self-estimate, that’s not commonly considered Dunning Kruger, in the colloquial use. It’s called imposter syndrome, or not labeled at all.
The researchers aren’t really disagreeing with that. They found that novices had a wider range of self-estimates of their performance than experienced people. So in the novice group you were more likely to find someone who was grossly over-confident in their abilities, but you also found people who underestimated themselves.
> instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
Which doesn’t precisely contradict the idea that among novices you can find people who overestimate their skills. Which is how it’s commonly used.
So I can believe it’s a statistical wash when averaging across all subjects. But I never considered the common use of Dunning-Kruger to be applied to averaged groups of people. It was always brought out for those outliers on the long tail of the novice grout who thought didn’t even know what they didn’t know.
by Aurornis - You've hit the nail on the head, it's a conditional defenition, rather than an absolute one.by Tactical45
- > The strict academic definition hasn’t followed the colloquial usage for a long time. Maybe ever:
This is my understanding as well, and as I recall the results of the research were also more nuanced than most people seem to indicate. So not only were the novices with a wider range of self-estimates, the average result of those perceived scores was still below the experts. So it was never that low performers thought they were experts (although this may show up in the raw results with some portion of the population tests), just that on average they perceived their performance to be better than it actually was. And the high competence group while overestimating their results, still thought they'd do better than the low performers.
by kevin_nisbet - I think people overlook the natural experience of Dunning-Kruger we all have.
You think you are good at chess because you win against your neighbour, then eventually a class mate absolutely embarrasses you multiple times and you realize there is a whole world you don't know, and before now you didn't know that world existed. Then you eventually beat your classmate before getting whomped by the beginner in the chess club and the whole cycle happens again. After this happens a few times, you should begin to find each time less surprising than the last.
To me this is the essence of DK and not really something to be proven/disproven by maths. You don't know what you don't know, and it takes some learning before you are wise enough to realize there are unknown unknowns.
> instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
I think this is trying too hard on the researcher's part... if the data were reinterpreted such that small estimation errors are considered correct then the effect would reappear.
- Interesting, maybe the fact that it is replicated by random noise is actually confirming the theory though. The article claims "completely random guessing replicates the DK graph". But that is compatible with "the ability to assess one's skill is not correlated with measured skill". If random guesses reproduce DK, it implies that people might be randomly guessing at their level of skill.
As a result people with little knowledge over-estimate their ability because a random dice roll will tend to be higher than their low ability (and the converse for people of high ability).
If you look at the DK graph, its close to random, maybe a very slight positive slope -- but ultimately the DK theory would be even stronger if it was truly random, because it means people have NO ability to assess their skill (0 correlation). I think the claim in DK is slightly weaker, that people at least have some ability, just not as much as we like to think.
by program_whiz - So your claim is that if there is no correlation between A and B then it proves that there is in fact negative correlation between A and B because there is no correlation?
And so the Dunning Kruger effect becomes true of all things that are uncorrelated?
by klustregrif - The chart they show from the original paper is approximately just:
(With the caveat that "ability" is a percentile rather than some absolute measure.)perceived_ability = 0.5 + (actual_ability / 4)That's not the negative correlation that it's often presented as in discussions. And it could as easily relate to mis-measuring everyone else's skill as to mis-measuring your own skill.
by unwise-exe - Very hard to understand the meat behind all the fluff of the article, especially as the simulation code is not available, and as the presented simulated and original graphs are effectively the same (I don't see a disagreement).
It's clear that the perceived curve will be differently sloped, as no one will evaluate themselves as the topmost or the bottommost percentiles, so the edges will be biased.
And if in both cases we draw differences between perceived and actual, we will get the same curve that everyone knows, biased or not.
by 5555watch - It's annoying they didn't plot the new and old "perceived" curves on the same figure, but if you pay attention to the y-axis, there is a very big difference.
In the old plot, the bottom quartile has about a 50 percentage point margin between actual and perceived performance while the new one is 30 percentage points, which is a 50% difference between the old and new curve. The second quartile has 3x more margin in the old version relative to the new one.
by jszymborski - Huh? The point is that the two graphs come out looking the same, making the original no more meaningful than random.
Source code is here: https://github.com/pem725/Dunning-Kruger (found here: https://pem725.github.io)
by card_zero - A similar argument was made a couple of years ago, here's the rebuttal of the argument from back then [2022]: https://andersource.dev/2022/04/19/dk-autocorrelation.htmlby renlo
- It's an interesting read. Curiously, it doesn't really debunk anything.
The fact that given X and Y random and independent, that Y-X is correlated with X doesn't disprove the Dunning Kruger. It in fact proves that Y = 1 X is a poor predictor, and the true model is Y = 0 X. In other words, perceived ability (of the human) cannot predict the actual test scores. Which is exactly what DK claims, but to a very extreme effect.
Note that, if there is actual signal (plus noise), e.g., if Y = X + eps; so the actual score is exactly the perceived score plus some added variation, the (Y-X)~X will be uncorrelated. In such case, there will be no DK effect, because the users are good at predicting their actual test scores, plus some constant variation.
by 5555watch - Damn, I remember I worked through the whole damn thing some years ago when I read this article for the first time, and it didn't hold up to it's claim. I was actually contemplating to do a write-up, but it wasn't really important to me so I didn't. Seeing the same article re-appear on HN for the second time now I really wish that I did. Unfortunately I've forgotten the details by now.by _0ffh
- If you ever do, post it! I'm sure it would be an interesting readby audreyfei
- The reason people find the DK effect persuasive is that everyone has run into this many times in their own lives. Like with an idiot boss who thinks he knows your job better than you do, or an article by someone that gets things completely wrong about a topic that you yourself know very well, through personal experience or extensive study. And we have all been dismayed about the impact such people can have, like through their decisions or influencing each other.
If the research can't demonstrate the DK effect, then the research is bad, like poorly designed.
- Confirmation bias. You don't notice when people back down from their bad takes because that's normal and expected. Or possibly just selection bias if you are in situations where people are more likely to be arrogant.by namlem
- Your anecdote is not describing dunning-kruger, though, it's describing "some people exist in society with too much confidence in their ability" which can be true on its own, and is not exclusive to beginners.by usef-
- >If the research can't demonstrate the DK effect, then the research is bad, like poorly designed.
Why do the research if you are so convinced from your own personal anecdotes? Quite a weird thing to say...
by qsera - >If the research can't demonstrate the DK effect, then the research is bad, like poorly designed.
I don't have an opinion about Dunning-Kruger either way, but I wanted to point out that this is a pre-replication crisis mindset. Social psychology spent decades producing catchy, intuitive results that turned out not to replicate. We've learned over and over that we can't trust it when a result feels right, we need to rely on proven statistical methods.
by fasterik - > If the research can't demonstrate the DK effect, then the research is bad, like poorly designed.
It’s the opposite problem. The research is bad because it “demonstrates the DK effect” even for cases where it cannot possibly be true. Like “people who tend to roll lower dice rolls overestimate their own ability to roll high dice values and people who tend to roll higher values tend to underestimate their own ability”
Which is just obviously incorrect and the true result should have been “there’s no correlation between people’s estimate of their ability to roll dice and their actual outcomes”
If you want to demonstrate an actual effect like DK, you have to set up an experiment such the if it truly doesn’t exist then the experiment tells you so. A great way of checking this is to throw random data at it in which case it should come out saying “I saw no correlation” but that’s no the case with their setup.
by klustregrif - Maybe the problem is that DK isn't about individuals, but about topics? I.e. perhaps everyone exhibits the phenomenon about topics they are most ignorant about.
(and I am ignorant about DK, so...)
by rootusrootus - Related: The Gell-Mann Amnesia Effect. In Michael Crichton's own words:
"Briefly stated, the Gell-Mann Amnesia effect works as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward-reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them. In any case, you read with exasperation or amusement the multiple errors in a story-and then turn the page to national or international affairs, and read with renewed interest as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know."
by jeffnash - We want theories to be backed by more rigorous data precisely because "having run into this many times in their own lives" is not reliable. People's own perception is incredibly biased. We can see a bias at play in your very own closing sentence — "your study is bad if it doesn’t not confirm my existing belief" is plainly not science.by pibaker
- Replication crisis. More than half of all psychology studies are not reproducible.
I'm at the point honestly, where I don't even consider psychology to be a science anymore.
by datakan - If you ever took psych 101, it was immediately obvious it's mostly horse shit. Made up nonsense theory, with studies that have few participants and are done by people who don't know math.