Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • There is such a smell around this whole literature.

    I wonder what it's like to be a grad student in social psychology right now. When the norm is surprising results, any responsible, competent researcher is going to realize that they can't do it honestly. So this leaves behind people who are willing to bend the rules, or people who don't see the problem with, for example, the garden of forking paths.

    by jsw97
  • In good universities replication crisis and the underlying structures are taught. (Un?)surprisingly, not much has changed though. Except for some unis, some journals, and a bubble of rigorous research continuing to work on meta-science and replications. Have to admire their stamina.

    Source: did my PhD on exactly this topic

    by neurotroph
  • It's shocking to me how badly the data was falsified. You'd think a researcher who is A. well versed in statistics and B. has seen a lot of actual research data would be able to produce more believable data. Does this mean we're only seeing the tip of the iceberg? how many papers are based on hard to detect falsifications?
    by LeChuck
  • Why bother? Low effort data falsification is enough to land a high-ranking paper. Sure, really simulating data with some noise would be much more convincing but its just not necessary. Raw data is very rarely inspected during peer review, much less so after publication, and most reviewers do not address statistics — if they do, the usually refer to their pet model and/or use the same incorrect tools as most psychologists.
    by neurotroph
  • I hope that with LLMs we'll see a lot faster identification of these anomalies. Imagine that this guy taught at MIT and has his own lab at Duke, basically more than 20 years of taking up some of the most prestigious positions in his field, all based on fraud.

    The butterfly effect this has on science as a practice is huge. Some marginal researcher who actually deserved it was bumped out of those places, and somewhere down the line someone was bumped all the way out of academia because of this guy's lies. And this is hardly the only person in this situation.

    What's scary is that the next fraudster will read this take down and see what they have to do to avoid detection. It wouldn't be that hard to form data that looks a lot more plausible. Replication may be the only solution, unfortunately it's so expensive and time consuming, without any of the actual benefits of doing the original study. Maybe it should be government funded, the US replication lab, its only mission to replicate the top X papers from some named fields each year.

    by zhivota
  • They will be generated faster than they can be detected.
    by qmmmur
  • Having skimmed this article and the replication study itself (which references some studies that complicate our common understanding of the positive effects of deadlines) I’m left wondering what the constructive alternative is to deadlines and/or how deadlines should be maintained or determined.
    by hnacct2001
  • There is no alternative to deadlines because time is money, and those downstream of you can ill afford to wait on your deliverables. Part of management's job is to determine how much time it should take so a reasonable timetable that stakeholders can agree to can be set, and then hold you accountable for delivering within that time frame.

    I wish we could revive Tim Bryce's mind with AI, because I'd love to hear what he'd have to say in a podcast about this topic:

    "Fraud in a procrastination study? You don't say! I don't understand why all this taxpayer money is going to study procrastination in the first place. Procrastination is a well-known phenomenon with a well-understood cause: laziness. Procrastinators didn't have a boot placed to their behinds growing up, so they reached adulthood slacking and coasting on their trust-fund money or couch-surfing at their friends' places, mooching off others. You see this a lot in modern software organizations, especially among the programmers. And now those programmers are all crying because they're about to get replaced by Claude Code. Boo hoo! The military knew how to deal with procrastinators, too: send them to scrub latrines with a toothbrush, and drum them out if they didn't shape up after that! We don't need millions of dollars in government grants going to academic research to study this phenomenon. It's as old as humanity itself—we called it the deadly sin of sloth. Friends, keep the faith!"

    by bitwize
  • One thing I didn't know before reading this: an unusually large effect size can itself be a red flag. You don't have to understand all the statistics to at least notice when a result looks almost too good to be true.
    by mulanroo
  • This is one of the things that did in Jan Hendrik Schön.

    I can't remember the exact measurement but he was showing a set of curves that were simply too neat for a relatively small sample size - roughly 100 or so plastic transistors. In reality you cannot get that neat an effect without a huge sample population

    by zero_shift
  • It is interesting how all these famous non-replicating psych paper results feel reasonably intuitive to me. Like I can buy that having a more structured and detailed deadline helps you procrastinate less. I can even buy that doing a goofy little "power pose" exercise kind of hypes yourself up to give some self confidence and do better on an interview or whatever.

    You would almost expect the opposite and that all the famous studies would be the ones that run counter to intuition. Maybe those get more replication attempts, or people feel less confident faking them.

    The direction is intuitive, but the effect sizes are, as you say, unusually large. I'm no statistician, but if you have an effect size that's several multiples of the standard deviation, that would mean that it dominates the natural variation in results right? So you're kind of making an implicit claim that other factors in play (for eg in this proofreading task, how busy someone is or their reading level) are peanuts in comparison.

    Further, if there are these relatively simple factors like how a deadline is expressed that make you several times better, you'd expect anybody who stumbles into several of them and stacks them together to be absolutely off the charts. Somebody who listens to Mozart, gets the right priming, doesn't deplete their ego, and receives short deadlines is suddenly like 50X more productive, which doesn't seem to match reality.

    by recursivecaveat
  • I'm glad the authors seem to have acknowledged the problem and not doing much about it :-)
    by keyle
  • Per the post, the original authors have requested that the original paper be retracted. That’s well above the bar of “not doing much”.
    by fn-mote
  • Dan Ariely is well past the point that he needs to not be a faculty member at Duke University. Data Coda do a fantastic job ferreting out academic fraud. With these fraudulent behavioral science experiments like this relying on printed papers/markups, and coauthors not present and who did not even see the raw data, I wonder if the experiment was ever actually conducted. I would love to see that question answered where fraud is suspected, something I'd hope IRBs would step up to do.
    by dramm
  • I came here for a procrastination joke comment and didn't find it, but I suppose someone will get around to it eventually.
    by herewulf
  • I’ll do it later
    by peterbonney
  • > I came here for a procrastination joke comment and didn't find it, but I suppose someone will get around to it eventually.

    Too soon; they'll all do it later.

    by lelanthran
  • Here's one. I always leave my procrastination for later.
    by toast0
  • No paper should ever be referenced until it has been replicated by an independent group. The replication crisis in science is real, and it won't take long for no one to believe anything anymore.
    by freediddy
  • > The replication crisis in science is real, and it won't take long for no one to believe anything anymore.

    If you think replication in the sciences is in crises, wait till you see the field of psychology.

    by lelanthran
  • In the physical sciences, any sufficiently important paper is independently verified. Sounds like cost/effort of these multi year studies is not faesible to replicate, but then again, I know nothing about social science so w/e
    by alansaber
  • >The replication crisis in science is real

    In psychology, you mean. Many contest if it even qualifies as a science anymore after the complete annihilation of trust it has undergone.

    by dartharva
  • That rule would make it difficult to report on a failure to replicate a published paper.
    by Polizeiposaune
  • Not all science has this issue. But to reference Thiel, a lot of the sciences with "science" in the name, seem to.
    by prolly97
  • It does seem like its way too easy to do this kind of fraud.

    The study seems extremely easy to replicate (no fancy equipment needed, just ask some people to proofread), and yet nobody did until now. Is there some way to change incentives to get people to try and replicate? Assign undergrads to replicate random research papers?

    by bawolff
  • > Is there some way to change incentives to get people to try and replicate?

    Sure. Pay them to do it.

    by p-e-w
  • "easy to replicate (no fancy equipment needed, just ask some people to proofread)"

    ? This was a human subject study, that would cost $$ and there would be very little upside to confirmation and only a little notoriety with a failure to replicate (which by the way is not dispositive, people might need multiple failed replications to know who to believe, depending on the size of the replication). Maybe you didn't read the post: this was data fraud discovered by statistical irregularities. That is indeed becoming easy to spot with more widespread statistical literacy, at least for unsophisticated jobs like this from decades ago.

    by jasonfarnon
  • Fund it. Unfortunately there's hardly any money for regular studies, let alone replication studies. If you want better science there has to be tax dollars behind it. Otherwise corporations choose what gets funded, and they only fund narratives and experiments they think will be profitable for them.
    by ModernMech
  • Replication is overrated. There's no point in doing more of it. You'd just replicate a ton of bad science:

    https://blog.plan99.net/replication-studies-cant-fix-science...

    N.B. nobody needed to replicate this study. It only took basic sanity checks to spot that it was very unlikely to be true (and thus, very unlikely to replicate). And they went much further than needed. They could have stopped at "the claimed effect is much larger than the height difference between men and women". This obviousness is a common pattern:

    https://www.fantasticanachronism.com/p/whats-wrong-with-soci...

    > Prediction markets work well because predicting replication is easy. There's no need for a deep dive into the statistical methodology or a rigorous examination of the data, no need to scrutinize esoteric theories for subtle errors—these papers have obvious, surface-level problems.

    So you could also say, how do we change the incentives to get people to peer review properly? That would be a lot cheaper.

    The obvious solution is for people to get paid to audit work professionally, and be fired if they sign off on bad results. But that exists already, it's called a corporate lab. Managers are incentivized by carrots and sticks to properly understand what their researchers are doing and if anyone is doing fraud they will be fired, maybe prosecuted. Universities don't do this because so much of what they do exists purely to spend money, with no higher purpose. There's no specific goal or outcome they're trying to achieve, and so no reason to do unpleasant things like enforcing incentives to get there.

    So in the end the solution is just to defund university research. The social structures are fundamentally unfixable. You'll never get people to care if the only purpose of their job's existence is to convince voters governments are funding science.

    by mike_hearn
  • My proposal, once you complete your study, if you find a significant positive result you should be required (prior to publication) to describe your methodology in full, and have another lab (who you never have any contact with) blindly take your methodology, and repeat it (on video), and then do their own independent analysis of the data.

    It's okay if it's $200k or so to do each of these, the university should pay. The point is if this methodology were applied people would have incentive to cheat anymore. It would also serve as an immediate replication.

    Schools won't do this because they have a perverse incentive for big successes and quiet retractions. It's corruption all the way up.

    by zug_zug