I’ve not got the context on why Brazil was chosen here (paywall) - but I coincidentally read a story on here of Richard Feynman visiting Brazil whereby he assessed their teaching and tried to impart his teaching and learning techniques.

Is there any path forward to fixing the current reproducibility crisis in science? Individuals can do better, but that won't solve a problem at this scale. Could we make systemic changes to how papers are validated and approved for publication in major journals?

If they had just used NixOS, reproducibility would be less of a problem!

It would be interesting for reproducibility efforts to assess “consequentiality” of failed replications, meaning: how much does it matter that a particular study wasn’t reproducible? Was it a niche study that nobody cited anyway, or was it a pivotal result that many other publications depended on, or anything in between those two extremes?

I would like to think that the truly important papers receive some sort of additional validation before people start to build lives and livelihoods on them, but I’ve also seen some pretty awful citation chains where an initial weak result gets overegged by downstream papers which drop mention of its limitations.

I find it bizarre that people find this problematic.

Even Einstein tried to find flaws in his own theories. This is how science should actually work.

We need to actively try and falsify theories and beliefs. Only if we fail to falsify, the theories should be considered valid.

In my field, trying to reproduce results or conclusions from papers happens on a regular basis especially when the outcome matters for projects in the lab. However, whatever the outcome, it can't be published because either it confirms the previous results and so isn't new or it doesn't and no journal wants to publish negative results. The reproducibility attempts are generally discussed at conferences in the corridors between sessions or at the bar in the evening. This is part of how a scientific consensus is formed in a community.

And all the drugs and treatments derived from those "studies" are going to continue to be prescribed for another couple of decades, much like they were cutting people up to "cure ulcers" long after it was proven that an antibiotic is all you really need to cure it. It took about a decade for that bulletproof, 100% reproducible study to make much of a difference in the field.

Yet again more people in this site equating "failed to reproduce" with "the original study can't possibly be correct and is probably fraudulent"

That's not how it works. Science is hard, experiment design is hard, and a failure to reproduce could mean a bunch of different things. It could mean the original research failed to mention something critical, or you had a fluke, or you didn't understand the process right, or something about YOUR setup is unknowingly different. Or the process itself is somewhat stochastic.

This goes 10X for such difficult sciences as psychology (which is literally still in infancy) and biology. In these fields, designing a proper experiment (controlling as much as you can) is basically impossible, so we have to tease signal out of noise and it's failure prone.

Hell, go watch Youtube Chemists who have Phds fail to reproduce old papers. Were those papers fraudulent? No, science is just difficult and failure prone.

If you treat "Paper published in Nature/Science" as a source of truth, you will regularly be wrong. Scientists do not do that. Nature is a magazine, and is a business, and sees themselves as trying to push the cutting edge of research, and they will happily publish an outright fraudulent paper if there is even the slightest chance it might be valid, and especially if it would be really cool if it's right.

When discussing how Jan Hendrik Schön got tens of outright fraudulent papers into Nature despite nobody being able to even confirm he ran any experiments, they said that "even false papers can push the field forward". One of the scientists who investigated and helped Schon get fired even said that peer review is no indicator of quality or correctness. Peer review wasn't even a formal part of science publishing until the 60s.

Science is "self correcting" because if the "effect" you saw isn't real, nobody will be able to build off your work. Alzheimer's Amyloid research has been really unproductive, which is how we knew it probably wasn't the magic bullet even before it had fraud scandals.

If you doubt this, look to China. They have ENORMOUS amounts of explicit fraud in their system, as well as a MUCH WORSE "publish or perish" state. Would you suggest it has slowed them down?

Stop trying to outsource your critical thinking to an authority. You cannot do science without publishing wrong or false papers. If you are reading about "science" in a news article, press release, or advertisement, you don't know science. I am continually flabbergasted by how often "Computer Scientists" don't even know the basics of the scientific method.

Scientists understood there was a strong link between cigarettes and cancer at least 20 years before we had comprehensive scientific studies to "prove" it.

That said, there are good things to do to mitigate the harms that "publish or perish" causes, like preregistration and an incentive to publish failed experiments, even though science progressed pretty well for 400 years without them. These reproducibility projects are great, but do not mistake their "these papers failed" as "these papers were written fraudulently, or by bad scientists, or were a waste".

Good programmers WILL ship bugs sometimes. Good scientists WILL publish papers that don't pan out. These are truths of human processes and imperfect systems.

This doesn’t really surprise me at all. It’s an unrelated field, but part of the reason I got completely disillusioned with research to the point I switched out of a program with a thesis was because I started noticing reproducibility problems in published work. My field is CS/CE, generally papers reference publicly available datasets and can be easily replicated… except I kept finding papers with results I couldn’t recreate. It’s possible I made mistakes (what does a college student know, after all), but usually there were other systemic problems on top of reproducibility. A secondary trait I would often notice is a complete exclusion of [easily intuited] counter-facts because they cut into the paper’s claim.

To my mind there is a nasty pressure that exists for some professions/careers, where publishing becomes essential. Because it’s essential, standards are relaxed and barriers lowered, leading to the lower quality work being published. Publishing isn’t done in response to genuine discovery or innovation, it’s done because boxes need to be checked. Publishers won’t change because they benefit from this system, authors won’t change because they’re bound to the system.

As part of the larger reproducibility crisis including social science, I wonder how much these things contribute to declining public confidence in science and the post-truth era generally.