There have been a recent spate of news around bad science – results that can’t be replicated – and as a young scientist, I’ve been reading these with much interest and concern. As a Ph.D. student, I’ve come to see the “sausage making” process of science firsthand and sometimes it is an ugly and broken process. I thought I’d highlight some points here, but one should not walk away and give up on the scientific process.. and one should not walk away thinking they can ignore major tenets of modern science that have been replicated and supported over the years (I’m thinking of things like evolution, climate change, etc)..
The first news that caught my attention was a report of some researchers only able to replicate 6 out 53 “landmark” cancer research results (11% !!):
And because people don’t generally publish negative results, we always suffer from “publication bias”:
” The oncologist, whom Begley declines to name, had an easy explanation. “He said, ‘We did this experiment a dozen times, got this answer once, and that’s the one we decided to publish.’ “
Recently, this topic made it to the cover of the Economist:
with more details here:
As the economist article highlights, there may be many causes.. from bad statistics, to cherry-picking results, to no one having the time or money to replicate research before they build on it. Unfortunately, the solutions are complex and will involve many players.
As I’ve come to realize, most life scientists are terrible at statistics, with only a weak grasp of even the most basic of techniques and how to apply them. Thus, even during peer review, many mistakes are not caught:
” We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. “
Here, the editors at Nature lament on the misuse of error bars and more: http://www.nature.com/nature/journal/v492/n7428/full/492180a.html
The latest chapter is an article published in Nature that highlights the almost arbitrary value of p=0.05 that most scientists use as unacceptable… and offers a potential solution:
It seems like in the next couple years or decades, people will start moving away from traditional p value testing, and more advanced techniques like bayesian & information theory approaches (AIC) partly for the issues discussed in this post, and partly because our computers have become powerful enough to enable these new methods to be applied:
For some more on this debate..courtesy of xkcd.com..
If you aren’t quite sure what I’m taking about with all this statistics stuff, this video does a great job of explaining it:
PS: sorry many of the links are behind journal paywalls… a topic for another post on open access in science