I've been waiting for this for a while. Reproducibility Project: Cancer Biology

The Reproducibility Project: Cancer Biology was an 8-year effort to replicate experiments from high-impact cancer biology papers published between 2010 and 2012. The project was a collaboration between the Center of Open Science and Science Exchange with all papers published as part of this project available in a collection at eLife and all replication data, code, and digital materials for the project available in a collection on OSF.

The work tried to repeat 193 experiments from 53 papers and found a significant number of challenges.

In summary

• Replication effect sizes were 85% smaller on average than the original findings
• 46% of effects replicated successfully on more criteria than they failed
• Original positive results were half as likely to replicate successfully (40%) than original null results (80%)

This quote from In the Pipeline is perhaps a useful reminder.

A robust result can probably be reproduced even if you switch to a different buffer, or if your cell lines have been passaged a different number of times, or if the concentration of the test molecule is a bit off, etc. The more persnickity and local the conditions have to be, the less robust your result is, and in general (sad to say) the lower the odds of it having a real-world impact in drug discovery. There are certainly important things that can only be demonstrated under very precise conditions, don’t get me wrong – but when you’re expecting umpteen thousand patients to take your drug candidate and show real effects, your underlying hypothesis needs to be able to take a good kicking and still come through.