What Makes Science True? | NOVA

What makes science reliable? The ability to reproduce the results of an experiment, known as reproducibility, is one of the hallmarks of a valid scientific finding. But science is facing what many consider a reproducibility crisis, and the stakes are high. Many scientific claims cannot be replicated, and many clinical trials fail as a result.

This 15-minute video examines the reproducibility crisis and reports on the outcome of five experiments designed to test the reproducibility of cancer biology studies.

While NOVA had no editorial involvement in this video, we believe it is a thoughtful and well-researched exploration of a complex topic, and we are pleased to share it with you. The video was produced by Dakin Henderson and Kelly Thomson, and was funded entirely by the Laura and John Arnold Foundation (LJAF).

In the interest of full transparency: LJAF has an interest in this subject matter, is one of a handful of funders that supports the cost of conducting reproducibility experiments, and provided funding to the organizations featured in this video in order to carry out the reproducibility studies covered here. We have been assured that LJAF had no editorial involvement in the video.

You can also learn more about reproducibility by reviewing the following publications:

Many analysts, one dataset: Making transparent how variations in analytical choices affect results
https://osf.io/j5v8f/

False-Positive Psychology : Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
http://journals.sagepub.com/doi/pdf/10.1177/0956797611417632

The Statistical Crisis in Science
http://www.stat.columbia.edu/~gelman/research/published/ForkingPaths.pdf

Why Most Published Research Findings Are False
http://robotics.cs.tamu.edu/RSS2015NegativeResults/pmed.0020124.pdf

Journals unite for reproducibility
http://science.sciencemag.org/content/346/6210/679.full

Promoting an open research culture
http://karlan.yale.edu/sites/default/files/science-2015-nosek-1422-5.pdf

Estimating the reproducibility of psychological science
http://www.psykologforbundet.se/Documents/Psykologtidningen/Aktuellt%20Pdf/Science%20aug%202015.pdf

Design, power, and interpretation of studies in the standard murine model of ALS
https://www.researchgate.net/publication/5577813_Design_power_and_interpretation_of_studies_in_the_standard_murine_model_of_ALS

Raise standards for preclinical cancer research
http://www.nature.com/nature/journal/v483/n7391/full/483531a.html#t1

Believe it or not: how much can we rely on published data on potential drug targets?
http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html

A Survey on Data Reproducibility in Cancer Research Provides Insights into Our Limited Ability to Translate Findings from the Laboratory to the Clinic
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0063221

ASCB Member Survey on Reproducibility
http://www.ascb.org/wp-content/uploads/2015/11/final-survey-results-without-Q11.pdf

1,500 scientists lift the lid on reproducibility
http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970

Interpretations of the results of the Reproducibility Project Psychology and the National Institute for Neurological Disorders and Stroke replications vary depending on how “reproducibility” is defined. Original sources for each reproducibility study are:

Amgen: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html
Bayer: http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html
NINDS: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4146316/
ALS TDI: http://www.nature.com/news/preclinical-research-make-mouse-studies-work-1.14913
RP:P: https://osf.io/ezcuj/wiki/home/
RP:CB: https://osf.io/e81xl/wiki/home/

We hope that this information on reproducibility will start a conversation. We encourage you to share this video and the publications above, and if you have feedback, let us know in the comments. We’d like to hear from you.

Credits:

Produced and Directed by
Dakin Henderson & Kelly Thomson

Animation by
Kent Leon Welling

Original Music by
Louis Weeks

Editing, Cinematography and Narration by
Dakin Henderson

Funding provided by the Laura and John Arnold Foundation

Featuring
Trevor Butterworth
John Ioannidis
Elizabeth Iorns
Brian Nosek

Additional Music
“6 Ghosts I” and “18 Ghosts II” by Nine Inch Nails

Thank You
Jay Bradner
Peter Gotzsche
Marcia McNutt
Ben Pender Cudlip
Nicole Perfito
Lawrence Rajendran
Jackson Solway
Deborah Zarin

Archival Material
abc7 News
Atlantic Monthly Group, Inc.
Conus Archive
The Economist Group
Getty Images
KARGER
The Lancet/Elsevier
NASA
Nature Publishing Group
NBC Bay Area News
PLOS One
Pond5
Salon Media Group, Inc.
Science/AAAS
Scientific American
Shutterstock
The Week Publications, Inc.
Wiley

***

NOVA Facebook: https://www.facebook.com/NOVAonline
NOVA Twitter: https://twitter.com/novapbs
NOVA Instagram: https://www.instagram.com/novapbs/

Written by NOVA PBS Official