When done right, science remains the most reliable way of finding universal truths. Sadly, as the business of science has grown, the incentives to do it right have not kept up—by one estimate, nearly two-thirds of all published research cannot be confirmed by subsequent studies. In 2005, one groundbreaking analysis even tried to explain why that is, and how to change it.
In the year of its 10th anniversary, the author of the analysis, John Ioannidis, professor of medicine and health research and policy at Stanford University, and his colleague Bob Kaplan, emeritus professor at the University of California at Los Angeles, took questions from the public on Reddit on what can still be done to fix scientific research.
Quartz has condensed and edited the highlights of the discussion.
Nate Silver says that two-thirds of published research is unable to be replicated. Is that really true?
For most scientific fields, replication is not done frequently. In fields where empirical replication efforts have been done in a systematic scale, the failure-to-replicate rates are as high as you mention or higher.
For example, in psychology, two-thirds of 100 high-profile papers could not be replicated. In preclinical studies (such as oncology drug targets), failure-to-replicate rates have varied from 75% to 89%. Candidate gene studies failed-to-replicate about 99% of the time when we moved to the genome-wide era with more rigorous methods. But there is large diversity across scientific fields, so I am sure that there are fields where almost everything would replicate and others where almost everything would fail to replicate, if we had attempted replications.
Based on theoretical considerations, I would expect that over time the number of “correct” scientific results increases because we have more scientists, better methods, and more analyses done; but it could be that the proportion of “correct” results is actually declining, because increasingly we are working in more difficult areas of complex research where the yield of true discoveries may be relatively low and the noise is more prominent.
Even though scientists consider reproducibility an important aspect of how science is done, why do they not work towards making science more reproducible?
Sometimes I fear that we have met the enemy—and discovered it is us. To a large extent, decisions about promotion and tenure are governed by peers. And as a scientific community we have undervalued team science, replication, and evidence integration (as opposed to first or last authored original papers).
The tenure and promotion system can be a key determinant of what type of science we get done. If we reward people for the right reasons, this can be powerful. Tenure could be based on a combination of criteria which I call PQRST: productivity, quality, reproducibility, sharing, and translation.
Are science journals and the publishing industry a big part of the problem?
I think that there are indeed problems with the current incentive system in scientific publication and on how to advance in a scientific career.
Journals and publishing houses are perhaps part of the problem, but they are also part of the solution. For example, there has been progress in many journals to enhance transparency, sharing of data and protocols, use better standards for methods and statistics, and focus on reproducible research rather than just big claims. Many journals and their editors are actually at the forefront of these changes. It is an ongoing struggle and I would like to see journals as allies in this process rather than as competitors or enemies that need to be eliminated.
How do we prevent poorly designed research methods from being published?
Research methods are an interesting category of papers that is not as well studied in terms of its robustness. In most cases, there is a rather small community of scientists who can really evaluate a new proposed method. Also most new research methods don’t get applied in the community, even though they may be excellent and better than those that are widely used. I think we need to do some more thinking about how to improve this.
Do you think that offering better access to scholarly work will improve the media reporting in any way?
I am in favor of open access, but it will not solve all problems. Especially for the public, reading the full text of a paper will not allow most people to see where the problems are. Even top scientists may not be able to see the problems, unless they have access to protocols and raw data and they are willing to spend a lot of effort to probe into the work.
This being said, I have seen many press releases and news articles where indeed if someone could just take a 30 seconds look at the full paper, it would have been obvious that the paper has very little to do with the publicized hype.