The internet’s hero, John Oliver, has acknowledged what I’ve been saying all along—most science reporting is deeply suspect.
After training as a scientist, I became a journalist partly because I wanted to help correct this record. Though I’m just a tiny foot soldier against a deluge of terrible science reporting, it’s nice to have John Oliver at my side.
If you don’t have 20 minutes to watch the full segment, here is Quartz’s short guide to avoid getting duped by overblown scientific studies.
1. Are humans involved?
A lot of scientific studies are done on model animals because it is a less risky way of testing new drugs or hypotheses. These experiments can tell us many interesting things, but their relevance to the human population is limited. If a news story is based on a study done in mice, do not presume that the results will be applicable to humans.
2. Look at the sample size
If humans are involved, then look at how many of them participated in the experiment. Be skeptical of results that involve fewer than 100 people and fairly skeptical of those that involve fewer than 1,000 people. It is important to test a hypothesis on a larger population to be able to observe true and replicable results. That is why a drug is not approved before it has been tested on many thousands of patients.
3. What type of study?
Every scientific experiment is meant to test a hypothesis, and yet there are hundreds of ways in which that experiment can be designed. That is why it matters what kind of study design researchers used to test their claim. The most reliable studies, for instance, are “systematic reviews” or “meta-analyses,” which are an analysis of the many studies that tested a particular claim.
4. Check for “proof” or “cause”
In medicine, scientists will only have some degree certainty in their claim. Biology is so complex that, it is near impossible to be 100% certain. So when you read stories that claim that “Scientists prove X,” take it with a grain of salt (or discount the claim by 80%). If a health news story claims that something “causes” cancer, be very skeptical of the claim.
5. Search for caveats
A good health story will not only explain the value of a claim, but also give you warnings about why you shouldn’t trust the claim fully. For instance, a study claimed that you can become less sexist and racist while you sleep. Really? It may be possible, but there are caveats—all the study’s participants (a relatively small sample) were white and came from a university setting.