Coffee causes cancer, right? It doesn’t—but that doesn’t stop nearly a tenth of adults from thinking it does. Welcome to the messy world of overblown—or downright wrong—health claims.
The reasons why such misinformation spreads are complex, but we do know that there is an inherent bias toward publishing research and news that will grab people’s attention. While journalists take some blame, health claims are often exaggerated at their source by university press teams.
But there are good health studies—you just need to know what to look for. Here are five simple steps to help you discern whether a health claim is valid:
1. Are humans involved?
A lot of scientific studies are done on model animals because it is a less risky way of testing new drugs or hypotheses. These experiments can tell us many interesting things, but their relevance to the human population is limited. If a news story is based on a study done in mice, do not presume that the results will be applicable to humans.
2. Look at the sample size
If humans are involved, then look at how many of them participated in the experiment. Be skeptical of results that involve less than 100 people and fairly skeptical of those that involve less than 1,000 people. It is important to test a hypothesis on a larger population to be able to observe true and replicable results. That is why a drug is not approved before it has been tested on many thousands of patients.
3. What type of study?
Every scientific experiment is meant to test a hypothesis, and yet there are hundreds of ways in which that experiment can be designed. That is why it matters what kind of study design researchers used to test their claim.
The most reliable studies are “systematic reviews” or “meta-analyses,” which are an analysis of the many studies that tested a particular claim.
Then there are randomized controlled trials, where half the participants are given the drug and half are given a placebo. This is the gold standard for medical evidence, because you can tease out confounding factors and figure out whether your intervention actually had the effect that was observed.
Cohort studies involve following a large population for a long periods of time, and can provide relatively strong correlational link between, say, exercise and longer life. Case-controlled studies and case studies are the least reliable because they involve only a handful number of participants. But they are the cheapest and quickest way to see whether a hypothesis holds enough water to try a more rigorous and expensive experiment.
4. Check for “proof” or “cause”
In medicine, scientists will only have some degree certainty in their claim. Biology is so complex that, it is near impossible to be 100% certain. So when you read stories that claim that “Scientists prove X,” take it with a grain of salt (or discount the claim by 80%).
Moreover, most health studies draw mere correlations (or links) rather than direct causes.
If a health news story claims that something “causes” cancer, be very skeptical of the claim.
5. Search for caveats
A good health story will not only explain the value of a claim, but also give you warnings about why you shouldn’t trust the claim fully. For instance, a study claimed that you can become less sexist and racist while you sleep. Really? It may be possible, but there are caveats—all the study’s participants (a relatively small sample) were white and came from a university setting.
Believing in wrong health claims can have real consequences. Just follow Thomas Huxley’s advice: “skepticism is the highest of duties; blind faith the one unpardonable sin.”
If you come across bad health stories, send them my way at akshat@qz.com. Cropped image by pslee999 on Flickr under license CC-BY 2.0.