NUMBERS GAME

The first study to find brain games prevent dementia shows why statistics don’t tell the whole story

The brain-training industry is huge, and growing. Forecasts suggest people will spend some 4-10 billion of dollars on these types of games by 2020.

Not all are designed specifically to keep dementia at bay. Some are designed to improve your memory or IQ, for example. In any case, there is little evidence to suggest they work. A 2016 review of the scientific literature to date (paywall) showed that the biggest improvement people see when they play these games is in their ability to play them.

A study published on Nov. 16 reported the first evidence that a specific type of cognitive-training computer game, focused on speed decision-making, seems to lower dementia risk. This is the first time any lifestyle intervention—diet, exercise, or brain training—has shown to stave off this type of neurodegenerative disease.

But the findings, while “true” by scientific standards, are a good example of how statistics can be used to make weak results sound more important than they actually are.

For their study, researchers analyzed the results of the Advanced Cognitive Training for Independent and Vital Elderly (ACTIVE) study, which followed 1,220 people over 10 years. People were divided into four groups: Some participants came in and were taught memory tricks (like mnemonic devices), others were taught reasoning skills, and others trained with a processing-speed-training computer game. The last group of people weren’t given any kind of intervention, but came in for regular cognitive evaluations.

In a press conference, Jerri Edwards, a University of South Florida psychologist and lead author of the paper, described the simple game: it shows users a flash of an image, like a car, and then asks them what it was. (Notably, Edwards served for nine years as a consultant for Visual Awareness, Inc., the company that formerly held the intellectual property for this speed-training too, and now is on the board that monitors grants given by the National Institutes of Health to employees of Posit Science, which currently makes a commercially available version of the game, under the BrainHQ brand.)

The researchers evaluated participants’ mental capacities—like being able to remember words, directions, or make quick decisions—after six weeks, and then one, two, three, five, and 10 years later. The memory, reasoning skills, and processing-speed training groups each had 10 sessions over the first six weeks, and 80% had “booster” trainings after one and three years. At the end of 10 years, the researchers found that 10.8% (75 out of 695 total) of the control group developed dementia, compared to 10.1% (23 of 228) of the group that received reasoning-skills training, 9.7% (22 of 228) of those who received memory training, and only 5.9% (13 of 220) of those who received the speed training.

The only significant difference in dementia outcomes by science’s definition occurred with speed training. The preventative effect of speed training also seemed to work on a dose curve: those with more speed training were even less likely to develop dementia than those with less. The authors describe a possible reason for this correlation: a theory called the “brain reserve”’ which suggests that increasing your brain’s computing power though different types of training can protect it from dementia.

However, the statistics behind this claim aren’t nearly as strong as they sound.

“We in science have just generally come to the consensus [that] if the p-value is less than 0.05,” we should accept that the study’s results show truth, says Jason Hassenstab, a neurologist at Washington University in St. Louis unaffiliated with the paper. A p-value is technically a measure of the probability of getting results at least as extreme as the ones you observed, given that the null hypothesis (that there is no effect) is correct; practically, it is used as a statistical measure of how likely it is that an outcome is not the result of chance. For various reasons, a p-value of 0.05 has become the conventional level of significance in life sciences—and a proxy for true results.

The p-value for the outcome that speed-training decreased dementia risk is 0.049—just under the threshold. The researchers aren’t hiding anything; these data are all available for anyone to find and evaluate for themselves. But the fact of the matter is, had a few more people in the speed-training group developed dementia, the study would have been considered statistically irrelevant.

There are other troubling aspects of these statistics. As Dan Kopf, a Quartz’ economic reporter and a former data scientist, pointed out, in statistics, if you compare multiple outcomes against a control group, it increases the probability of finding a statistically significant result. (In this case, the study compared three alternative hypothesis to their control.) There are further calculations that can correct for this effect, but the authors of this paper appear to have omitted them.

This isn’t to say that the findings of this study are wrong; they’re just weak. They still are enough to say that speed-training at a later age may ward off dementia, and it certainly doesn’t hurt (minus the few hours of your day lost). But realistically, the reasons we develop dementia are so poorly understood scientists can’t say for sure that lifestyle change—like staying active, eating well, or brain training—can actually protect against it. Even Nobel prize winners and life-time athletes develop dementia, after all.


Read this next: An error made in 1925 led to a crisis in modern science—now researchers are joining to fix it

home our picks popular latest obsessions search