On March 26, researchers from the University of Washington’s Institute for Health Metrics and Evaluation (IHME) released an important forecast. Using data from current Covid-19 outbreaks—at the time, largely those in Wuhan, China—the institute had developed a model to prepare hospitals around the world for their peak equipment usage and deaths over the next four months.
In the United States, the institute’s model has gotten a lot of attention. It’s been cited repeatedly in White House briefings on Covid-19. Its predictions have a few important assumptions: That only a small proportion of the population would become infected (based on infection rates in other countries), and that all states would implement shelter-in-place orders by April 2. But if those held true, the model suggested, the country’s deaths would conceivably peak around mid-April and fall gently over the months after.
The model worked in at least one sense: Shortly after its release, US president Donald Trump and his team extended national social distancing guidelines for a full month, through April 30. The team cited the model as one of the factors taken under consideration. Although the White House guidelines (pdf) don’t mandate any particular behavior, several states and municipalities chose to implement shelter-in-place orders. These efforts have been working, but they’re under threat as governors of some states plan to relax distancing guidelines as soon as April 24.
Now that the outbreak in the US is further along, some 70 researchers are working to update the IHME model on a near-daily basis. The hope is to forecast the end of the Covid-19 epidemic. “We’re now very much in the policy sphere of when would it be prudent to lift some of the more stringent social distancing measures,” says Theo Vos, an IHME professor of health metrics sciences working on the project. On April 17, the most recent update the model at the time of publication, projections included “containment periods” in which some social distancing measures can be eased.
In this unprecedented situation, though, some statisticians and epidemiologists are skeptical of whether the model is still appropriate. “The IHME projections are based not on transmission dynamics but on a statistical model with no epidemiologic basis,” researchers from the Imperial College of London and the University of California, Berkeley wrote in an online opinion published in the Annals of Internal Medicine the week of April 13.
Statistical models base their projections on previous data and a particular mathematical formula. The IHME model uses a formula called the Gaussian error function, explained the biologist Carl Bergstrom in a detailed Twitter thread last week. (Bergstrom studies scientific information flow at the University of Washington, but is unaffiliated with IHME.) This function produces a something like a bell-shaped curve—the same kind you’d expect to see when plotting the scores of a well-designed exam. “They’re assuming the number of deaths will take the form of basically a normal distribution,” Bergstrom told Quartz on April 16.
The IHME model essentially assumes that the the speed at which death rates in some states ramp up is roughly the same speed at which they will ramp down. So if it took a month to get from no Covid-19 deaths to peak deaths, the model predicts it will also take about month to get from the peak to zero fatalities, Bergstrom explained. “I don’t think it was a bad thing to do to try to model that specific situation,” he said. “I just wish [they] would be clearer about its limitations.”
Even if IHME’s curve accurately modeled Covid-19’s behavior, the real-world death counts feeding it can fluctuate wildly from day to day, especially when public health officials change the way they count Covid-19 deaths. Currently, the IHME model relies on death counts from Johns Hopkins University. Though these data are from a reputable source, they don’t come in every day. Apparent decreases in deaths can sometimes make the model show that total daily deaths are on the decline, when they may not be. Vos says the model can incorporate these data point fluctuations, but still cautions that “we should watch for a potentially longer tail of cases coming down than the Gaussian error function would predict.”
The limitations of the Gaussian error function are especially clear in the IHME’s model of country-wide deaths in the US. The final model aggregates each state’s data. But some states, particularly New York, California, and Washington, had outbreaks earlier than other states. As a result, the cumulative total for US deaths disproportionately reflects these early states—especially New York, which is the epicenter of the country’s outbreak. Looking at the deaths per day curve for the entire US gives the impression that deaths in the country peaked on April 15.
Vos says that this aggregation function means that the US peak appears to be more stretched out—like a table top, rather than a mountain top. When asked about whether the US had truly hit a peak last week, Vos said he didn’t believe it had just yet, despite the fact that as of today (April 21), the US is supposed to be six days past its peak daily deaths. He agreed that New York and Washington may have peaked.
Even if IHME’s models correctly predict peak deaths and hospital surge, there’s a final important caveat: They’re only meant to look at the first wave of an outbreak. After municipalities and states ease up on social distancing, they could experience second and third waves. Although the newest updates include statewide data showing when the outbreak could, theoretically, start to be “contained”—meaning social distancing measures could be lifted—these predictions assume that testing and contact tracing will be sufficient to prevent resurgences. “I’m worried that this model is assuming that we’ll be successful and controlling things. We don’t know if that is feasible in the West,” says Bergstrom.
The IHME model has undoubtedly been useful. It reinforced the idea that social distancing was an important measure to control deaths to begin with. But by their very nature, models can’t be completely predictive; that’s why it’s best to rely on multiple models to get an idea of all possible outcomes. Other types of models, like mechanistic ones, account for the behavior of the disease itself. A model from the Imperial College of London, which also contributed to policies in the UK and US, factors in variables like transmission rates, incubation periods, and adherence to social distancing guidelines. Mechanistic models aren’t perfect either, of course; biological and behavioral factors are difficult to predict, particularly with an emerging infectious disease.
The Trump administration, meanwhile, appears to be using the IHME model as one of its main sources of information to decide when to lift social distancing measures and when to send in emergency equipment. Going off it alone, limitations and all, it would look like the time to end social distancing is coming up soon. If that interpretation is wrong, the effects could be catastrophic. If the US opens back up too early, the country risks undoing all the hard work so far, Bergstrom says.
The IHME says it will continue to update its models to make predictions. “We were lucky to get into the ear of the people of the White House,” says Vos.