In mid-March, as the coronavirus pandemic bloomed in the United States, the Food and Drug Administration laid out new guidelines for companies that wanted to market Covid-19 tests.
The change came nearly two months after the first confirmed US case, a period during which the FDA was criticized for slowing diagnostic test distribution by holding them to its normal high standards. The new regulations for tests were extremely loose—especially for another class of tests that identify antibodies to an infection. The FDA essentially allowed any antibody test to be marketed as long as it didn’t purport to be FDA-approved or claim that it could single-handedly diagnose Covid-19.
Hindsight and a growing volume of research indicate that may have been an overcorrection.
In the following weeks, dozens of antibody tests flooded the market; there are now more than 200 available, including 12 that have received an FDA emergency use authorization. Not all are high-quality: A review in late April led by University of California scientists of 14 prominent tests found only three that produced consistent, reliable results. Others have been outright frauds.
So the FDA is tightening its rules for the first time since mid-March.
“Flexibility never meant we would allow fraud,” two of the agency’s top officials said in a statement on Monday. “We unfortunately see unscrupulous actors marketing fraudulent test kits and using the pandemic as an opportunity to take advantage of Americans’ anxiety.”
Under the new guidance, all manufacturers, including those with tests already on the market, will be required to file a request for an EUA within 10 days of releasing the test, creating a new layer of paperwork that will slow down the process and force FDA to better scrutinize each new test.
The guidance also includes a minimum threshold for performance: Tests should accurately identify at least 90% of positive cases (what epidemiologists call sensitivity) and 95% of negative cases (specificity). That should weed out some of the worst tests; according to the Johns Hopkins Bloomberg School of Public Health, some tests currently in use have sensitivity as low as 87%.
But it may not be enough to fix more fundamental problems with the tests. For one thing, due to a convoluted bit of epidemiological math, even tests with high sensitivity and specificity can produce high rates of false positives when the prevalence of infection in a population is low. Steve Miller, director of the clinical microbiology lab at the University of California-San Francisco, said FDA would need to require a specificity more like 99.5% to overcome that obstacle (for more on that concept, take our serology test simulator for a whirl).
The FDA’s accuracy thresholds may be fine for confirming a diagnosis in a symptomatic patient who has also taken a swab test, Miller said, “but seem highly inadequate for general population screening.”
Moreover, it’s not clear that companies will calculate their self-reported sensitivity and specificity in a consistent way. The rules require manufacturers to judge their test using 30 samples confirmed to contain SARS-CoV-2 antibodies and 75 without them. But there’s a lot of room for variation within that protocol, Miller said. Positive samples might be loaded with a high concentration of antibodies, goosing the odds that the test will find them. The same test might produce a very different result if used early in an infection, when antibody concentrations are low (it can take at least a week for antibodies to develop). Will negative samples be completely free of all antibodies, or might they contain some to other viruses that throw the test off?
“If you don’t include a lot of variety in your sample set—and there’s no requirement to—you can get very good performance with an easy sample set that doesn’t reflect the patients you’re actually using it on,” Miller said. “The key is not to cook the numbers, but to understand the true limitations of your assay.”
Emma Spaulding, an FDA spokesperson, said in an email that when evaluating whether to authorize a test, the agency “considers information pertaining to how samples were sourced, or selected, and collected,” including “evaluating when the sample was collected relative to the time since symptom onset and/or clinical diagnosis, if known.” The FDA will also independently evaluate certain antibody tests, Spaulding wrote, in addition to evaluating the data provided by companies.
Miller said he’s confident that there are now enough antibody tests being produced by well-established manufacturers—one released this weekend by Roche, for example—that are both highly accurate and dependably tested that most clinics should have at least a few good options. But because the demand for these tests is so high and urgent, it may be necessary to let a few on the B-list into the mix.
“There’s a balance that needs to happen here,” Miller said. “You can’t expect perfection and achieve that in the short timeframe that we have.”