Bad science

Why researchers keep citing retracted papers

This week, Retraction Watch released a list of the retracted research papers with the most citations, and many of the citations came after those papers were pulled.

The second-most-cited paper on this list, published in The Lancet in 1998, is widely known to be bogus. In it, researcher Andrew Wakefield drew a connection between autism and the vaccine for measles, mumps, and rubella (MMR). It was retracted in 2010, after the data behind it was found to be faked, but researchers continue to cite it.

Post-retraction citation, it turns out, is a frequent problem. One recent study looked exclusively at the works of the researcher Scott S. Reuben. Reuben was convicted in 2009 for falsifying data, and 25 of his papers were pulled as a result. The study found that 20 of those papers had been cited 274 times since the retractions.

In 2014, 45% of the retracted articles had been cited at least once. In only 25.8% of citing articles was it clearly stated that Reuben’s work had been retracted. … Our data show that even 5 years after their retraction, nearly half of Reuben’s articles are still being quoted.

Notice that about a quarter of those papers actually acknowledged that the cited work had been retracted. Ivan Oransky, co-founder of Retraction Watch, told Quartz this can happen for several reasons. Some papers, like the Reuben study, cite the retracted work to examine the retraction itself. Others may simply be pointing out that the findings of a withdrawn paper have been proven false.

Still, the majority of these citing articles fail to acknowledge the retractions. So how does this happen? How do researchers end up citing retracted work? Helmar Bornemann-Cimenti, the lead author of the Reuben study and professor at the Medical University of Graz in Austria, told Quartz in an email that researchers can miss retractions for several reasons.

One issue is that research databases like PubMed or EMBASE don’t always properly flag retractions, Bornemann-Cimenti said, pointing to a 2011 study on the issue. Another problem is how journals themselves report that studies have been pulled. In print journals, Bornemann-Cimenti said, “notes of retractions are typically printed in a not-so-prominent section.” On the web, “up to 32% of the withdrawn articles are not labeled as retracted,” he said, pointing to another 2011 study.

Then there’s the issue of local file storage. Bornemann-Cimenti pointed out that some researchers may compile their own electronic library of research documents, and therefore miss retraction notices that may (or apparently may not) appear online. Researchers may also simply use Google to find and download studies, and end up with the pre-retracted versions. “You sometimes find .pdf-files of research articles which are stored in online repositories or private or institutional websites,” Bornemann-Cimenti said. “Most often, they will not be removed if the original article is retracted.”

Many of these issues have been perpetuating for years. More than 16 years ago, in 1999, a study that analyzed post-retraction citations found problems not unlike those uncovered in the Reuben study:

Most of the citations it analyzed either directly cited the findings of the retracted papers as if they were valid (“explicitly positive”), or simply listed them without acknowledging the retractions (“implicitly positive”). (AIM stands for Abridged Index Medicus, a standard collection for medical journals, which later became the United States National Library of Medicine.)

John Budd, lead author of the 1999 study and professor emeritus at the University of Missouri, offered another explanation for how researchers may make this mistake. He told Quartz in an email that some authors may simply be using research that cites the retracted papers, and copy the citations over. “That way the retraction wouldn’t be evident,” Budd said.

The most cited paper on the Retraction Watch list was published in Science in 2005 and pulled in 2007. It claimed to identify a new fat cell protein, but later research showed the protein was identical to one that was already known. Oransky, the Retraction Watch co-founder, told Quartz that 68 of the citations for that paper occurred in 2015, though it wasn’t immediately clear how many of those acknowledged the retraction. In any case, the only papers on the list without post-retraction citations were just retracted this year, and it seems clear that these could continue to be cited. If they are, Oransky said, Reaction Watch will continue to update the list.

home our picks popular latest obsessions search