Surveys are key tools of social science research. But sometimes they’re tainted by responses from subjects who fail to follow the instructions they’ve been given.
Researchers looking into this problem have confirmed what the more conscientious among us have long suspected: Habitually careless people are the worst kind of people. And their carelessness can distort results of survey-based research.
Sloppy survey respondents tend to slack off in other areas, too. Those enrolled at universities (where students are a common source of study subjects for researchers) tend to have lower GPAs and skip more classes than their more precise peers, a recent study found. Acquaintances rate them as less pleasant, less conscientious, and less stable emotionally.
We are all guilty of occasionally putting a half-hearted job into something we don’t care much about. But in great enough numbers, the chronically careless can skew results of expensive and time-consuming studies simply by not paying close enough attention to what’s being asked of them.
If 10% of respondents in a survey don’t follow basic instructions, it can throw the whole study off, said Nathan Bowling, a professor of psychology at Wright State University and lead author of the paper on careless people’s attributes.
“Most people are not careless. It’s not the typical behavior. But it does happen enough that it can affect our research results,” Bowling said.
Researchers have tended to deal with the problem by tossing out sloppy respondents’ data. The problem with that, Bowling points out, is that it, too, skews results. If you want a representative snapshot of a general population, surveying only its most organized members doesn’t work.
In his work on laziness—a problem referred to diplomatically in the literature as “insufficient effort responding”—people respond well to carrots and sticks. If people are told that a poor job may cancel a promised reward for participating in the survey, like payment or course credit, they work harder, Bowling said.
Likewise, carefulness increases when participants are promised a bonus for paying particularly close attention, like being entered into a gift card drawing.
The most effective deterrent against carelessness, though, is face time with the researcher who is counting on their efforts.
When a graduate student working with Bowling brought participants directly into a lab for a study instead of asking them to fill out surveys amid the distractions of home, thoughtless answers virtually disappeared. But that wasn’t a happy outcome in this particular case.
“The study ended up flopping. You really can’t do a study on [insufficient effort responding] if everyone’s careful,” Bowling said.
For most other kinds of studies, a personal appeal might be more helpful. But it’s costly to bring subjects in for every study. Instead, Bowling is studying the efficacy of having researchers attach an introductory video to their online surveys, in which they personally—if remotely—face down the subject and explain why his or her effort is valuable to the research project.