In 2015, Nextdoor, the location-based social networking app, gained a reputation as a locus of racial profiling. Users were sending alerts for merely spotting hoodie-wearing Black men walking in their neighborhoods. One damning report described Nextdoor as “a forum for paranoid racialism—the equivalent of the nosy Neighborhood Watch appointee in a gated community.”
In studying the problem, Nextdoor realized that its easy-to-use app was partly to blame.
Creating intuitive, easy-to-use products and interfaces is the domain of user experience designers. Don’t Make Me Think is both the title of a core textbook in the field and many designers’ professional mantra. An approach called “anticipatory design” even takes the virtue of speed to the extreme, promising a system so seamless that consumers never have to make a single choice.
But the very features that are meant to make technology easier and faster to navigate—like shortcuts and auto-complete functions—can also encourage users to make snap judgements and put marginalized populations at risk. This is just one way racial bias is bundled into everyday systems, albeit sometimes unintentionally.
Speaking at TED conference earlier this month, Jennifer Eberhardt, a social psychologist who helped Nextdoor address its racial profiling problem explained how designing for speed can sometimes foster injustice. “Categorization and the bias that it seeds allow our brains to make judgements more quickly and efficiently,” she said. “But just as the categories allow us to create quick decisions, they also reinforce bias. The very things that help us see the world also can blind us to it. They render our choices effortless—friction free.”
This concern is partly why, over the past few years, a cadre of UX designers have been championing the utility of interruptions, or “friction,” in design processes. Awakened to the the ethical quagmires of big data during the 2016 US elections and seedy “dark patterns” of social media platforms, many are questioning the norms and metrics of their profession. Now, as the Black Lives Matter movement gains momentum, the urgency of addressing pre-programmed biases in tech products and instruments is even more pronounced. To check these biases, some designers believe, like Eberhardt, that we need more purposeful interruptions to slow the user down.
Working closely with Nextdoor’s in-house teams, Eberhardt recommended adding a checklist for users to go through before they report an incident. They also tweaked the language of its prompts—changing the oft-cited security slogan,”if you see something, say something,” to “see something suspicious, say something specific”—to get people to reflect if they’re actually witnessing criminal behavior. Within a few months, the San Francisco-based start-up was able to curb racial profiling incidents by 75%.
Racial profiling (aka “the Karen problem”) still plagues Nextdoor and similar neighborhood watchdog apps like Citizen and Amazon Ring today. But Eberhardt’s work proves that there are ways to design systems that can be part of the solution.
Eberhardt, who is co-director of Stanford University’s SPARQ lab, says that design can nudge users to reconsider their actions by adding additional steps in the form of dialog boxes, warnings, or confirmation messages. Her lab has also worked with the the Oakland Police Department in 2016 to investigate racial profiling in routine traffic and pedestrian stops. She asked officers to check their bias by asking themselves a question before flagging someone down. “Is this stop intelligence-led? Yes or no?” The extra step was meant to slow officers down enough to consider if they had specific evidence against the person. The simple self-query resulted in 40% fewer routine stops, while the crime rate continued to dip.
“We’re all vulnerable to bias, but we don’t act on bias all the time,” Eberhardt explained at TED. “What we don’t want to do is to create an easy-to-use system that can amplify bias and deepen racial disparities.”
Evie Cheung, an experience designer with a background in psychology, has also been studying the various ways bias seeps in tech. For her graduate project, she researched how the voices in Alexa and Siri can foster racial and gender stereotypes in children.
She believes introducing friction into technology at the right moments is useful. For one thing, a warning or pop-up box may remind us that our actions have an effect on people beyond our own bubbles. “We’ve gotten so used to this overly-optimized world; we’re so used to getting food delivery or a car at the click of a button,” she observes. “We sometimes don’t think about how and who makes these products and services happen.”
“It’s interesting to think about how technology can dehumanize the process and also actually humanize it,” she adds. “Can we actually design against the sort of the assumptions that are being made today?”
To do this, Cheung says agencies need to bring more diversity to the drawing table. “It’s about lived personal experience… I will never know what it’s like to be a Black man in America,” says Cheung, a first generation Asian American. “There’s a breaking point with empathy when you kind of have to acknowledge your privilege.”
Similarly, Huge, the Brooklyn-based digital agency that promoted fully-automated “anticipatory design” systems in 2015, has since tweaked its stance. “Anticipatory design isn’t something we preach of a lot any more,” says Emily Wengert, the agency’s VP of user experience.
Seamless design may be useful for anticipating problems—say, sending automatic alerts to a family member if an elderly person has had an accident or missed a meal at home. But Huge has also learned that it’s not necessarily useful for everyday conveniences like re-ordering soap at the push of an Amazon Dash button. “Really great anticipatory design is reliant on great data and great content,” she explains. “What we’ve learned in the intervening years is that data and content are biased as the humans who made it.”
Wengert says that Huge is promoting discernment or “mindfulness” in UX design these days. “I think the new task for designers is to know when technology can think with you—be proactive—and when it can actually trigger you to engage more—to be mindful,” she explains.
To be clear, introducing friction doesn’t mean reverting back to the headache-inducing web interfaces of the 1990s and early 2000s—a vestige of which lingers in some financial data tools, if you want a refresher. Nor does it mean creating unnecessary hurdles for users. Friction, when applied judiciously, is meant to bolster users’ agency over their own transactions.
Bill Gribbons, a professor of information design and corporate communication at Bentley University and director of the largest UX design graduate program in the US, approves of the evolution in the field. “It’s not even friction or adding hurdles; it’s just good design,” he says. “It’s recognizing that people have this implicit bias and coming up with an appropriate response to counter it.”
Gibbons recognizes that his this rubric for good design isn’t always shared by profit-driven companies who employ his grad students. “But I’m encouraged,” he tells Quartz. “Like look at what’s happening at Facebook right now where the developers are speaking up,” he said, referring to a recent employee walkout in protest of Facebook’s inaction against US president Donald Trump’s incendiary posts on the platform.”It’s not an easy thing, but they’re using the power because without their workers, [companies realize] they’re nothing.”
Gribbons, who is a veteran of the advertising industry, says that today’s ethical-design reckoning is particularly crucial as tech seeps into every system in society—including our own bodies.
“Now is the time for us to get it right,” he says. “With the internet of things, wearables and implantables,” the influence that design has on all our lives will be even less apparent.