Every day, machine learning algorithms underpinning Google and Facebook decide which information we see and who we interact with. It might seem that artificial intelligence’s decisions would be made without the socially constructed prejudices of humans, but algorithms, too, can be biased towards or against certain genders, ethnicities, or social groups—though that’s likely our fault.
Research from Princeton University suggests that these biases, like associating men with doctors and women with nurses, come from the language taught to the algorithm. As some data scientists say, “garbage in, garbage out”: Without good data, the algorithm isn’t going to make good decisions.
Quartz first wrote about this research when it appeared on the internet in August 2016. We spoke again with Aylin Caliskan, a Princeton postdoctoral researcher and co-author of the final peer-reviewed paper published Thursday in Science, to see how this research can actually rear its head on the internet.
The interview has been edited and condensed for clarity.
Quartz: How do the language biases taught to algorithms actually affect people day to day?
Caliskan: One of my native languages is Turkish. It’s a genderless language, meaning there is no gender in nouns, pronouns, adjectives, verbs, or anything. There’s one pronoun for “he,” she,” and “it.” When I tried to translate from Turkish that “he…”, “she…”, or “it…is a doctor,” it translates to English or any gendered language as “he’s a doctor.” Then I tried [using] ”a nurse,” [and] it’s translated always to “she’s a nurse,” in gendered languages. These are gender biases, or stereotypes, about the world and occupations. When I tried “teacher,” it translated it as “she’s a teacher” and “professor” translated to “he’s a professor” or “he’s a programmer” or so on.
In that case, when you directly translate, and “nurse” is “she,” that’s not accurate. It should be “he or she or it” is a nurse. We see that it’s making a biased decision—it’s a very simple example of machine translation, but given that these models are incorporated on the web or any application that makes use of textual data, it’s the foundation of most of these applications. If you search for “doctor” and look at the images, you’ll see that most of them are male. You won’t see an equal male and female distribution.
But these models also contain statistical facts about the world. I went to the 2015 US Bureau of Labor Statistics web page, they had occupation names and the percentages of women in those occupations. By taking the system’s most common occupations and then measuring their associations with being male or female, we’re getting 90% correlation with the actual percentage of women. But at the same time it makes us realize “nurse” or “teacher” is biased towards women, and “professor,” “doctor,” and “programmer” are regarded as men as a stereotype, but at the same time this is reflecting the actual statistic in the world with 90% correlation.
And it’s not just text, but kind of cascades to everything text touches, like images or videos.
Yes, anything that text touches. Images and videos are labelled to they can be used on the web. The labels are in text, and it has been shown that those labels have been biased.
Who’s responsible for making sure this doesn’t affect society?
I would say all of us. Everyone throughout the development process is responsible for the ethics of their applications—they need to be able to guarantee that at the output level, there’s a human expert in the loop that makes sure that the output is not biased towards whatever their data is.
So even if the data is biased, the human would catch the error. And what other kinds of data besides text can be biased?
Any kind of data that humans perceive and humans end up labelling or classifying in some sense. Think about a case where there is video surveillance used by law enforcements—and nowadays they’ve tried to use a lot of automated methods to easily detect anomalies or intrusions, unexpected behavior and so on in video. And if this [training data] is biased, we can see how risky that can get, and how dangerous it can be for certain populations, labelling them as dangerous or risky out of nothing.
Nevertheless, you cannot directly remove the bias from the dataset or model because it’s giving a very accurate representation of the world, and that’s why we need a specialist to deal with this at the application level. Just make an accurate decision, and then make sure it’s not biased towards certain people.
And that’s the human in the loop.
Exactly.