The ethical blindness of algorithms

What’s your supermarket  assuming about you?
What’s your supermarket assuming about you?
Image: Reuters/Rick Wilking
We may earn a commission from links on this page.

Can an algorithm be racist? It’s a question that should be of concern for all data-driven organizations.

From analytics that help law enforcement predict future crimes, to retailers assessing the likelihood of female customers being pregnant (in the case of Target, without their knowledge), the increasing scale of computer cognizance is raising difficult ethical questions for business.

Witness the controversy that the crime app SketchFactor caused in launching its crowdsourced service in the US. The app works by allowing users to report, in real time, how subjectively “sketchy” a particular neighborhood may be, enabling an algorithm to determine the apparent safety of the area for pedestrians. Inevitably, the app has drawn accusations of racism, with some commentators labeling it a service that literally color-codes neighborhoods.

Of course, marketers have always targeted racially defined customer-bases—typically to adjust price ranges along socio-economic lines. But with ever more data becoming available, the risk of ethical error becomes harder to avoid. In the digital age, customers are defined by where they click and by the information they register. It’s one thing to be labeled as more likely to buy a pair of ski gloves, but quite another to be thought of as less likely to pay a credit card bill just because you have an ethnic-sounding name.

There is very little regulation to protect consumers from data misuse in this regard, and little guidance for companies as to how data can be segmented and sold to third parties in ethical ways. As Frank Pasquale writes in his new book, The Black Box Society, some data-broker customer-targeting lists include such ethically questionable consumer categories as “probably bipolar,” and “gullible elderly.”

Ultimately, the owner of a supermarket doesn’t really care who shops there or what they buy—only about the total value of the basket. If he or she identifies, for example, that people who buy premium ice cream often also buy other premium items, such as high-margin wine and or gourmet snack foods, it would be a good strategy to discount the ice cream to increase the overall basket value.

But if an email marketing campaign for that same brand of ice cream goes out to a list of 1,000 customers, chosen by an algorithm, and none are from ethnic minorities, is the algorithm racist?

Let’s look at how the calculation is made: The algorithm might start with a million names and attempt to prune that list down to 1,000 by rejecting groups in which 80% of people aren’t interested in the offer, and selecting groups in which 80% are. Fairly crude criteria are employed to segment these groups, some of which could be race or gender-related.

Does this leave a company open to accusations of racism?

The answer is complicated, but a recent EU ruling may help clarify things. The European Court of Justice decided that car insurance firms could no longer consider a customer’s gender when calculating premiums. This was bad news for car insurers specialized toward female drivers, such as Sheilas’ Wheels in the UK, which built its business model on targeting women with lower premiums based on their superior safety record. In effect, the algorithm was telling the insurer that offering priority rates to women was the best way to price-risk. The European Court of Justice agreed, but ruled that the algorithm was sexist on principle.

The problem is that because the role of analytics is to aggregate, and aggregation by definition emphasizes group dynamics over individual traits, there is a strong possibility that the data will identify patterns which can be matched to specific minority groups.

In the case of our supermarket, there is of course no racist intention, but the analytics do reinforce societal unfairness, particularly if the supermarket targets more affluent customers.

Some campaigns can, of course, be monitored to ensure they don’t break the law. But as marketing becomes an increasingly individualized practice, it becomes tougher for companies to identify discriminatory segmentation until after campaigns have already launched. It’s a numbers game. Running a campaign across Europe, for example, might involve targeting 150 million adults. It would be impossible for a human to review each specific case, monitor every individual source of data. Ethically questionable patterns slip through.

Ultimately, the more decisions a company automates—the more data it uses—the more it risks engaging in potentially racist, sexist, or classist marketing.

After the European Court of Justice ruling, the market’s response was a lesson in itself. Launched in 2013, Drive Like a Girl, another UK insurance provider, charges premiums based on driver behavior alone. Data are taken from a black box in the engine, and if the telematics prove the customer drives smoothly and doesn’t speed, they are offered cheaper premiums. The end result is that, on aggregate, British women will receive cheaper insurance, but now based on individual behavior—not their very womanhood.

The challenge for other sectors is how to leverage the new power of analytics to act on behavior, as opposed to harmful stereotypes—and therefore, avoid damage to company reputations caused by ethical blunders. This, while helping to create a more harmonious, accepting society worldwide.