From 2014 until 2017, Amazon tried to build a tool that used artificial intelligence to rate the best job candidates like products on its website, from one to five stars, according to a new report from Reuters.
But a year into the project, the programers realized that the algorithmic system wasn’t good at identifying people’s job potential: It was just good at identifying men.
Artificial intelligence systems like the one Amazon tried to build typically require lots of data to learn. Gigabytes of image data are needed to train an algorithm to distinguish between a picture of a cat and a horse, and similarly Amazon’s algorithm would have needed historical data to tell what was desirable in a job candidate.
“Everyone wanted this holy grail,” one of the Amazon employees Reuters interviewed said. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”
That’s where the bias crept in, according to Reuters. The team decided to train the system on the previous 10 years of resumes sent to Amazon, which were mostly men. When the algorithm reached its conclusions for what would be good and what would be bad in an applicant, it mirrored the hiring biases towards men that Amazon had shown in the past. In 2014, Amazon released diversity numbers that revealed 63% of Amazon employees were male, and that number grew to 75% when only looking at managers.
The algorithmic system went so far as to penalize the word “women” on a resume, as in a women’s club or sport, and downgraded all-women’s colleges as less preferable. Amazon reportedly tried to make the algorithm more neutral, but there was no guarantee that it actually was less biased.
The algorithm was also just not very good: It would recommend jobs for applicants where they had no expertise or applicable skills. Amazon reportedly shut down the project after coming to this conclusion.
“This was never used by Amazon recruiters to evaluate candidates,” an Amazon spokesperson told Quartz.
This is far from the first example of an artificial intelligence system that only mirrors the biases found in its training data. The idea has been brought up in testimony to the US Congress, and has been acknowledged by tech companies like Google, IBM, and Microsoft. Bias along gender, ethnicity, and geographic lines have even been seen in healthcare applications of the technology.
Yet for all the discussion, there are no best practices or industry guidelines for developing AI technology without biases. That could change in the near future, though, as details about algorithms like Amazon’s are revealed amid growing congressional scrutiny into whether facial recognition and forms of AI making governmental decisions could break civil rights laws.
Update (Oct. 11): This story has been updated to include a comment from an Amazon spokesperson.