We review products independently, but we may earn affiliate commissions from buying links on this page. Terms of use.

Amazon had to shut down a computer algorithm used to recruit new staff because it developed a bias against women.

According to Reuters, which cited five people familiar with the program, the program was developed in 2014 with the goal to score job candidates based on their resumes.

However, after examining patterns from submitted applications over 10 years, most of which were from men, the system started penalizing women. Resumes that included the word "women's" or mention of all-women colleges received lower scores.

The artificial intelligence would rate the candidates out of five stars, similar to how products are ranked on Amazon's retail site. According to a source, "Everyone wanted this holy grail...they literally wanted it to be an engine where I'm going to give you 100 resumes, it will spit out the top five, and we'll hire those."

Amazon changed the algorithms to ensure that gender-related terms were ranked neutrally, but there was still the possibility that the bias shown by the machine would manifest through other means. The project was shut down in 2017.

Although more companies are implementing machine learning for things like harassment detection, this highlights why AI is not yet a magic bullet. We saw it with Microsoft's Tay Twitter bot, while Google's Perspective AI was recently found to be easily fooled by typos and extraneous words. In 2015, Google's Photos app also mistakenly tagged two black people as gorillas.