One day last summer, Microsoft’s director of artificial intelligence research, Eric Horvitz, activated the Autopilot function of his Tesla sedan. The car steered itself down a curving road near Microsoft’s campus in Redmond, Washington, freeing his mind to better focus on a call with a nonprofit he had cofounded around the ethics and governance of AI. Then, he says, Tesla’s algorithms let him down.

“The car didn’t center itself exactly right,” Horvitz recalls. Both tires on the driver’s side of the vehicle nicked a raised yellow curb marking the center line, and shredded. Horvitz had to grab the wheel to pull his crippled car back into the lane. He was unharmed, but the vehicle left the scene on the back of a truck, with its rear suspension damaged. Its driver left affirmed in his belief that companies deploying AI must consider new ethical and safety challenges. Tesla says Autopilot is intended for use by a fully attentive driver.

At Microsoft, Horvitz helped establish an internal ethics board in 2016 to help the company navigate potentially tricky spots with its own AI technology. The group is cosponsored by Microsoft’s president and most senior lawyer, Brad Smith. It has prompted the company to refuse business from corporate customers, and to attach conditions to some deals limiting the use of its technology.

Horvitz declined to provide details of those incidents, saying only that they typically involved companies asking Microsoft to build custom AI projects. The group has also trained Microsoft sales teams on applications of AI the company is wary of. And it helped Microsoft improve a cloud service for analyzing faces that a research paper revealed was much less accurate for black women than white men. “It's been heartening to see the engagement by the company and how seriously the questions are being taken,” Horvitz says. He likens what’s happening at Microsoft to an earlier awakening about computer security—saying it too will change how every engineer works on technology.

Many people are now talking about the ethical challenges raised by AI, as the technology extends into more corners of life. French President Emmanuel Macron recently told WIRED that his national plan to boost AI development would consider setting “ethical and philosophical boundaries.” New research institutes, industry groups, and philanthropic programs have sprung up.

Microsoft is among the smaller number of companies building formal ethics processes. Even some companies racing to reap profits from AI have become worried about moving too quickly. “For the past few years I’ve been obsessed with making sure that everyone can use it a thousand times faster,” says Joaquin Candela, Facebook’s director of applied machine learning. But as more teams inside Facebook use the tools, “I started to become very conscious about our potential blind spots.”

At Facebook’s annual developer conference this month, data scientist Isabel Kloumann described a kind of automatic adviser for the company’s engineers called Fairness Flow. It measures how machine-learning software analyzing data performs on different categories—say men and women, or people in different countries—to help expose potential biases. Research has shown that machine-learning models can pick up or even amplify biases against certain groups, such as women or Mexicans, when trained on images or text collected online.

Kloumann’s first users were engineers creating a Facebook feature where businesses post recruitment ads. Fairness Flow’s feedback helped them choose job recommendation algorithms that worked better for different kinds of people, she says. She is now working on building Fairness Flow and similar tools into the machine-learning platform used company-wide. Some data scientists perform similar checks manually; making it easier should make the practice more widespread. “Let's make sure before launching these algorithms that they don't have a disparate impact on people,” Kloumann says. A Facebook spokesperson said the company has no plans for ethics boards or guidelines on AI ethics.

'Let's make sure before launching these algorithms that they don't have a disparate impact on people.' Isabel Kloumann, Facebook

Google, another leader in AI research and deployment, has recently become a case study in what can happen when a company doesn’t seem to adequately consider the ethics of AI.