Intelligent machines catastrophically misinterpreting human desires is a frequent trope in science fiction, perhaps used most memorably in Isaac Asimov’s stories of robots that misconstrue the famous “three laws of robotics.” The idea of artificial intelligence going awry resonates with human fears about technology. But current discussions of superhuman A.I. are plagued by flawed intuitions about the nature of intelligence.

We don’t need to go back all the way to Isaac Asimov — there are plenty of recent examples of this kind of fear. Take a recent Op-Ed essay in The New York Times and a new book, “Human Compatible,” by the computer scientist Stuart Russell. Dr. Russell believes that if we’re not careful in how we design artificial intelligence, we risk creating “superintelligent” machines whose objectives are not adequately aligned with our own.

As one example of a misaligned objective, Dr. Russell asks, “What if a superintelligent climate control system, given the job of restoring carbon dioxide concentrations to preindustrial levels, believes the solution is to reduce the human population to zero?” He claims that “if we insert the wrong objective into the machine and it is more intelligent than us, we lose.”

Dr. Russell’s view expands on arguments of the philosopher Nick Bostrom, who defined A.I. superintelligence as “an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills.” Dr. Bostrom and Dr. Russell envision a superintelligence with vast general abilities unlike today’s best machines, which remain far below the level of humans in all but relatively narrow domains (such as playing chess or Go).