If you’re worried about omniscient computers, you should read Levesque on the subject of GOFAI. Computer scientists have still not answered fundamental questions that occupied McCarthy and Minsky. How might a computer detect, encode, and process not just raw facts but abstract ideas and beliefs, which are necessary for intuiting truths that are not explicitly expressed?

Levesque uses this example: suppose I ask you how a crocodile would perform in the steeplechase. You know from your experience of the world that crocodiles can’t leap over high hedges, so you’d know the answer to the question is some variant of “Badly.”

What if you had to answer that question in the way a computer can? You could scan all the world’s text for the terms “crocodile” and “steeplechase,” find no instances of the words’ being mentioned together (other than what exists now, in references to Levesque’s work), and then presume that a crocodile has never competed in the steeplechase. So you might gather that it would be impossible for a croc to do so. Good work—this time. You would have arrived at the right answer without knowing why. You would have used a flawed and brittle method that is likely to lead to ridiculous errors.

So while machine-learning technologies are making it possible to automate many tasks humans have traditionally done, there are important limits to what this approach can do on its own—and there is good reason to expect human labor to be necessary for a very long time.

Reductionism

Hold on, you might say: just because no one has a clue now about how to get machines to do sophisticated reasoning doesn’t mean it’s impossible. What if somewhat smart machines can be used to design even smarter machines, and on and on until there are machines powerful enough to model every last electrical signal and biochemical change in the brain? Or perhaps another way of creating a flexible intelligence will be invented, even if it’s not much like biological brains. After all, when you boil it all down (really, really, really down), intelligence arises from particular arrangements of quarks and other fundamental particles in our brains. There’s nothing to say such arrangements are possible only inside biological material made from carbon atoms.

This is the argument running through Life 3.0: Being Human in the Age of Artificial Intelligence, by MIT physics professor Max Tegmark. Tegmark stays clear of predicting when truly intelligent machines will arrive, but he suggests that it’s just a matter of time, because computers tend to improve at exponential rates (although that’s not necessarily true—see “The Seven Deadly Sins of AI Predictions”). He’s generally excited about the prospect, because conscious machines could colonize the universe and make sure it still has meaning even after our sun dies and humans are snuffed out.

Tegmark says the “near-term opportunities for AI to benefit humanity” are “spectacular”—“if we can manage to make it robust and unhackable.”

Tegmark comes from a humanistic point of view. He cofounded the nonprofit Future of Life Institute to support research into making sure AI is beneficial. Elon Musk, who has said AI might be more dangerous than nuclear weapons, put up $10 million. Tegmark is understandably worried about whether AI will be used wisely, safely, and fairly, and whether it will warp our economy and social fabric. He takes pains to explain why autonomous weapons should never be allowed. So I’m not inclined to criticize him. Nonetheless, he’s not very convincing in his proposition that computers could take over the world.

Tegmark laments that some Hollywood depictions of AI are “silly” but nonetheless asks readers to play along with an oversimplified fictional sketch of how an immensely powerful AI could elude the control of its creators. Inside a big tech company is an elite group of programmers called the Omegas who set out to build a system with artificial general intelligence before anyone else does. They call this system Prometheus. It’s especially good at programming other AI systems, and it learns about the world by reading “much of the Web.”

Set aside any quibbles you may have about that last part—given how much knowledge is not on the Web or digitized at all—and the misrepresentations of the world that would come from reading all of Twitter. The reductionism gets worse.

As Tegmark’s hypothetical story continues, Prometheus piles up money for its creators, first by performing most of the tasks on Amazon’s Mechanical Turk online marketplace, and then by writing software, books, and articles and creating music, shows, movies, games, and online educational courses. Forget hiring and directing actors; Prometheus makes video footage with sophisticated rendering software. To understand which screenplays people will find entertaining, it binge-watches movies humans have made and inhales all of Wikipedia.