"The traditional separation of the agent from its environment seems even less attractive when one considers... it may become better than any human at the task of making itself even smarter, leading to an 'intelligence explosion' and leaving human intelligence far behind," wrote Benja Fallenstein and Nate Soares in a paper for the Machine Intelligence Research Institute. "It seems plausible that an [artificial general intelligence] undergoing an intelligence explosion may eventually want to adopt an architecture radically different from its initial one, such as one distributed over many different computers, where no single entity fulfills the agent’s role from the traditional framework."

These questions of computational reasoning represent a complex problem without a clear solution: How do we build machines that will make the world better, even when they start running themselves? And, perhaps the bigger question therein, what does a better world actually look like? Because if we teach machines to reflect on their actions based on today's human value systems, they may soon be outdated themselves. Here's how MIRI researchers Luke Muehlhauser and Nick Bostrom explained it in a paper last year:

Suppose that the ancient Greeks had been the ones to face the transition from human to machine control, and they coded their own values as the machines’ final goal. From our perspective, this would have resulted in tragedy, for we tend to believe we have seen moral progress since the Ancient Greeks (e.g. the prohibition of slavery). But presumably we are still far from perfection. We therefore need to allow for continued moral progress. One proposed solution is to give machines an algorithm for figuring out what our values would be if we knew more, were wiser, were more the people we wished to be, and so on. Philosophers have wrestled with this approach to the theory of values for decades, and it may be a productive solution for machine ethics.

Experts say we are running out of time to figure out the answers to these questions. Again, Bill Gates: "There will be more progress in the next 30 years than ever. Even in the next 10 problems like vision and speech understanding and translation will be very good. Mechanical robot tasks like picking fruit or moving a hospital patient will be solved. Once computers/robots get to a level of capability where seeing and moving is easy for them then they will be used very extensively."

Fruit-picking robots, though, do not a humanity-replacing singularity make. Or not right away anyway. Humans are actually—go figure—pretty bad at predicting AI timelines, according to MIRI. For decades, people have been predicting the rise of artificial intelligence is just around the corner, 15 to 25 years away.

"Expert predictions are not only indistinguishable from non-expert predictions, they are also indistinguishable from past failed predictions," wrote the authors of a report on failed predictions. "Hence it is not unlikely that recent predictions are suffering from the same biases and errors as their predecessors."

We want to hear what you think about this article. Submit a letter to the editor or write to letters@theatlantic.com.