The computer programme that defeated the world Go champion taught itself how to improve its game by playing millions of matches against itself, according to the head of the Google subsidiary that developed the software.

Demis Hassabis, who co-founded DeepMind – the London-based unit that built the AlphaGo programme, said that he hoped to use the same technique to help Google improve its own products, such as its phone assistants and search engines.



“We call it deep reinforcement learning,” Hassabis said. “It’s the combination of deep learning, neural network stuff, with reinforcement learning: so learning by trial and error, and incrementally improving and learning from your mistakes and your errors, so that you improve your decisions.”

In AlphaGo’s case, that involved splitting itself in half and playing millions of matches against itself, learning from each victory and loss. In one day alone, AlphaGo was able to play itself more than a million times, gaining more practical experience than a human player could hope to gain in a lifetime. In essence, AlphaGo got better at Go simply by thinking extremely hard about the problem.

Hassabis was speaking to the Guardian on the eve of his trip to Seoul, where AlphaGo proceeded to win the best-of-five match by taking the first three games and the fifth, although South Korean Lee Sedol did win the fourth.

DeepMind initially taught AlphaGo how to play the game by taking a huge database of professional Go matches, and training a programme to try to predict what move would come next in any given situation. But the next stage of training was to take those models and improve them through trial and error.

The team also used the deep reinforcement approach to test different versions of the software in order to decide which approach was bearing fruit and which should be dropped.

“Sometimes we’ll play them against each other for a few weeks, sometimes for a day,” Hassabis said. “At some point that version does top out, there’s no further improvements that you can do, and you need some radical new approach.”

CEO of Google DeepMind, Demis Hassabis Photograph: Lee Jin-man/AP

DeepMind hopes to be able to bring a similar approach to bear on issues that Google is already attempting to tackle, such as voice recognition and comprehension.By 2015, Google had replaced more than 60 different internal systems that had previously used “handcrafted” code with a more intelligent model using deep learning networks of the sort DeepMind works on, according to DeepMind’s cofounder Mustafa Suleyman.

But in the long term, Hassabis hopes to put DeepMind to use fixing less prosaic problems with the world.

“Games have been a great testbed for developing and testing AI algorithms quickly and efficiently, but ultimately we want to apply these algorithms to important real world problems,” Hassabis said. “Because the methods we have used to create AlphaGo are general purpose, our hope is that one day they can be extended to help … scientists tackle some of society’s toughest and most pressing problems, from climate modelling to healthcare and disease analysis.”

DeepMind is also looking at releasing a version of the programme to run on home computers. “We haven’t figured out how, and we would need to optimise it a bit so it could fit on a normal machine … but certainly the intent is that this could be an amazing tool,” said Hassabis.

In its competition form, AlphaGo runs on Google’s cloud computer network, using 1,920 processors and a further 280 GPUs, specialised chips capable of performing simple calculations in staggering quantities, but a simpler version of the programme was built that could be run on one machine (albeit still one with 48 processors and eight GPUs).

Hassabis hopes that releasing AlphaGo to home players could revolutionise the game in the west. “If you’re a promising Go player in the west, and you get good quite quickly, there aren’t any 9-dan [the highest rank of player, equivalent to a Chess grandmaster] professionals that you can learn from, or go and get taught by.

“So if you have this very strong machine that you have at home, to train against, and to analyse your games on, like they do with chess computers now, I think we might see some really strong players in the west.

“And actually that’s one thing I hope about this game, and I think it’s already done that, this match, is that it has already brought Go to Europe and the US, which I think the game deserves.”

Hassabis says he’s fascinated to find out just how strong AlphaGo can actually become as a player. Although the machine beat Sedol in a best-of-five match, it didn’t manage to win every single game, which suggests that the system still has room for improvement.

“What level can Go actually be played at, in theory. What could the perfect player do? It’s incredible from a games perspective, and also from the AI perspective.

“How good are the best humans, given that we’ve played this game for thousands of years, very seriously? There were the Edo castle games in the Edo period in Japan: for 200 years there were professional Go players there. It was quite a big deal. And it is interesting to see: How good is Sedol, and Lee Chang-ho, and all these great players in history, relative to the maximum you could play at?

“What I can say to you is that we haven’t seen any ceiling in our improvement curves. We probably will carry on to some degree, and see where that tops out at. But the really interesting thing is, we haven’t seen a ceiling on performance yet, in tests that we’ve been doing. We haven’t hit that ceiling, or even approached it.”

