Measuring the intelligence of AI is one of the trickiest but most important questions in the field of computer science. If you can’t understand whether the machine you’ve built is cleverer today than it was yesterday, how do you know you’re making progress?

At first glance, this might seem like a non-issue. “Obviously AI is getting smarter” is one reply. “Just look at all the money and talent pouring into the field. Look at the milestones, like beating humans at Go, and the applications that were impossible to solve a decade ago that are commonplace today, like image recognition. How is that not progress?”

What does intelligence mean for developing AI?

Another reply is that these achievements aren’t really a good gauge of intelligence. Beating humans at chess and Go is impressive, yes, but what does it matter if the smartest computer can be out-strategized in general problem-solving by a toddler or a rat?

This is a criticism put forward by AI researcher François Chollet, a software engineer at Google and a well-known figure in the machine learning community. Chollet is the creator of Keras, a widely used program for developing neural networks, the backbone of contemporary AI. He’s also written numerous textbooks on machine learning and maintains a popular Twitter feed where he shares his opinions on the field.

In a recent paper titled “On the Measure of Intelligence,” Chollet also laid out an argument that the AI world needs to refocus on what intelligence is and isn’t. If researchers want to make progress toward general artificial intelligence, says Chollet, they need to look past popular benchmarks like video games and board games, and start thinking about the skills that actually make humans clever, like our ability to generalize and adapt.

In an email interview with The Verge, Chollet explained his thoughts on this subject, talking through why he believes current achievements in AI have been “misrepresented,” how we might measure intelligence in the future, and why scary stories about super intelligent AI (as told by Elon Musk and others) have an unwarranted hold on the public’s imagination.

This interview has been lightly edited for clarity.

In your paper, you describe two different conceptions of intelligence that have shaped the field of AI. One presents intelligence as the ability to excel in a wide range of tasks, while the other prioritizes adaptability and generalization, which is the ability for AI to respond to novel challenges. Which framework is a bigger influence right now, and what are the consequences of that?

In the first 30 years of the history of the field, the most influential view was the former: intelligence as a set of static programs and explicit knowledge bases. Right now, the pendulum has swung very far in the opposite direction: the dominant way of conceptualizing intelligence in the AI community is the “blank slate” or, to use a more relevant metaphor, the “freshly-initialized deep neural network.” Unfortunately, it’s a framework that’s been going largely unchallenged and even largely unexamined. These questions have a long intellectual history — literally decades — and I don’t see much awareness of this history in the field today, perhaps because most people doing deep learning today joined the field after 2016.

It’s never a good thing to have such intellectual monopolies, especially as an answer to poorly understood scientific questions. It restricts the set of questions that get asked. It restricts the space of ideas that people pursue. I think researchers are now starting to wake up to that fact.

In your paper, you also make the case that AI needs a better definition of intelligence in order to improve. Right now, you argue, researchers focus on benchmarking performance in static tests like beating video games and board games. Why do you find this measure of intelligence lacking?

The thing is, once you pick a measure, you’re going to take whatever shortcut is available to game it. For instance, if you set chess-playing as your measure of intelligence (which we started doing in the 1970s until the 1990s), you’re going to end up with a system that plays chess, and that’s it. There’s no reason to assume it will be good for anything else at all. You end up with tree search and minimax, and that doesn’t teach you anything about human intelligence. Today, pursuing skill at video games like Dota or StarCraft as a proxy for general intelligence falls into the exact same intellectual trap.

This is perhaps not obvious because, in humans, skill and intelligence are closely related. The human mind can use its general intelligence to acquire task-specific skills. A human that is really good at chess can be assumed to be pretty intelligent because, implicitly, we know they started from zero and had to use their general intelligence to learn to play chess. They weren’t designed to play chess. So we know they could direct this general intelligence to many other tasks and learn to do these tasks similarly efficiently. That’s what generality is about.

Assumptions about human intelligence shouldn’t apply to machines

But a machine has no such constraints. A machine can absolutely be designed to play chess. So the inference we do for humans — “can play chess, therefore must be intelligent” — breaks down. Our anthropomorphic assumptions no longer apply. General intelligence can generate task-specific skills, but there is no path in reverse, from task-specific skill to generality. At all. So in machines, skill is entirely orthogonal to intelligence. You can achieve arbitrary skills at arbitrary tasks as long as you can sample infinite data about the task (or spend an infinite amount of engineering resources). And that will still not get you one inch closer to general intelligence.

The key insight is that there is no task where achieving high skill is a sign of intelligence. Unless the task is actually a meta-task, that involves acquiring new skills over a broad [range] of previously unknown problems. And that’s exactly what I propose as a benchmark of intelligence.

If these current benchmarks don’t help us develop AI with more generalized, flexible intelligence, why are they so popular?

There’s no doubt that the effort to beat human champions at specific well-known video games is primarily driven by the press coverage these projects can generate. If the public wasn’t interested in these flashy “milestones” that are so easy to misrepresent as steps toward superhuman general AI, researchers would be doing something else.

Focus on AI beating video games is “driven by the press coverage”

I think it’s a bit sad because research should about answering open scientific questions, not generating PR. If I set out to “solve” Warcraft III at a superhuman level using deep learning, you can be quite sure that I will get there as long as I have access to sufficient engineering talent and computing power (which is on the order of tens of millions of dollars for a task like this). But once I’d have done it, what would I have learned about intelligence or generalization? Well, nothing. At best, I’d have developed engineering knowledge about scaling up deep learning. So I don’t really see it as scientific research because it doesn’t teach us anything we didn’t already know. It doesn’t answer any open question. If the question was, “Can we play X at a superhuman level?,” the answer is definitely, “Yes, as long as you can generate a sufficiently dense sample of training situations and feed them into a sufficiently expressive deep learning model.” We’ve known this for some time. (I actually said as much a while before the Dota 2 and StarCraft II AIs reached champion level.)

What do you think the actual achievements of these projects are? To what extent are their results misunderstood or misrepresented?

One stark misrepresentation I’m seeing is the argument that these high-skill game-playing systems represent real progress toward “AI systems, which can handle the complexity and uncertainty of the real world” [as OpenAI claimed in a press release about its Dota 2-playing bot OpenAI Five]. They do not. If they did, it would be an immensely valuable research area, but that is simply not true. Take OpenAI Five, for instance: it wasn’t able to handle the complexity of Dota 2 in the first place because it was trained with 16 characters, and it could not generalize to the full game, which has over 100 characters. It was trained over 45,000 years of gameplay — then again, note how training data requirements grow combinatorially with task complexity — yet, the resulting model proved very brittle: non-champion human players were able to find strategies to reliably beat it in a matter of days after the AI was made available for the public to play against.

If you want to one day become able to handle the complexity and uncertainty of the real world, you have to start asking questions like, what is generalization? How do we measure and maximize generalization in learning systems? And that’s entirely orthogonal to throwing 10x more data and compute at a big neural network so that it improves its skill by some small percentage.

So what would be a better measure of intelligence for the field to focus on?

In short, we need to stop evaluating skill at tasks that are known beforehand — like chess or Dota or StarCraft — and instead start evaluating skill-acquisition ability. This means only using new tasks that are not known to the system beforehand, measuring the prior knowledge about the task that the system starts with, and measuring the sample-efficiency of the system (which is how much data is needed to learn to do the task). The less information (prior knowledge and experience) you require in order to reach a given level of skill, the more intelligent you are. And today’s AI systems are really not very intelligent at all.

In addition, I think our measure of intelligence should make human-likeness more explicit because there may be different types of intelligence, and human-like intelligence is what we’re really talking about, implicitly, when we talk about general intelligence. And that involves trying to understand what prior knowledge humans are born with. Humans learn incredibly efficiently — they only require very little experience to acquire new skills — but they don’t do it from scratch. They leverage innate prior knowledge, besides a lifetime of accumulated skills and knowledge.

[My recent paper] proposes a new benchmark dataset, ARC, which looks a lot like an IQ test. ARC is a set of reasoning tasks, where each task is explained via a small sequence of demonstrations, typically three, and you should learn to accomplish the task from these few demonstrations. ARC takes the position that every task your system is evaluated on should be brand-new and should only involve knowledge of a kind that fits within human innate knowledge. For instance, it should not feature language. Currently, ARC is totally solvable by humans, without any verbal explanations or prior training, but it is completely unapproachable by any AI technique we’ve tried so far. That’s a big flashing sign that there’s something going on there, that we’re in need of new ideas.

Do you think the AI world can continue to progress by just throwing more computing power at problems? Some have argued that, historically, this has been the most successful approach to improving performance. While others have suggested that we’re soon going to see diminishing returns if we just follow this path.

This is absolutely true if you’re working on a specific task. Throwing more training data and compute power at a vertical task will increase performance on that task. But it will gain you about zero incremental understanding of how to achieve generality in artificial intelligence.

If you have a sufficiently large deep learning model, and you train it on a dense sampling of the input-cross-output space for a task, then it will learn to solve the task, whatever that may be — Dota, StarCraft, you name it. It’s tremendously valuable. It has almost infinite applications in machine perception problems. The only problem here is that the amount of data you need is a combinatorial function of task complexity, so even slightly complex tasks can become prohibitively expensive.

Take self-driving cars, for instance. Millions upon millions of training situations aren’t sufficient for an end-to-end deep learning model to learn to safely drive a car. Which is why, first of all, L5 self-driving isn’t quite there yet. And second, the most advanced self-driving systems are primarily symbolic models that use deep learning to interface these manually engineered models with sensor data. If deep learning could generalize, we’d have had L5 self-driving in 2016, and it would have taken the form of a big neural network.

Lastly, given you’re talking about constraints for current AI systems, it seems worth asking about the idea of superintelligence — the fear that an extremely powerful AI could cause extreme harm to humanity in the near future. Do you think such fears are legitimate?

No, I don’t believe the superintelligence narrative to be well-founded. We have never created an autonomous intelligent system. There is absolutely no sign that we will be able to create one in the foreseeable future. (This isn’t where current AI progress is headed.) And we have absolutely no way to speculate what its characteristics may be if we do end up creating one in the far future. To use an analogy, it’s a bit like asking in the year 1600: “Ballistics has been progressing pretty fast! So, what if we had a cannon that could wipe out an entire city. How do we make sure it would only kill the bad guys?” It’s a rather ill-formed question, and debating it in the absence of any knowledge about the system we’re talking about amounts, at best, to a philosophical argument.

“We don’t need superintelligence in order for certain AI applications to represent a danger.”

One thing about these superintelligence fears is that they mask the fact that AI has the potential to be pretty dangerous today. We don’t need superintelligence in order for certain AI applications to represent a danger. I’ve written about the use of AI to implement algorithmic propaganda systems. Others have written about algorithmic bias, the use of AI in weapons systems, or about AI as a tool of totalitarian control.

There’s a story about the siege of Constantinople in 1453. While the city was fighting off the Ottoman army, its scholars and rulers were debating what the sex of angels might be. Well, the more energy and attention we spend discussing the sex of angels or the value alignment of hypothetical superintelligent AIs, the less we have for dealing with the real and pressing issues that AI technology poses today. There’s a well-known tech leader that likes to depict superintelligent AI as an existential threat to humanity. Well, while these ideas are grabbing headlines, you’re not discussing the ethical questions raised by the deployment of insufficiently accurate self-driving systems on our roads that cause crashes and loss of life.

If one accepts these criticisms — that there is not currently a technical grounding for these fears — why do you think the superintelligence narrative is popular?

Ultimately, I think it’s a good story, and people are attracted to good stories. It’s not a coincidence that it resembles eschatological religious stories because religious stories have evolved and been selected over time to powerfully resonate with people and to spread effectively. For the very same reason, you also find this narrative in science fiction movies and novels. The reason why it’s used in fiction, the reason why it resembles religious narratives, and the reason why it has been catching on as a way to understand where AI is headed are all the same: it’s a good story. And people need stories to make sense of the world. There’s far more demand for such stories than demand for understanding the nature of intelligence or understanding what drives technological progress.