Jeff Hawkins recently re-read his 2004 book On Intelligence, where the founder of Palm computing – the company that gave us the first handheld computer and later, first-generation smartphones – explains how the human brain learns. An electrical engineer by training, Hawkins had taken a deep interest in how the brain works and founded the Redwood Neuroscience Institute, a private, nonprofit research organization focused on understanding how the neocortex processes information, at UC Berkeley in 2002.

The big surprise? “There was very little I would change about that book,” Hawkins says. “There’s a lot I would add. There’s a ton of stuff where I know exactly how it works, that I didn’t know when I wrote it.”

Today, Hawkins heads research at Redwood City, Calif.-based Numenta, a company building computer memory systems based on very basic relationships between neurons in the brain. The small, privately funded company was founded in 2005, largely based on the principles of Hawkins’ book, to build software that analyzes huge data sets to spot trends and anomalies. So far, Numenta’s algorithms have been put to work on data firehoses ranging from GPS coordinates to performance metrics on cloud computing systems. IBM recently assigned a 100-person team in its Almaden research lab to do further testing on Hawkins’ algorithms and look into designing next-gen computers that can run them, according to recent reports.

Along with other researchers, Hawkins believes that the brain’s structure holds the keys for the next generation of computers that will interpret our world with far more insight than our eyes and ears ever could. I chatted with Hawkins recently about how the biggest opportunities in deep learning all circle back to the human brain.

It never hurts to set definitions. In your words, what is intelligence?

We should start by acknowledging there is one system that we all agree is intelligence and that’s the human brain, and maybe some other mammals as well. That is the gold standard, if you will, the defining structure that we are trying to understand.

So then the question is: what does the human brain do that makes intelligence? Intelligence is a methodology of learning in the world. It’s not learning. It’s not a set of knowledge. It’s about a methodology for learning. Our brains use a particular type of learning to discover the structure in the world, and the universe, and in everything we do. It also has particular levels of sophistication. A monkey can be intelligent. A dog can be intelligent. A human can be intelligent. But they all use the same learning methodologies.

Do the neuron and the brain represent the data and the software from a machine intelligence point of view?

The neocortex is the part of the brain that we are concerned about as it relates to human intelligence. It is a memory structure, a type of memory system. It has operating principles, which you could call software if you want to call it that. But it’s a memory system and what it stores is what it learns. It doesn’t come in with any prior knowledge about the world. You have to learn to do everything. You have to learn how to see. When you’re born you can’t do anything. You have to learn the world from scratch. Our intelligence system starts out with no knowledge.

It’s that learning process that makes intelligence. We continue to learn for our entire life. Intelligence isn’t something you reach. It’s a continuous process you go through.

The key component of how the brain learns is through action. You don’t learn by sitting and absorbing things. You have to manipulate the world, move through the world, touch something. This is an essential aspect of how brains work. You can’t really separate out the motor component from the rest of it. And so today when we talk about AI, that’s why there’s such a huge difference between what biological intelligence is, and what machine learning technologies do today.

You’ve talked about the concept of “universal algorithms” in the past. Can you explain what that means?

The organism of intelligence is the neocortex. It basically solves all these different things we do with the same basic methodology, the same basic algorithm. The neural tissue that implements vision, language, and touch—they all work on the same principle. From the brain’s point of view, there’s nothing really different about vision and touch and hearing. It’s mathematics. This is why we are capable of doing so many things we didn’t evolve to do.

Now of course it’s a little more complicated than that. But the basic idea is correct. So when we build a technology on machine intelligence, we don’t have to build a vision system that works on one method, and a language system that works on another method, and a robotics system that works on another method. The biology tells us they’re all basically the same, and so we can build a common substrate that relies on this.

We’ve already proven this to be true. We’ve taken these cortical algorithms and applied them to new types of data that the brain’s never been exposed to and they work great. We’ve been able to make an encoding for GPS signals. So we can take GPS coordinates and feed it into an artificial brain and it understands, no problem.

The brain has the amazing property of plasticity. It can relearn tasks and do different things. You can lose a finger and the brain will reconfigure the part that controlled the finger. Is there an analogy for hardware and software?

We understand this plasticity well. Our systems exhibit this plasticity. I’m not speaking in speculation here.

In the brain, the whole thing is made of neurons. At any point in time there are a few percent of the cells that are active. And most of them are inactive. This is what we call sparse encoding. Because the brain has this distributed code, any part is not essential. You can lose some cells and the system continues to work, because it’s a population of cells that matter.

This is different than a computer where every transistor and resistor makes a difference. Lose a transistor and the machine stops running. In the brain, this is a distributed system so if some parts fail, other parts can continue to work.

The other part of plasticity comes from when the brain is running, it’s forming new connections. Therefore if it can’t connect one way, it’ll connect another way. It’s constantly wiring itself in the brain.

I have a pretty clear idea of how this will be done in silica. We will have memory systems built in silica that work on the same principles as the brain. Those memory systems will be distributed. Parts of the memory system talk with other parts of the memory system so if some part craps out, not only can things keep working but it will route the information to new areas. It can recover. Like the way a brain can recover after some trauma. We’ve exhibited this in our software. We can destroy parts and it recovers.

Many experts are banking on computer vision as one of the biggest applications for AI. What’s your take?

What people are doing today in computer vision isn’t really vision—they’re doing image classification. That’s a subset of vision. “Here’s an image; what is it?”

When I think about vision, it’s not like that at all. I’m walking around, turning my head; I’m looking out the window right now. The inputs are coming into my brain, little discs changing every 10 milliseconds or so. That’s a different problem. That’s not image classification.

That’s valuable, but I don’t think it’ll lead to explosive machine intelligence. I think it’ll lead to a lot of people doing image classification. A huge amount of the neocortex is dedicated to solving the vision problem. To do what a human does with real vision requires an amount of hardware—neurons and memory—that we can’t even come close to approximating in software today.

When you think about the growth curves in AI computing, what limits do you see?

At the moment we’re not even close to seeing the limits of what we can do. Some people say, “How big a brain can you make?” My answer is: I don’t see why I can’t make one that’s 100,000 times bigger than a human’s. I don’t see why I can’t make one that runs a million times faster. I don’t see a reason that I can’t interface it to 100 different sensors as opposed to three or four. We’re talking far out in the future here. But I don’t see the limits.

Perhaps the biggest limits are training times. As you make a bigger brain, you still have to train it. We have to train humans and look how long it takes to train them. Our brains are learning continuously for years and years. This is the issue for us [at Numenta] too. We spend most of our time training the system. Once you build a system, you’ve got to feed in the data. That’s where most of the CPU cycles go, and memory and so on. A lot of the issue will be how we accelerate that.