In 2014, the US announced a new effort to understand the brain. Soon, we would map every single connection within the brain, track the activity of individual neurons, and start to piece together some of the fundamental units of biological cognition. The program was named BRAIN (for Brain Research through Advancing Innovative Neurotechnologies), and it posited that we were on the verge of these breakthroughs because both imaging and analysis hardware were finally powerful enough to produce the necessary data, and we had the software and processing power to make sense of it.

But this week, PLoS Computational Biology published a cautionary note that suggests we may be getting ahead of ourselves. Part experiment, part polemic, a computer scientist got together with a biologist to apply the latest neurobiology approaches to a system we understand far more completely than the brain: a processor booting up the games Donkey Kong and Space Invaders. The results were about as awkward as you might expect, and they helped the researchers make their larger point: we may not understand the brain well enough to understand the brain.

On the surface, this may sound a bit ludicrous. But it gets at something fundamental to the nature of science. Science works on the basis of having models that can be used to make predictions. You can test those models and use the results to refine them. And you have to understand a system on at least some level to build those models in the first place.

To give an example, imagine trying to figure out the strange, quantum behavior of an electron if we hadn't already done a detailed characterization of electrons, understood wave mechanics, and argued for centuries over whether light was a wave or a particle. Basic facts and an intellectual framework have to be in place before you can start building models and using them to tell you what other data you need. Are we at that point with the brain? If we could map every functional unit and connection in the brain and track their activity, would we have the tools to make sense of what we've discovered?

That's where Donkey Kong comes in.

Games on early Atari systems were powered by the 6502 processor, also found in the Apple I and Commodore 64. The two authors of the new paper (Eric Jonas and Konrad Paul Kording) decided to take this relatively simple processor and apply current neuroscience techniques to it, tracking its activity while loading these games. The 6502 is a good example because we can understand everything about the processor and use that to see how well the results match up. And, as they put it, "most scientists have at least behavioral-level experience with these classical video game systems."

So they built upon the work of the Visual 6502 project, which got ahold of a batch of 6502s, decapped them, and imaged the circuitry within. This allowed the project to build an exact software simulator with which they could use to test neuroscience techniques. But it also enabled the researchers to perform a test of the field of "connectomics," which tries to understand the brain by mapping all the connections of the cells within it.

To an extent, the fact that their simulator worked is a validation of the approach. But, at the same time, the chip is incredibly simple: there is only one type of transistor, as opposed to the countless number of specialized cells in the brain. And the algorithms used to analyze the connections only got the team so far; lots of human intervention was required as well. "Even with the whole-brain connectome," Jonas and Kording conclude, "extracting hierarchical organization and understanding the nature of the underlying computation is incredibly difficult."

They then used the simulator to try out a variety of approaches that have been used in neurobiology. The first is termed a lesion analysis, where they disable individual transistors and see what happens. While this was great for understanding which transistors were essential for which game, it actually didn't tell them much at all about how the processor operated. And, in fact, the results were largely artifacts. Even though they could identify transistors that were essential for one game or another, "a given transistor is obviously not specialized for Donkey Kong or Space Invaders."

In other words, at least when applied to processors, the approach produced results that relied nearly entirely on the implementation of a given game.

They then turned to spike analysis. Rather than switching between on and off states, neurons convey information through a collection of pulses of activity, or spikes. The authors considered the on-off transitions of each transistor as a spike, and subjected it to the same sorts of analysis as we'd use on neurons. When tested, they were able to find correlations between the spikes of some transistors and how bright the most recently drawn pixel was. But guessing what the significance might be without having a detailed understanding of the software was impossible.

(To be honest, this test wasn't that compelling. There are really no parallels between a transistor switching state and an individual neural spike, so you wouldn't expect the analysis to tell you anything in the first place.)

The team next analyzed activity in larger regions of the processor and showed that the average activities of these regions produced data that looked similar to what was gathered from functional MRI scans of the brain. But again, a lot of this was simply an artifact of the software implementation rather than telling us anything about the flow of information within the processor. They were also able to spot synchronized activity in different regions—exactly what you'd expect in a processor driven by a clock. But we also see this in the brain, where we're not sure about whether they're central to the activity or simply a byproduct of whatever processes neurons use.

Overall, the authors generally found that neurobiological approaches produced data that looked interesting but didn't actually tell them anything. "We have found that the standard data analysis techniques produce results that are surprisingly similar to the results found about real brains," they conclude. "However, in the case of the processor, we know its function and structure and our results stayed well short of what we would call a satisfying understanding."

On some level, this is all trivial. Brains and computers are different, so you wouldn't expect the tools designed to understand one to work when applied to the other.

But it also shows just how much work we need to do for our models to become more sophisticated. We can understand transistors, processors, and software fully because we made them. And even then, it's hard work to understand what's going on in a processor when a simple game loads. In contrast, there are gaps in our understanding at every level of neurobiology, from how individual neurons function, through how small groups of neurons interact, and all the way up to how information flows within the brain.

Given that situation, the authors argue, it's not clear whether all the data that will be flowing in from the BRAIN project will help us as much as we'd like.

PLoS Computational Biology, 2016. DOI: 10.1371/journal.pcbi.1005268 (About DOIs)

Correction: noted the role of the Visual 6502 project.