The parallel existence of an intelligent species closely related to us has long fascinated scientists and the public alike. The most debated issue is why Neandertals ultimately disappeared. Potential explanations include violent conflict with Homo sapiens, disease, difficulty adapting to rapid environmental and climate change, interbreeding with modern humans and differences between the two species in technical, social and cognitive abilities.

Now new research from a group led by paleoanthropologist Takeru Akazawa of Kochi University of Technology in Japan has shed light on the question using computational techniques to reconstruct the Neandertal brain and estimate differences in the size of specific regions between species. The results add weight to the idea that cognitive differences contributed to the Neandertals being out-competed and ultimately replaced by our ancestors.

The study, published April 5 in Scientific Reports, used fossils of four Neandertals and four early human skulls to estimate the shapes and structures of their brains. “Previous studies studied shape differences of braincases but there are no studies of reconstruction of the brain itself,” says biomechanical engineer and co-lead author Naomichi Ogihara of Keio University in Japan. “Our method allows estimation of the shape and volume of each brain region, which is quite impossible just by analyzing the endocranial surfaces.”

The team achieved this feat by averaging brain scans from 1,185 living humans to generate a model of the average human brain. “[The authors] deform a statistical model of the human brain onto Neandertals’ braincases, proposing a new method to delineate the brain in fossil species,” says paleoneurologist Emiliano Bruner of the National Research Center for Human Evolution in Spain who was not involved in the study. This allowed them to estimate what the brains of the two species may have looked like and how specific regions may have differed between them.

The results show that although there was no difference in the overall size of Neandertals’ brains, significant differences may have characterized the dimensions of specific regions, particularly the cerebellum. “This was surprising since the cerebellum is traditionally considered important for motor-related functions, Ogihara says. “We initially expected that the frontal lobe would be different between the two species because it has been considered to be related to higher cognitive functions, but it was not the case.”

The researchers, however, went further by analyzing correlations between brain scans and behavioral data in an existing database (from the Human Connectome Project). They found greater cerebellum volume is associated with abilities such as cognitive flexibility, attention, language processing and memory. “The paper gives the impression the cerebellum is intimately involved in a large number of higher cognitive functions, says evolutionary psychologist Robin Dunbar of the University of Oxford in England who was not involved in the work. “This isn’t strictly true—its function seems to be rather one of coordination between different brain units and cognitive processings—in effect, making sure computations are done in the right order. That role is almost certainly crucial to higher cognitive functions and allows us to do what we do,” he says. One reason for the importance of cerebellum volume, the authors suggest, is that unlike other regions it consists of a large array of identical processing units, so larger volumes logically equate to higher processing capacity.

One caveat, Bruner says, is the methods used in the study would be blind to any brain changes that occurred after Neandertals split from humans on the evolutionary tree. “Deforming a modern brain into a Neandertal one may obscure changes specific to the two lineages,” he says. “After their separation, the Neandertal and modern lineages could have undergone some specific changes that cannot be detected by this method.” The authors reason, however, that since chimpanzee and bonobo brains can be morphed into each other, and they diverged around two million years ago, the approach is a reasonable one, because Neandertals more recently diverged from humans some 700,000 years ago.

The findings do not conclusively prove what caused the Neandertals’ extinction but they do suggest brain differences probably contributed to their disappearance. “What we can say based on the present study is that innate differences in brain structure actually existed between the two species, possibly leading to differences in cognitive and social abilities,” Ogihara says. “Although the difference could be subtle, such a difference may become significant in terms of natural selection.”

Dunbar has previously shown a relationship between brain size and social group size in primates, including human social networks. Those findings involved parts of the cortex, but similar effects may be at work here. “If [the cerebellum’s function was reduced in Neandertals, it confirms that their cognitive abilities weren’t quite as advanced as those of modern humans,” he says. “That of course doesn’t make them any less human, or shambling ape-men, but it does mean their social and cultural capacities—the traits that survival really hinges on, especially in tough times—wouldn’t have been quite as effective as those of modern humans, which may well explain why they went extinct and our lineage didn’t.”

Ogihara says they would next like to develop their new methods by exchanging ideas with researchers working on human brain evolution. “We are also interested in applying our methodology to brain reconstruction of other hominins,” he says.