The first draft of the Neanderthal genome, published in 2010, came with some titillating news. It showed that 50,000 years ago, these ancient hominids interbred with the ancestors of many modern humans. If you have European or Asian ancestry, an estimated 1 to 4 percent of your DNA came from Neanderthals.

On the off chance that your mind hasn’t gone there, allow me: Our ancestors, looking pretty much like we do today, had sex with the short, extremely muscular, big-nosed, big-browed, big-headed Neanderthals. Were the differences between the two species mostly physical, with shared intellectual and cultural pursuits the subjects of their pillow talk? Or were Neanderthals violent, mute, and stupid, as so often depicted in popular culture? Or something in between?

Neanderthals almost certainly weren’t as brutish as assumed a century ago. Anthropologists now know that they used tools, made art, and may have talked. Still, nobody fully knows how their brains worked, or how their thinking was different from ours. The uncertainty is understandable considering the evidence. All scientists have to go on are the fossilized skulls the Neanderthals left behind.

Using a new and somewhat controversial (more on that later) method of analyzing these ancient skulls, scientists in England have proposed a theory about the structure of the Neanderthal brain. Although the brains of our ancestors and Neanderthals were about the same size, Neanderthals had larger brain areas related to vision and body control, according to a study out today in Proceedings of the Royal Society B: Biological Sciences.

This implies, the researchers say, that compared with our ancestors, Neanderthals had less brain space for dealing with other skills and behaviors. For example, if the Neanderthals had less brain area devoted to social cognition, it might explain why they traveled shorter distances, had fewer symbolic artifacts and lived in smaller communities.

“One of the implications of differing brain organization we propose is that Neanderthals had smaller social networks than modern humans because Neanderthals had smaller areas in their brains to deal with social complexity,” says investigator Eiluned Pearce, a graduate student working with experimental psychologist Robin Dunbar at the University of Oxford.

It’s an intriguing theory, no doubt. But some researchers wonder whether this isn’t paleo-phrenology*. Can crude anatomical relationships of the skull really reveal patterns of complex behavior?

View Images An endocast of a heavily mineralized cranium once belonging to that of Homo sapiens who lived about 130,000 years ago. Courtesy of Natural History Museum London.

Pearce’s team began with published data from a few dozen cranial ‘endocasts’, or rubber moldings made from the inside of skulls to show the shape of the outer brain. For this study, the researchers weren’t interested in the shape of the endocasts but rather their volume, to use as a proxy for brain size.

For each endocast, they also looked at the size of the eye sockets, or orbits. Studies of other primates have shown an interesting anatomical relationship: The bigger the eye, the bigger the visual cortex, the region at the back of the brain that interprets light signals from the retina to produce vision.

Comparing the endocasts made from 21 skulls of Neanderthals and 38 skulls of our ancestors, the researchers found that Neanderthals had larger orbits (after controlling for body size). That suggests that they also had larger eyes and visual cortices.

The findings agree with studies of endocast shape showing that Neanderthals had relatively larger occipital lobes (where the visual cortex resides) than our ancestors did, notes Emiliano Bruner, an anthropologist from the National Research Center on Human Evolution in Burgos, Spain. “We must seriously take into consideration that different human species may have had different cognitive capacities,” he says. “It is worth noting that ‘different’ does not mean worse or better, but just different.”

Why would the Neanderthals have larger eyes than our ancestors? The study suggests it’s because the Neanderthals evolved in Europe, at higher latitudes than hominids in Africa. At higher latitudes, they were exposed to lower light levels, requiring larger eyes for the same level of visual acuity. But other experts say this has nothing to do with vision. According to Bergmann’s Rule, species living in colder climates are larger than those living in warmer climates. “Humans at higher latitudes are bigger, and therefore have bigger orbits, than humans at lower latitudes,” says Trenton Holliday, an anthropologist at Tulane University.

Another problem, Holliday says, is that the researchers didn’t correct for the size of the face. Orbit size is known to increase with face size, and Neanderthals had larger faces than our ancestors did. “What I suspect is that if they correct for facial size, then the differences in relative size of the visual part of the brain will disappear,” he says.

The effect of face size “is definitely an avenue for further research,” Pearce says. But she doesn’t think it will make a difference. “Although overall body or face size might influence orbit size to some extent, a larger orbit still means a larger eye and therefore a larger visual cortex, which is our argument.”

But those are all technical concerns. The more interesting issue, to me, is the notion that the size of a brain area — the visual cortex, say — can say anything about how the Neanderthal brain worked. If there’s one thing that we’ve learned in the last century of neuroscience, it’s that the brain isn’t really modular. Yes, certain regions are specialized to process certain types of sensory inputs and are active during certain tasks. But they’re all part of distributed functional networks, and we’re nowhere near understanding how those networks lead to this or that behavior. Plus, we’ve learned from studies of injury that the brain is incredibly plastic, capable of finding several neural routes to carry out the same behavior.

So given all that, does it make sense to claim that Neanderthals didn’t have higher-order social cognition simply because their brains aren’t set up for it exactly like ours are?

*

Franz Gall, the founder of phrenology, had some things to say about the occipital lobe of female homo sapiens. According to the 2003 book Labeling People: “Gall also thought that, since women’s heads were larger in the back and their foreheads lower and smaller than those of men, they therefore sensed and judged differently, and their inferior organization made them superstitious.”

Postscript (11:52am EST):

The authors of the new study and one of my other sources have written me with responses to this post that I think are interesting and important, so I’ve copied them below.

The authors also asked me why I included the quote about phrenology at the end, and I think that’s a fair question. My intent was not to imply that this study is essentially phrenology, and I’m sorry if it came across that way. I guess my point is simply that we (in the media, but also scientists) must always be careful about how to interpret any particular finding. In this case, the study shows a contrast between the visual systems of Neanderthals and our ancestors. That could underlie a difference in their social processing, or it could very well not. The Gall example shows how these sorts of interpretations sometimes go too far. (I think today’s news coverage of this study makes my point pretty clearly.)

Now for what the real experts think:

Chris Stringer of the Natural History Museum (one of the study’s authors):

Re. larger face = larger eyes, these separately or together both point in the same direction of requiring more working space in the brain (somatic + visual)

“So given all that, does it make sense to claim that Neanderthals didn’t have higher-order social cognition simply because their brains aren’t set up for it exactly like ours are?”

No it doesn’t make sense to claim that, and I don’t think we claimed that – the implication instead is that, for example, Neas would not have been able to regulate such large social groups, and therefore would not have had the benefits of those larger social groups. A smaller size for the latter would have had implications for their level of social complexity and their ability to create, conserve and build on innovations.

I am surprised by the “debate” moving around on this paper. I mean, I agree we are dealing with inferences and speculation, but this is science, based on probability. Caution is recommended: as always. People find “normal” and fascinating making behavioural inferences from a single gene or molecule, but raise doubts for this complex analysis which takes into consideration so many factors. Furthermore, this study seems extremely detailed and careful when compared with usual standards in paleontology, which are often based on simply descriptions or basic statistics. Frequently, simple or even superficial approaches generate agreement or, at least, no criticism. In contrast if an approach is more complicated, it generates diffidence.