We won't ever come face to face with a real-life Neanderthal. They went extinct thousands of years ago. All we can do is use their remains to reconstruct what they were like.

In many ways they were a lot like us. In fact they were so similar, our species actually interbred with theirs.

Nevertheless there were some differences. One stands out: they had weirdly large eyes.

On the face of it, big eyes sound like a good thing. Presumably, having bigger eyes meant the Neanderthals could see better than us.

But according to one controversial theory, Neanderthals' big eyes played a key role in their demise.

Neanderthals were around before we evolved. They first appeared around 250,000 years ago and spread throughout Europe and Asia.

Our own species, Homo sapiens, evolved in Africa about 200,000 years ago. They reached Europe around 45,000 years ago, and found it was inhabited by Neanderthals.

Both their eyes and their brain's visual system were larger than ours

We co-existed with them for 5,000 years, according to the latest estimate. But eventually they disappeared, perhaps as early as 40,000 years ago.

In 2013, a team led by Eiluned Pearce of the University of Oxford in the UK proposed a radical explanation: their eyes were to blame.

From a detailed analysis of modern human and Neanderthal skulls, Pearce found that both their eyes and their brain's visual system were larger than ours.

Their big eyes meant that they devoted a larger part of their brain to seeing.

However, Pearce suggests that this came at a cost to their social world. Other parts of their brain would in turn have been smaller.

We all get by with help from our friends, but Neanderthals did not have enough friends to help out

"Since Neanderthals evolved at higher latitudes and also have bigger bodies than modern humans, more of the Neanderthal brain would have been dedicated to vision and body control, leaving less brain to deal with other functions like social networking," Pearce said at the time.

The theory goes that, unlike us, they could not devote large parts of their brain to developing complex social networks. So when they were faced with major threats, such as a changing climate or competition from modern humans, they were at a disadvantage.

Teamwork would have been vital in these situations, so if they lacked the ability to form large groups, they would not have had the support they needed. We all get by with help from our friends, but Neanderthals did not have enough friends to help out.

"The substantive issue is not the opening through which the eye peers, but the area of the retina at the back," says co-author Robin Dunbar, also from the University of Oxford.

Our species, on the other hand, evolved in Africa where there is plenty of light

This area is so important because it records all the incoming light from the world. Neanderthals lived in northern regions where the light was dimmer, and their large eyes may have helped them to see better.

"To see more clearly, you need to gather more light into the eye, and that means having a bigger retina," says Dunbar. "The size of the retina is determined by the size of the eyeball."

Because of this, Dunbar and Pearce argue, a bigger "computer" was needed to process all this additional visual information. "By analogy, there is no point in having an incredibly large radio telescope attached to a tiny computer that gets overwhelmed by the information coming in," Dunbar says.

Our species, on the other hand, evolved in Africa where there is plenty of light. We did not need such a large visual processing system. Instead we evolved a bigger frontal lobe, allowing us to develop more complex social lives.

It's a neat story. But other biologists are far from convinced, and some of them have set out to unpick the idea.

They have now published their findings in the American Journal of Physical Anthropology. The new analysis suggests that Neanderthals' large eyes did not contribute to their extinction after all.

We actually think that eyes have nothing to do with social groups

John Hawks of the University of Wisconsin-Madison and his colleagues looked at 18 living primate species, to find out whether the size of their eye sockets was linked to the size of their social groups.

Rather than bigger eyes resulting in smaller social groups, they found that the opposite was true. "Big eyes actually indicate bigger social groups in other primates," says Hawks.

"If we could believe that logic, we would expect Neanderthals to be better social animals than we are today. Now, we don't believe any of it: we actually think that eyes have nothing to do with social groups."

To truly understand how Neanderthals socialised with each other, we would be better off looking at clues from the archaeological record, says Hawks. These clues show "that they were sophisticated social beings", not socially-inept loners.

There are other reasons to question Pearce and Dunbar's idea.

Neanderthals in general were slightly larger than the average modern human. Their eyes might simply be proportionally larger in the same way as the rest of their face is.

In 2012, Pearce and Dunbar showed that some modern humans living in high latitudes also have larger eyes than average. Yet the other parts of their brain are not smaller, as far as we know. "Basically, eyes don't tell you anything useful about cognitive abilities in living people," says Hawks.

Vision and cognition are not separate

The issue is further complicated by the fact that the brain is extremely interconnected. The visual cortex is involved in processing visual information, but it does not paint the whole picture of our world.

How we interpret what we see is in part defined by our pre-existing knowledge of the world. For example, our memories are closely linked with our emotions. All of these cognitive processes occur in slightly different parts of the brain, and vision plays a role in them all.

In other words, vision and cognition are not separate.

They are "intrinsically related", says Robert Barton from Durham University in the UK, who was not involved with either study. In 1998, he showed that a larger visual area of the brain can result in the expansion of other areas, not a reduction.

The truth is that after we initially perceive an object in the real world, the information is projected into several areas of the brain. "It's difficult to distinguish what particular areas of the cortex are not involved with vision," Barton says.

Lastly, it is true that large eyes also give the holder the benefit of increased visual sensitivity in low light. Many nocturnal species have large eyes for that purpose.

Neanderthals' big eyes may have been crucial to their success

However, it is only the ability to see fine details that increases the computational demand within the brain's visual system, says Barton.

Pearce's study does not differentiate between this visual acuity and simple sensitivity to faint light, says Barton. "[Sensitivity] is a matter of the basic physics of light capture," he says, so higher sensitivity doesn't need more brainpower.

Nocturnal primates like bushbabies illustrate this point. They have very large eyes but do not have a corresponding larger visual cortex.

If Barton is right, Pearce and Dunbar have the story backwards. Neanderthals' big eyes may have been crucial to their success, allowing them to flourish in regions with dim light. But they need not have led to their owners' downfall.

Follow Melissa Hogenboom and BBC Earth on twitter