When scientists discovered a few years ago that modern humans shared swaths of DNA with long-extinct Neanderthals, their best explanation was that at some point the two species must have interbred.

Now a study by scientists at the University of Cambridge has questioned this conclusion, hypothesising instead that the DNA overlap is a remnant of a common ancestor of both Neanderthals and modern humans.

When the genetic sequence of Homo neanderthalensis was published in 2010, one of the headline findings was that most people outside Africa could trace up to 4% of their DNA to Neanderthals. This was widely interpreted as an indication of interbreeding between Neanderthals and early Homo sapiens just as the latter were leaving Africa. The two species would have lived in the same regions around modern-day Europe, until Neanderthals died out about 30,000 years ago.

But Andrea Manica said the analysis had over-estimated the amount of shared DNA between Neanderthals and humans that could be explained by interbreeding. The analysis had not taken into account the genetic variation already present between different populations of the ancestors of modern humans in Africa.

"The idea is that our African ancestors would not have been a homogeneous, well-mixed population but made of several populations in Africa with some level of differentiation, in the way right now you can tell a northern and southern European from their looks. The mixing is not complete within continents."

Taking these population differences, known as "substructuring", into account for early humans living in Africa, Manica and his colleague Anders Eriksson worked out that modern humans and Neanderthals must have shared a common ancestor some 500,000 years ago and that the subsequent evolution of this species was enough to account for the DNA crossover.

"There was an ancestor of both Neanderthal and modern humans – some archaeologists would call that Homo heidelbergensis – that would have covered Africa and Europe about half a million years ago," he said. "It wouldn't have been a single well-mixed population, it would have been like modern humans – populations that are closer to each other are more similar."

The results are published in the Proceedings of the National Academy of Sciences.

About 350,000 years ago, the European and African ranges of this last common ancestor became separate: the European range would later evolve into Neanderthals and the African range into anatomically modern humans, who left the continent 70,000 years ago to cover the world.

Prof Svante Pääbo, at the Max Planck Institute for Evolutionary Anthropology in Leipzig, who led the sequencing of the Neanderthal genome in 2010 and has championed the idea that modern humans interbred with Neanderthals, said he was surprised that Manica's work had been published, since his original paper had admitted a role for substructuring in Africa in the sharing of DNA between humans and Neanderthals. "But we regard this as a less parsimonious explanation," he said.

Pääbo has co-authored a paper, which is yet to undergo peer-review, to further support his thesis that humans and Neanderthals did in fact interbreed. "We find that the last gene flow from Neanderthals (or their relatives) into Europeans likely occurred 37,000-86,000 years before the present, and most likely 47,000-65,000 years ago," he writes. "This supports the recent interbreeding hypothesis, and suggests that interbreeding may have occurred when modern humans carrying Upper Paleolithic technologies encountered Neanderthals as they expanded out of Africa."