The mystery of what killed off the Neanderthals about 30,000 years ago comes a step closer to being solved with a study suggesting that they formed a tiny population that had been teetering on the brink of extinction.

Neanderthals first appeared in Europe at least 300,000 years ago but they disappeared after the arrival of anatomically modern humans, Homo sapiens, who first arrived in Europe 50,000 years ago. This has led to speculation about whether the Neanderthals interbred with the new arrivals to form a hybrid population that became submerged in the human gene pool, or were instead wiped out by them, either through competition for resources or by violence.

The latest evidence, an analysis of DNA recovered from a 38,000-year-old fossilised thigh bone, suggests the Neanderthals did not interbreed with modern humans but were eradicated by them.

Download the new Independent Premium app Sharing the full story, not just the headlines

DNA extracted from an adult Neanderthal man who lived near caves in what is now Croatia also revealed that the Neanderthals in Europe probably never numbered more than 10,000 individuals at any one time – a precariously small population size.

The new evidence about the demise of the Neanderthals comes from the complete sequence of DNA within tiny cellular structures known as mitochondria. This mitochondrial DNA is maternally inherited and is easier to isolate from ancient bones than the conventional DNA found within the cell nucleus.

The scientists repeatedly decoded the mitochondrial DNA from the 38,000-year-old Neanderthal bone 35 times to make sure that they had the correct genetic sequence, so that they could use it as an accurate comparison against the mitochondrial DNA of modern humans and chimpanzees – man's closest living relative

"For the first time, we've built a sequence from ancient DNA that is essentially without error," said Richard Green, who led the investigation at the Max Planck Institute for Evolutionary Anthropology in Leipzig, Germany.

"It is still an open question for the future whether this small group of Neanderthals was a general feature, or was this caused by some bottleneck in their population size that happened late in the game," said Dr Green.

Archaeological evidence shows that Neanderthals and anatomically modern humans, known as Cro-Magnon man, occupied the same habitats and sites at overlapping periods of time but there is no hard evidence that there was any direct contact between the two last species of humans to share living space.

"There's no proof that they saw each other, only that they inhabited the same place at about the same time but I think it's likely that they came across one another," said Adrian Briggs, a researcher at the Max Planck Institute, who was part of the study.

Daily coronavirus briefing No hype, just the advice and analysis you need Enter your email address Continue Continue Please enter an email address Email address is invalid Fill out this field Email address is invalid Email already exists. Log in to update your newsletter preferences Register with your social account or click here to log in I would like to receive morning headlines Monday - Friday plus breaking news alerts by email Update newsletter preferences

"What we've done is confirm that the mitochondrial DNA of Neanderthals and modern humans was so different that it forms powerful evidence that there was very little if any interbreeding between the two species," said Dr Briggs.

"We have also got tantalising evidence that the Neanderthals formed a small population and we can only speculate as to what happened to them. Small population sizes are always more prone to extinction and they have a greater chance of something going wrong."

Speculation about who the Neanderthals were, and what happened to them, has raged ever since the first Neanderthal skull was excavated from the Neander Valley, near Düsseldorf, in 1856.

It is now generally agreed that they were not the direct ancestors of modern humans but a side-branch on man's extensive family tree. However, some anthropologists have clung to the belief that they must have interbred with humans at some stage in their history, which means that there is a little bit of Neanderthal in us all.

However, a number of DNA studies, including the latest published in the journal Cell, have found little to support that theory. Whenever it has been possible to analyse the sequence of heavily degraded DNA fragments extracted from Neanderthal bone, it shows that the genetic variation lies well outside the variation seen in modern humans.

The latest study suggests, for instance, that the Neanderthals last shared a common ancestor with modern humans some 660,000 years ago – long before the emergence in Africa of Homo sapiens as a distinct species about 100,000 years ago.

However, the scientists who carried out the study emphasised that their work cannot as yet completely rule out the possibility that there was some limited, small-scale interbreeding between Neanderthals and modern humans, at some place between the Caucasus and western Europe – the geographic range of the Neanderthals.

One of the best bits of evidence in support of that idea emerged about a decade ago when scientists found the skeleton of a young boy who had died about 25,000 years ago in what is now Spain. His thick-set features suggested he was hybrid of Neanderthal and Cro-Magnon – but other scientists believed he was just an unusually stocky lad.

There is little doubt that Neanderthals would have looked very different from the new arrivals in Europe. Their rib cage flared out, so they would have had no waists, which would have exaggerated their thick-set appearance.

Heavy jaws, a double arch over the eyebrow resulting in a beetle brow and strong muscles added to the overall thuggish look. But despite the reputation for being thick, Neanderthals were intelligent – they used quite sophisticated stone tools, controlled fire, wore animal skins and buried their dead. The presence of a hyoid bone in their throats also suggested they could speak, although few experts believe that they were capable of the sophisticated language being developed at the time by early modern humans.

Professor Chris Stringer, head of human origins at the Natural History Museum in London, believes that the long period of separation – and genetic isolation – between the Neanderthals and early modern humans meant that profound physical and mental differences had evolved between them.

"The question then is whether, when the populations met, they regarded each other as simply people, enemies, aliens or even prey," he said. "We simply don't know the answer, and the answer may have varied from one time and place to another, especially given the vagaries of human behaviour."

We may never know what happened when modern humans came to live in the same space inhabited by the Neanderthals. They may simply have avoided one another, with Neanderthals retreating to their last stronghold in Europe – a cave system in Gibraltar where the most recent Neanderthal bones have been found.

Or the two species might have engaged in the sort of brutal conflict that has been the hallmark of human history throughout time.

Genetic differences

Neanderthal Man

Heavy jawbone and beetle brow would have made him look like a rugby player on steroids. Flared rib cage meant he did not have a defined waistline. Strong muscles would have added to his thick-set appearance. Used stone tools, controlled fire and buried his dead but little evidence for more symbolic behaviour. Presence of hyoid bone in the throat suggests the use of speech, but experts doubt that language was sophisticated. Well adapted to cold conditions, with short limbs and heavy torso, which would have helped to survive the successive ice ages in Europe.

Modern Man

Graceful chin and jaw and lack of a double arch over the eyebrow gives his face a softer appearance. Tapering rib cage separated the lower abdomen from the chest, giving a marked waistline. Used sophisticated stone tools, controlled fire and buried their dead with ritual. Indicates the development of symbolic thought and being able to plan for the future. Hyoid bone indicates speech as an important feature in terms ofsocial organisation. Gracile limbs and body indicates adaptation to a warm climate.