The more research that is done into the prehistoric origins of humans, the more crowded it gets.

According to a new study, there may well have been multiple species of primitive humans between 60,000 and 120,000 years ago – including a mysterious people who do not fall under the categories of either Neanderthals, Denisovans, or modern humans.

Skull fragments and teeth discovered in 1976 in a cave in Xujiayoa, China are raising surprising questions about the prehistoric origins of humanity. Dental remains were recovered from four different individuals, and were examined for size, shape and surface of the teeth, as well as other defining characteristics. When the teeth were compared to a database of over 5,000 teeth of known species, María Martinón-Torres of the National Research Centre on Human Evolution found that they did not match any of the accepted hominids.

Shape, size, placement and wear on teeth are indicators of species. Credit: Martinon-Torres

BBC Earth reports, “We know there were as many as four other early humans living on Earth when modern humans were still confined to Africa. The Neanderthals lived in Europe, the Denisovans in Asia and the "hobbit" Homo floresiensis in Indonesia: plus there was a mysterious fourth group from Eurasia that interbred with the Denisovans.”

The question is now whether the teeth represent a hybrid of Denisovans and modern humans, or whether scientists have an entirely new human species on their hands.

Martinón-Torres and colleagues recently published a study in the American Journal of Physical Anthropology , providing a comparative analysis of the teeth found in northern China, and stated that the teeth are different from those of modern humans, Homo sapiens. The remains possess some primitive features similar to, but not exactly matching Neanderthals or Homo erectus.

“Teeth are like ‘landscapes in miniature’. Each of those slopes, grooves, valleys define a pattern or combination of features that can be distinctive of a population,” Martinón-Torres told BBC Earth.

While the findings are surprising and mysterious, experts are reluctant to confirm it represents a whole new species. "What we have seen is an unknown group for us," continued Martinón-Torres. "It's not H. sapiens and it's not H. neanderthalensis . They have a mixture of something very primitive, which is currently unknown. We cannot go further to say it's a new species because we need to compare it to other things."

Still other researchers insist that while further research needs to be done, the teeth are unique enough to represent an unknown variant based on their surface features alone.

A mix of hominid (genus Homo) models; (from right to left) H. habilis, H. ergaster, H. erectus; H. antecessor - male, female, H. heidelbergensis; H. neanderthalensis - girl, male, H. sapiens sapiens. Public Domain

These aren’t the first potential ancient humans to have confounded researchers. In 2013 a Royal Society meeting on ancient DNA revealed a dramatic finding – the genome of one of our ancient ancestors, the Denisovans, contains a segment of DNA that seems to have come from another species that is currently unknown to science. The discovery suggested that there was rampant interbreeding between ancient human species in Europe and Asia more than 30,000 years ago. But, far more significant was the finding that they also mated with a mystery species from Asia – one that is neither human nor Neanderthal.

Could there be a connection between the Chinese remains and that unidentified branch of humanity?

Mark Thomas, an evolutionary geneticist at University College London said of the 2013 DNA find, “What it begins to suggest is that we're looking at a 'Lord of the Rings'-type world - that there were many hominid populations.”

China is home to another controversial prehistoric population – the Red Deer Cave people, dated between 14,500 and 11,500 years. Fossils of the early hominids were discovered in a cave in southern China . While their facial features do not resemble modern humans, experts are reluctant to classify them as a unique species pending DNA extraction.

Red Deer Cave people, skull and reconstruction. Wikimedia, ( CC BY 2.5 )

If the Xujiayoa teeth studied by Martinón-Torres and colleagues do reveal an ancient Asian ancestor, history books and the conventional theory which says Africa was the cradle of modern humans will have to be changed.

Human evolution is not as simple as we like to think, but as more remains and fossils are recovered, especially with the aid of DNA evidence, these mysteries may quickly be resolved. The further we proceed into our future, the more we reveal about our ancient past.

Featured Image: Comparison of Modern Human and Neanderthal skulls. (CC BY-SA 2.0)

By Liz Leafloor