They're not quite Neanderthals and not quite modern humans. They're something else, but no one is sure what.

Newly-examined fossils suggest that an unknown species of human was roaming parts of northern China between 60,000 and 120,000 years ago. Alternatively, the fossils could be the result of interbreeding between two of the known species.

We know there were as many as four other early humans living on Earth when modern humans were still confined to Africa. The Neanderthals lived in Europe, the Denisovans in Asia and the "hobbit" Homo floresiensis in Indonesia: plus there was a mysterious fourth group from Eurasia that interbred with the Denisovans.

The new findings suggest the picture is even more complicated.

The Chinese remnants were first discovered in a cave in the Xujiayao site in 1976. They consist of some skull fragments, and nine teeth from four individuals. A comprehensive analysis of the teeth has now been published in the American Journal of Physical Anthropology.

Teeth are like 'landscapes in miniature'. Each of those slopes, grooves, valleys define a pattern or combination of features that can be distinctive of a population

María Martinón-Torres of the National Research Centre on Human Evolution in Burgos, Spain and her colleagues looked at the size and shape of the crown and root system, the grooves, cusps and crests, and their positions relative to each other. These were then compared to a pool of over 5,000 teeth representing nearly all the known hominin species.

"Teeth are like 'landscapes in miniature'," says Martinón-Torres. "Each of those slopes, grooves, valleys define a pattern or combination of features that can be distinctive of a population."

It was clear that the teeth did not resemble those of modern humans, H. sapiens. Instead, they have several primitive features, some of which look like the older species H. erectus, while some look more like Neanderthals.

Other skeletal parts found at the same site, described last year, don't neatly fit the known species either.

Nevertheless, Martinón-Torres is reluctant to claim that the teeth represent a new species.

"What we have seen is an unknown group for us," she says. "It's not H. sapiens and it's not H. neanderthalensis. They have a mixture of something very primitive, which is currently unknown. We cannot go further to say it's a new species because we need to compare it to other things."

They might actually fit an existing species. "They could even be Denisovans," adds Martinón-Torres.

The Denisovans co-existed and even interbred with us. But hardly anything is known about them. The only fossils come from a cave in Siberia and consist of two teeth and a tiny finger bone. DNA analysis revealed that they were distinct from both Neanderthals and modern humans but had aspects of both.

The Xujiayao teeth show a similar pattern, Martinón-Torres says.

Not everyone agrees. While the sample is small, it "strongly suggests the presence of a previously unrecognized species," says Darren Curnoe of the University of New South Wales in Sydney, Australia. "There's little doubt in my mind that these teeth stand out as something unique." The surface features on the teeth alone should be enough to argue for a new species, he says.

Conceivably the remains come from a hybrid of modern humans and Denisovans, but that is pure speculation

Curnoe previously described another set of mysterious hominin fossils from China, the "Red Deer Cave People", though these lived more recently than the Xujiayao hominins.

Others echo Martinón-Torres' caution. Matthew Skinner of the University of Kent, UK says fossil samples from Asia are so sparse that it is hard to infer species status.

Fred Spoor of University College London in the UK agrees with Skinner. He says the remains show a mix of modern and primitive features. "What it means is another matter." Conceivably the remains come from a hybrid of modern humans and Denisovans, "but that is pure speculation".

Many of the supposedly separate Homo species might just be variants of a single species, says Erik Trinkaus of Washington University in St Louis, Missouri. "As the fossil record fills in, most of the purported gaps between such 'species' are gradually disappearing," he says. "Real species in the real world, especially for large bodied animals like us, are widespread and variable."

On that view, finding teeth that don't fit the known "species" just isn't surprising.

More bones would help, and they might turn up soon, as parts of Asia are turning out to be rich in fossils.

But it may be that only DNA evidence will offer definitive answers, says Matthias Meyer of the Max Planck Institute for Evolutionary Anthropology in Leipzig, Germany. "It would be great to get more data from east Asia, but obviously, this is difficult."

"There was probably more than one species of early human in Asia, which wouldn't be surprising considering the size of the continent and how isolated it is," says Martinón-Torres. Last year she helped describe some teeth from the same time period in east China, which also did not neatly fit the known species.

Some of these populations could even have been ancestors of modern Europeans, according to some researchers. Most think Africa was the cradle of modern humans, so this is controversial. But Martinón-Torres thinks people will return to the idea, as more and more Asian fossils are analysed.

Curnoe agrees. "We've neglected East Asia for far too long," he says.

"Now we're starting to get a few surprises that don't fit with conventional wisdom based on fossils from Europe and Africa."

Follow Melissa Hogenboom and BBC Earth on twitter