The fossilised remains of stone age people recovered from two caves in south west China may belong to a new species of human that survived until around the dawn of agriculture.

The partial skulls and other bone fragments, which are from at least four individuals and are between 14,300 and 11,500 years old, have an extraordinary mix of primitive and modern anatomical features that stunned the researchers who found them.

Named the Red Deer Cave people, after their apparent penchant for home-cooked venison, they are the most recent human remains found anywhere in the world that do not closely resemble modern humans.

The individuals differ from modern humans in their jutting jaws, large molar teeth, prominent brows, thick skulls, flat faces and broad noses. Their brains were of average size by ice age standards.

"They could be a new evolutionary line or a previously unknown modern human population that arrived early from Africa and failed to contribute genetically to living east Asians," said Darren Curnoe, who led the research team at the University of New South Wales in Australia.

"While finely balanced, I think the evidence is slightly weighted towards the Red Deer Cave people representing a new evolutionary line. First, their skulls are anatomically unique. They look very different to all modern humans, whether alive today or in Africa 150,000 years ago," Curnoe told the Guardian.

"Second, the very fact they persisted until almost 11,000 years ago, when we know that very modern looking people lived at the same time immediately to the east and south, suggests they must have been isolated from them. We might infer from this isolation that they either didn't interbreed or did so in a limited way."

One partial skeleton, with much of the skull and teeth, and some rib and limb bones, was recovered from Longlin cave in Guangxi province. More than 30 bones, including at least three partial skulls, two lower jaws and some teeth, ribs and limb fragments, were unearthed at nearby Maludong, or Red Deer Cave, near the city of Mengzi in Yunnan province.

At Maludong, fossil hunters also found remnants of various mammals, all of them species still around today, except for giant red deer, the remains of which were found in abundance. "They clearly had a taste for venison, with evidence they cooked these large deer in the cave," Curnoe said.

The findings are reported in the journal PLoS ONE.

The stone age bones are particularly important because scientists have few human fossils from Asia that are well described and reliably dated, making the story of the peopling of Asia hopelessly vague. The latest findings point to a far more complex picture of human evolution than was previously thought.

"The discovery of the Red Deer Cave people shows just how complicated and interesting human evolutionary history was in Asia right at the end of the ice age. We had multiple populations living in the area, probably representing different evolutionary lines: the Red Deer Cave people on the East Asian continent, Homo floresiensis, or the 'Hobbit', on the island of Flores in Indonesia, and modern humans widely dispersed from northeast Asia to Australia. This paints an amazing picture of diversity, one we had no clue about until this last decade," Curnoe said.

Much of Asia was also occupied by Neanderthals and another group of archaic humans called the Denisovans. Scientists learned of the Denisovans after recovering a fossilised little finger from the Denisova cave in the Altai mountains of southern Siberia in 2010.

The fossils from Longlin cave were found in 1979 by a geologist prospecting in the area. At the time, researchers removed only the lower jaw and a few fragments of rib and limb bones from the cave wall. The rest of the skeleton was left encased in a block of rock, which sat in the basement of the Yunnan Institute of Cultural Relics and Archaeology in Kunming, Yunnan, for 30 years. The fossils were rediscovered in 2009 by Ji Xueping, a researcher at the institute, who teamed up with Curnoe to examine the remains.

"It was clear from what we could see that the remains were very primitive and likely to be scientifically important. We had a skilled technician remove the bones from the rock, and they were glued back together. Only then was it clear what we had found: a partial skeleton with a very unusual anatomy," Curnoe said.

The fossils at Maludong were found in 1989 but went unstudied until 2008.

Lumps of charcoal uncovered alongside the Longlin fossils were carbon dated to 11,500 years, a time when modern humans in southern China began to make pottery for food storage and to gather wild rice in some of the first steps towards full-scale farming.

Marta Mirazón Lahr, an evolutionary biologist at Cambridge University, is convinced the remains are from modern humans. The unusual features, she said, suggest the Red Deer Cave people are either "late descendants of an early population of modern humans in Asia" or a very small population that developed the traits through a process known as genetic drift.

Chris Stringer, head of human origins at the Natural History Museum, London, was similarly sceptical.

"The human remains from the Longlin Cave and Maludong are very important, particularly because we do not have much well-described and well-dated material from the late Pleistocene of China.

"The fossils are unlike recent populations of modern humans in several respects, and the mosaic of more archaic features could indicate the dispersal of a poorly known and more primitive form of modern human that left Africa before the main exodus at about 60,000 years. This dispersal could have reached as far as China, surviving there for many millennia, before disappearing in the last 12,000 years."

But he added: "There might be another possible explanation for the more archaic features. Could these alternatively be attributed to gene flow from a more archaic population that survived alongside modern humans? In the case of the Longlin Cave and Maludong fossils, the most likely candidate would be the enigmatic Denisovans who apparently interbred with the ancestors of modern Australasians somewhere in south east Asia. Could these Chinese fossils be further evidence of such hybridisation?"