More than a decade ago, researchers in a limestone cave on the Indonesian island of Flores unearthed the bones of an ancient race of tiny humans. Now, in sandstone laid down by a stream 700,000 years ago, they have found what appear to be the creatures’ ancestors.

The new fossils are not extensive. A partial lower jaw and six teeth, belonging to at least one adult and two children, are all researchers have. But the importance of the remains outweighs their number. They suggest that dwarf humans roamed the island - hunting pygmy elephants and fending off komodo dragons - for more than half a million years.

The first bones belonging to the miniature humans were dug from the floor of the Liang Bua cave on Flores in 2004. The 50,000-year-old fossils pointed to a now-extinct group of humans that stood only a metre tall. Named Homo floresiensis, but swiftly nicknamed the “hobbits”, they made simple stone tools and had desperately small brains, one third the size of ours.

For all that was known about the diminutive humans, countless questions remained. How the species arose was anyone’s guess. Meanwhile, some experts argued they were not a new species at all, simply modern humans whose growth had been stunted by disease.

The origins of the species known as ‘the hobbit’ - a human relative only a little over a metre tall - have been debated ever since its discovery in 2004. New fossils may reveal the ancestors of this strange species and help us to understand its history.

The newly discovered fossils from Mata Menge, a large basin overshadowed by volcanoes in central Flores about 50km east of Liang Bua, effectively rule out the modern human theory. The tiny individuals were alive and making stone tools on Flores half a million years before modern humans existed. “This is the final nail in the coffin for that hypothesis,” said team leader Gert van den Bergh at the University of Wollongong in Australia. “700,000 years ago, there were no Homo sapiens.”

But the fossils, described in two papers in Nature, lend weight to another explanation already favoured by some paleontologists. In this scenario, a founder population of Homo erectus, a forerunner of modern humans, washed up on Flores from a neighbouring island, perhaps clinging to plant debris uprooted by powerful tsunamis that crash through the region. Marooned on Flores with limited food at hand, evolution favoured the small. Over 300,000 years, the new arrivals rapidly lost stature.

The jawbone is least 20% smaller than that of the Liang Bua ‘hobbit’. Photograph: Kinez Riza

Adam Brumm at Griffith University in Queensland, who co-led the excavations, said: “The island is small and it has limited food resources and few predators, other than komodo dragons, so large-bodied mammals that wound up on this rock would have been under immediate selective pressure to reduce their body mass. Being big is no longer an advantage when you’re trying to survive in such an isolated and challenging environment.”

The team from Australia, Indonesia and Japan worked with 140 locals on Flores to excavate the fossils. The jawbone is tiny, at least 20% smaller than that of the Liang Bua “hobbits”, but CT scans showed the wisdom tooth had erupted, a sign that it came from an adult. The shape of the jawbone resembles a smaller version of that found in Homo erectus, as does a molar tooth.

Among the ancient remains were the bones of beasts. Pygmy elephants and komodo dragons were commonplace, but crocodiles, giant rats, frogs and birds shared the island with the “hobbits”. Giant rats persist on the island today. The researchers have one, the size of a cat, at the house they rent in the nearby town of Mengeruda. “It’s a really cute animal. If we can domesticate them, they can be kept as pets,” said van den Bergh. “They would be a cheap source of meat for the people.”

Six teeth were found on Flores that belonged to at least one adult and two children. Photograph: Kinez Riza

Simple stone tools, mostly sharp-edged flakes, were also found at the site, though no signs of butchery have been spotted on the animal bones. Curiously, the tools at Mata Menge are similar to those found at the Liang Bua cave, but simpler and smaller than others found at an older site on the island called Wolo Sege. These heftier, more advanced implements, including carefully shaped core tools known as picks, may be the million-year-old work of the island’s large-bodied founding population, said Brumm, a technology later lost as the dwarfing process took its toll in the islanders’ brains.

Still more fossils are needed to complete the picture. The researchers are eager to find long bones at Mata Menge that prove beyond doubt the site holds the remains of archaic dwarf humans. Elsewhere on the island, the search is on for the elusive first arrivals, perhaps full-sized Homo erectus, or maybe a smaller ancestor. Where they came from remains another open question. The trip from Java is 300 miles, a long way to cling to a bobbing mat of vegetation. Sulawesi to the north is more distant still, but strong currents could have done the job. “That’s a reasonable chance,” said Van den Bergh. “But we’ll probably never know.”

Dean Falk, an evolutionary anthropologist at Florida State University, said the new fossils will help to convince all but the most diehard sceptics that Homo floresiensis is a legitimate species. She said the 700,000 year-old date for the new finds, and the fact that they are at least as small as the Liang Bua individuals, is exciting. She added: “Although in my opinion one still cannot rule out the possibility that Homo erectus and Homo floresiensis may have shared a common ancestor that was an unknown small-bodied and small-brained hominin.”