A furious international dispute has erupted over the publication of a paper that claims the hobbit man of Flores was a modern human who had Down's syndrome. Published in the Proceedings of the National Academy of Sciences (PNAS) this month, the research has been denounced by scientists around the world. The tiny Homo floresiensis, discovered on Flores, an island in Indonesia, is definitely a member of a distinct ancient species of hominins, they insist.

The dispute has its roots in an expedition by Australian and Indonesian researchers in 2003. The scientists were working in a limestone cavern called Liang Bua, on Flores, when one uncovered a small skull and lower jaw. Although tiny, the skull had adult teeth. "This was no child, but a tiny adult – one of the smallest adult hominins ever found," the expedition leader, the late Mike Morwood, of Australia's University of Wollongong, announced.

Other remains were dug up and identified as those of a hominid species they called Homo floresiensis. But one small group of scientists has persistently disputed this claim, arguing that the bones really belong to a modern human with anatomical abnormalities. It is their most recent claim, that the Flores hobbit had Down's syndrome, that has enraged fossil experts who insist the tiny bones are those of distinct lineage. "It is interesting their paper contains no images of skeletons of Down's syndrome individuals," said floresiensis expert Professor Dean Falk, of Florida State University. "If it had, you would see clearly that they look nothing like the Flores specimen. The idea is nonsense."

A model of the tiny skull discovered on a remote Indonesian island. Photograph: Stephen Hird/Reuters

Scientists have also attacked the editors of PNAS – the journal of the US National Academy of Sciences – for permitting the authors of the Down's syndrome paper to avoid independent peer review because one is an academy member and so is allowed to select his own referees when submitting the paper.

"The article is a contributed submission from an academy member, Kenneth Hsu, an 89-year-old hydrologist who has absolutely no expertise in the subject and who selected referees that were also without expertise in fossil hominin skeletons," said one of the key scientists involved in the discovery of Homo floresiensis, Professor Peter Brown, of the University of New England in Australia. "This is an outrageous abuse of the peer review process."

It is still not known how these early hominins got to Flores or how they evolved their small stature. Yet most scientists accept their authenticity – with the exception of the late Indonesian palaeoanthropologist Teuku Jacob, who claimed the remains were those of a modern human with an abnormal skull. At one point Jacob and his followers took, without permission, the hobbit bones dug up by Morwood's team. They were eventually returned, though some had been badly damaged.

The measurements taken by Jacob's team have since been used by researchers Maciej Henneberg, of Adelaide University, Robert Eckhardt, of Pennsylvania State University, and Hsu, of China's National Institutes of Earth Sciences, as the bases of articles that claim the hobbit is really a deformed modern human. "First they claimed the hobbit was really a modern person with microcephaly – an abnormally small head," said Falk. "We showed that this could not be true. Then they claimed he had Laron syndrome, a form of dwarfism. Again my team showed this was not true. Now they are taking a shot with Down's syndrome. Again they are wrong."

This last point was backed by Professor William Jungers of the State University of New York. "They say Homo floresiensis is similar to a modern person with Down's syndrome, but no one with that condition has a tiny cranium only 400cc in capacity as floresiensis does, nor do they have thick cranial bones as it does. This is shockingly bad science riddled with errors of fact and attribution."

Crucially, each of the three papers authored by Henneberg and his associates since 2006 was contributed to PNAS by Hsu. "Hsu is a distinguished geologist and polymath, but he is not an expert on human evolution and anatomy," said Professor Chris Stringer, of the Natural History Museum, London. "Indeed, he is on record fiercely attacking Charles Darwin and conventional evolutionary ideas. Yet he has now co-authored and seen through three papers critiquing Homo floresiensis in PNAS." Jungers agreed. "This is just cronyism," he added.

But Hsu's role was defended by the papers' lead author, Maciej Henneberg. "Kenneth's contributions to all three papers are essential from the perspective of an excellent and accomplished geologist, who studied in detail changes in sea levels during the Pleistocene that affected Flores," he told the Observer.

Additional reporting by Meredith Thomas