The yeti, and his shambling hairy cousins the bigfoot, almasty, sasquatch and migyur, may still be out there, high in the snowy peaks of the Himalayas, Rocky Mountains or Urals – but they have escaped a team of scientists who have been testing dozens of samples, all claimed to be genuine chunks of yeti fur.

They have turned out to be hairs from depressingly familiar animals including cows, raccoons, horses, dogs, sheep, a Malayan tapir, a porcupine, and, in the case of one sample from Texas, a human being. And also a blade of grass and a strand of fibreglass.

"Don't give up yet, the yeti may still be out there," Bryan Sykes said reassuringly. The professor of human genetics at Oxford and an expert on ancient DNA, said he launched the project, writing to museums and collectors all over the world, with only a 5% hope of success.

"That would normally be too slim a margin to launch a major study," Sykes said, "but I did think there was just a chance we would uncover something extraordinary."

What he hoped for was less the abominable snowman of legend than evidence for a surviving Neanderthal, which some say could be the origin of yeti stories.

He found neither. However, the team, which publishes its findings in this week's Proceedings of the Royal Society, has found something almost equally extraordinary, in two samples of bear fur from Bhutan and the Indian Himalayas.

Although one is reddish brown and the other golden brown, the bear's closest relative turned out to be a precise match for DNA extracted from fossil remains of a polar bear that lived 40,000 years ago. The samples were quite unlike modern polar bears. This raises the intriguing possibility that descendents of a prehistoric polar bear are at large in the Himalayas.

The sample from Ladakh in India is said to be from an animal shot 40 years ago by an experienced hunter, who said the creature's behaviour was very different from the brown bears he knew well.

He kept the pelt hidden, and, according to Sykes, was very reluctant to hand over samples to the French explorer who brought them back to the west. The other sample came from Bhutan, where yetis – referred to by the scientists as "anomalous primates" – are known as migyur.

"Polar bears have some quite distinct behaviour, including deliberately hunting human prey," Sykes said. "It would be very interesting to go and see if this is a behavioural pattern which has endured in the Himalayan bears."

The paper is the first such study in a peer-reviewed journal. Sykes, who is also publishing a book on yetis this autumn – "I wouldn't have done this as a young man, before I had an established reputation as a scientist," he admitted – said he was struck that science was accused by yeti enthusiasts of rejecting the notion of their existence. "This conflicts with the basic tenet that science neither rejects nor accepts anything without examining the evidence," the team wrote.

Scientists had largely avoided "this often murky field" for more than half a century, Sykes said, since they joined expeditions in the 1950s led by Sir Edmund Hillary and other explorers. Back then, they could not have conducted sophisticated DNA testing.

Samples poured in from collections all over the world, but the only mystery about most was the enduring one of human credulity. Two sent from Russia were claimed to be hairs from an almasty, the Russian version of the bigfoot. They proved to be raccoon and black bear hairs, both natives of North America. The tapir hair came from Sumatra, cow hairs from several places in the US along with the porcupine quill, and a hair from a serow, a goat like creature, from Nepal.

"Absence of evidence is not evidence of absence, and this survey cannot refute the existence of anomalous primates," the authors wrote. There is more work to be done, Sykes said – but the grass and the fibreglass have definitely been eliminated from his inquiries.