Modern-day felines may be the ultimate mutts.

Different species of cats mated with each other at several points in history, a new genetic study of felines reveals.

These ancient cat hybrids may have shaped the course of feline evolution, researchers said.

The new cat family tree could also help explain many of the mysteries of cat evolution that have emerged in recent years, scientists added. [Real or Fake: 8 Bizarre Hybrid Animals]

Twisted family tree

When creating family trees of species, researchers can discover how closely related two species are by looking at the level of similarity between their DNA.

In the past decade or so, as scientists analyzed cat genomes to learn more about feline evolution, strange discrepancies arose. This was related to the two types of DNA found in cells: nuclear DNA, from cell nuclei, and mitochondrial DNA, from cell mitochondria, which are the power plants of cells. Family trees of cat species drawn up from nuclear DNA sometimes differed from those drawn up from mitochondrial DNA, said study co-author William Murphy, a mammalian geneticist at Texas A&M University in College Station.

One possible solution to this puzzle was that interbreeding between different cat species occurred. This explanation could work because whereas nuclear DNA is generally inherited from both parents, mitochondrial DNA is inherited only from mothers. As a result, if a female tiger mates with a male lion, for example, their cubs would generally carry an equal mix of lion and tiger nuclear DNA, but their mitochondrial DNA would come solely from the mother, the tiger. The cubs, in turn, would have different family trees depending on which type of DNA was used in the analysis.

What's more, previous research found that different cat species that do not live anywhere near each other in the wild can interbreed in zoos. This, for example, can happen between lions, which live in Africa, and tigers, which live in Asia. The results are hybrids that have traits of both species, such as ligers, crosses between male lions and female tigers, and tigons, the offspring of female lions and male tigers.

"One of the world's most popular cat breeds, the Bengal, is a hybrid between the domestic cat and the Asian leopard cat, and several other increasingly common cat breeds are of hybrid origin," Murphy said in a statement.

Ancient interspecies mating

To help solve these mysteries of cat evolution, Murphy and his colleagues investigated DNA from 38 cat species from eight lineages of felines, ranging from the domestic cat to big cats such as lions, tigers, cheetahs, leopards and jaguars. The researchers analyzed cat nuclear DNA — both the X and Y sex chromosomes, as well as nonsex chromosomes — in addition to cat mitochondrial DNA. [Iconic Cats: See Photos of All 9 Subspecies of Tigers]

The scientists constructed a cat family tree of maternal and paternal lineages within each species. They found at least 10 instances of hybridization shaping cat evolution, and "we will likely find more once we have complete genome sequences from all living cat species," Murphy told Live Science.

The researchers identified traces of hybridization within the genomes of more than half of the eight cat lineages – specifically, the domestic cat, leopard cat, lynx, ocelot and panthera lineages.

"Seeing the evidence for ancient genetic exchange across many lineages was a bit of a surprise," Murphy said.

For example, the mitochondrial genomes of snow leopards and lions are more similar to each other than are their nuclear genomes. This suggests that female hybrid progeny of male ancestors of modern snow leopards and female ancestors of modern lions interbred with male ancestors of modern snow leopards. In addition, while pumas were more closely related overall to domestic cats and Asian leopard cats, puma females had signs of ancestry from lynxes and bay cats, the researchers reported in their paper, published in the January issue of the journal Genome Research.

"Ancient hybridization in the wild is consistent with extensive evidence for hybridization that has occurred between many distantly related cat species in captivity, such as the liger," Murphy said in a statement.

These findings support other research suggesting that hybridization between different species is more common than previously thought and may actually introduce helpful genes into species, Murphy said. For instance, recent studies hint that interbreeding between modern humans and Neanderthals may have endowed some modern humans with robust immune systems.

Future research can analyze cat genomes further to better understand the frequency of hybridization in wild cats, as well as to identify genes that either support interbreeding to blur the lines between species or work against interbreeding to help create new species, Murphy said.

Follow Charles Q. Choi on Twitter @cqchoi. Follow us @livescience, Facebook & Google+. Original article on Live Science.