To female molly and Limia fish, nothing is hotter than a male with a large dorsal fin. But these fins aren’t just decorations to attract females.

Males also use them to fight or intimidate rivals. For scientists who study evolution, the fins present a chicken-and-egg dilemma. Which came first — ornamental fins for courtship displays, or fighting fins only later used in displays?

In a new paper, biologists from the University of California, Riverside, studied the evolution of 40 molly and Limia species, and concluded dorsal fin displays arose first for males to compete with other males, only later being used in courtship displays to females. These changes in fin function went hand in hand with enlargement of the male dorsal fin. The fins reached extreme sizes in a few species and appear to be associated with rapid evolution, especially in mollies.

Video shows variety of molly males, from closest to ancestral state to farthest from ancestral state.

When Charles Darwin proposed his theory of evolution, he noted that some features arise not from natural selection by outside pressures like predators and food sources, but from competition within a species to find mates and reproduce. He called this “sexual selection,” and scientists have used it to explain why males from many species have elaborate ornamentation and are larger than females.

Simply put, the ornament can evolve either because females prefer to mate with ornamented males, or because the ornament helps males defeat rivals. The male peacock’s tail, for example, probably evolved through female choice. Deer antlers, on the other hand, probably evolved to compete with other males.

But whether male ornamentation arises through female choice or male-male aggression also has implications for how a group of animals diverges from an ancestral population and evolves into a separate species. In existing species, male-male aggression and covert or “forced” mating occurs much more often than courtship displays and female choice, leading some biologists to predict that, in general, male ornamentation probably evolved first for aggression.

“This is something that biologists have argued about for over a century, whether it’s all in male-male competition for females or how important is female choice of a male,” said Daniel Goldberg, the paper’s first author, who conducted the research for his master’s thesis. “The problem is, even though it was predicted in the 1990s that male ornamentation first evolved for aggression in many species of animals, studies that have been able to prove this for a particular group of animals have been few and far between.”

Senior author David Reznick, a professor of biology at UC Riverside, has been working to unravel this problem in guppies. Guppies, like mollies and Limia, give birth to live young instead of laying eggs. The physiology of live birth comes in two main forms for these fish. In some species, the female produces a fully yolked egg, which is simply retained within her body, with no further nourishment after fertilization. In other species, the females have something like a placenta that nourishes eggs throughout development.

“The first step, which came before Goldberg’s work, was to show that the male traits were tied to the type of female reproduction. You only get this kind of ornament in the ones that don’t have placentas,” explained Reznick. In live-bearing nonplacental fish, the female makes all her investment before the egg is fertilized, and the only thing left to ensure a quality offspring is to choose a high-quality male to fertilize it. The ornamentation could signal male quality.

Mollies are found along the Gulf Coast of North America and throughout Central America, while Limia are native to islands of the Greater Antilles in the Caribbean. Both are popular aquarium pets. Goldberg took comprehensive measurements of various body dimensions of male and female molly and Limia species, along with detailed observational notes, to develop an ornamentation index that measures dorsal fin size relative to body size. He then used this information from the living species to follow the evolutionary tree back through time to figure out what the original ancestors would have looked like.

“If you look outside and you see a tree, where we have two branches that emanate from a single branch, that corresponds on an evolutionary tree to a speciation event,” explained co-author Mark Springer, a professor of biology at UC Riverside who worked on the evolutionary sequence. “We’re interested in trying to reconstruct what the ancestral state was like at each one of those branching points in the tree.”

“We know what the branch tips are like, we know what each species is, and so we use that information to go back through time and envision their common ancestor,” added Reznick.

As predicted, the researchers found that in both mollies and Limia, males evolved larger dorsal fins first for fighting other males, only later using them in courtship displays. At that point, it was only a matter of time until they reached huge sizes in a few exceptionally ornamented modern species: three species of aptly-named sailfin mollies, and one species of Limia: the humpback Limia.

The evolution of ornamentation really took off with the evolution of courtship.

“Mollies and Limia have a lot of different species, and the evolution of elaborate courtship displays seems to go hand in hand with rapid speciation,” said Goldberg, who is now pursuing a doctorate in biology at Illinois State University. Reznick and Springer are studying the speciation rate in a separate project.

The findings may be broadly applicable to other species and may help biologists solve the chicken-and-egg riddle of male ornamentation.

The paper, “In Love and War: The Morphometric and Phylogenetic Basis of Ornamentation, and the Evolution of Male Display Behavior, in the Livebearer Genus Poecilia,” was published Jan. 17 in Evolution. In addition to Goldberg, Springer, and Reznick, authors include Joseph A. Landy and Joseph Travis at Florida State University.