A male mouse's desire to mate with either a male or a female is determined by the brain chemical serotonin, scientists report in a new study. The finding demonstrates for the first time that a neurotransmitter governs sexual preference in mammals.

Serotonin is known to regulate sexual behaviors, such as erection, ejaculation and orgasm, in both mice and men. The compound generally dampens sexual activity; for instance, antidepressants that increase the amount of serotonin in the brain sometimes decrease sex drive. [Top 10 Aphrodisiacs]

Neuroscientist Yi Rao of Peking University and the National Institute of Biological Sciences in Beijing, and his collaborators have now shown that serotonin also underlies a male's decision to woo a female or another male. They published their results in the March 24 issue of the journal Nature.

Rao and his team genetically engineered male mice to lack either serotonin-producing neurons or a protein that is crucial for making serotonin in the brain. Both types of altered mouse couldn't make serotonin.

Unlike typical males, mice deficient in the neurotransmitter showed no inclination to mount sexually receptive females more than males, nor did they prefer to smell females' genital odors or bedding. Instead, they climbed onto males and serenaded them with ultrasonic love songs more frequently than normal. Males emit these vocalizations when they encounter females to make them more receptive to mating.

While all of the males who possessed serotonin mounted females first, nearly half of the mice that lacked serotonin clambered onto males before females, and about 60 percent spent more time sniffing or hovering over the genital odors and bedding from males than from females.

When the researchers injected a compound into these mice to restore neurotransmitter levels, they found that the animals mounted females more than males. But too much serotonin reduced male-female mounting, suggesting that the amount of this chemical must stay within a certain range to foster heterosexual rather than homosexual behaviors.

"An unavoidable question raised by our findings is whether [serotonin] has a role in sexual preference in other animals," the authors wrote in the paper. But one of the co-authors, neuroscientist Zhou-Feng Chen of Washington University, cautioned against forming hasty conclusions about the potential influence of this neurotransmitter on human sexual orientation.

Elaine Hull, an expert in rodent sexual behavior at Florida State University who was not involved in the study, said that the findings "may have implications for homosexuality or bisexual behavior in humans," adding that the neurotransmitter could help to guide sexual development.

Still, she agreed with Chen, cautioning against overinterpreting the results.

"A lot of people are going to be reading more into this than may or may not be warranted," Hull told LiveScience. "Much more information is needed to specify the brain areas involved and possible developmental regulation of serotonin in those areas, before we can jump to the conclusion that serotonin is the factor that inhibits male-to-male attraction."

You can follow LiveScience on Twitter @livescience.