Courtship rituals within the animal kingdom can get rather elaborate. Even fruit flies use sex-specific pheromones and have a courtship maneuver that includes the male buzzing its wings alluringly. But the genetic control behind performing these mating rituals seems, at least in the flies, to largely be separate from the system that controls how they're targeted—mutations in a single gene can flip fly behavior so that male animals start pursuing other males. Now, researchers have identified a key regulator of mate choice in an organism much closer to us—the mouse.

Like flies, mice use pheromones to help identify viable mates, and male mice use a series of ultrasonic chirps as part of their courtship ritual, which ends in them attempting to mount the object of their attention. Male mice that don't have a working pheromone system, however, are more likely to direct their attention to other males, suggesting that, as in flies, the targeting of these behaviors operated separately from performing them.

In the new work, the authors followed up on some indications that the serotonin signaling system might be involved in mate choice. They took advantage of a targeted mutation that eliminates the production of most serotonin-producing neurons in the brain of mice, and tested whether the mice responded normally to sexual cues. These included some very atypical lab assays, like determining how much time the mice spent sniffing around the bedding of males and females, and some straightforward ones, like figuring out how often a male mouse would try to mate with other males instead of responding aggressively to them. (Sample experiment: "a slide with one half smeared with female genitals and the other half with male genitals was presented to a test male.")

The results were pretty clear. If given access to a female or her scent, males that lacked serotonin signaling would respond just as their peers did. But, if given equal access to males and females, they seemed to have no preference whatsoever. Instead of responding aggressively to a challenge from another male, the mutant mice "showed significantly more mounting of male intruders."

This wasn't a failure of the pheromone system, though. The same mice could be trained to avoid electric shocks based on the presence of male and female scents (in this case, the ones present in mouse urine).

The mutation involved here blocks the development of serotonin-producing neurons, so it's possible that it has some rather general effects on the mouse's nervous system. To make sure the effects were specific to serotonin, a second set of mice were bred—these lack a gene essential for producing a chemical precursor to serotonin. These mice weren't especially picky about the mating partners, again as measured by factors like "mounting latency" and "mounting frequency."

The neat thing about these mice is that the serotonin deficiency can be easily fixed, as the chemical precursor can be supplied by injecting it into the abdomen. It only took about a half hour after this injection for a normal mating bias to be restored.

Although serotonin signaling had been implicated in mating behavior by previous work, this new study shows that it's not so much the behavior that's affected, but how it's directed. As the authors note, "an unavoidable question raised by our findings is whether 5-HT has a role in sexual preference in other animals," namely humans. There have been a couple of studies of serotonin signaling and sexuality in people, but none that directly touch on this question. You can bet, however, that the authors are already collaborating with people who can do this sort of work in humans.

Nature, 2011. DOI: 10.1038/nature09822 (About DOIs).