Once upon a time, animal courtship was thought to run something like a Barbara Cartland novel. The rakish males battle it out for a chaste female, who sits around choosing the prince charming to father her young. While her mate may sow his wild oats far and wide, she patiently tends her brood.

Notwithstanding a few counterexamples, these roles were thought to be largely the same across the animal kingdom: males were thought to be promiscuous, dominant and aggressive and the females chaste and passive. For many people, it was just the natural order of the world.

But have we been blinkered by our own cultural prejudices, casting animals in the kinds of roles we saw in the society around us? That is the view of a small but growing number of biologists. "It's almost like they are using this locker-room logic – counting which males 'score' the most," says Joan Roughgarden at the Hawai'i Institute of Marine Biology.

The dividing line between male and female is frequently blurred or easily crossed

Researchers such as Roughgarden argue that it was a classic case of "confirmation bias". Many biologists were seeing what they wanted to believe, and then using the results to justify prevailing cultural norms. "You get this back-and-forth: science is reinforcing societal mores, and the mores are reinforcing what the science is saying," says Zuleyma Tang-Martinez at the University of Missouri – St Louis.

The result, Tang-Martinez and Roughgarden believe, is that scientists have often failed to recognise astonishingly diverse sexual behaviours across the animal kingdom. There are now myriad examples of animals that break the rules entirely – from intersex kangaroo to a fish with four separate "genders".

If they are right, we should rethink many of our assumptions about sex differences. As with humans, the dividing line between male and female is frequently blurred or easily crossed.

Much of our modern understanding of sex differences came from Charles Darwin's struggles to explain the peacock's tale. How could such a cumbersome and extravagant display ever contribute to the animal's survival? "The sight of a feather in a peacock's tail, whenever I gaze at it, makes me sick!" he wrote in an 1860 letter to his colleague Asa Gray.

Darwin saw the same patterns – males being "passionate", females "coy" – across the animal kingdom

Darwin's solution was "sexual selection": a form of evolution that comes directly from the challenges of reproduction.

When many males compete for a single female, each male has to show off his worth in some way; either through direct combat, or in a showy display that proves he would be the healthiest father for her young. The resulting arms race led to the evolution of ever more excessive traits in the males of certain species: hence the peacock's tale, which helps it to advertise its good health to the peahen.

Darwin saw the same patterns – males being "passionate", females "coy" – across the animal kingdom. Later, the evolutionary biologist Angus John Bateman argued that this could be explained through basic economics.

Eggs, Bateman said, are huge and packed full of nutrients, making them costly to produce. By contrast, sperm are so small they can be produced in their millions.

The bottom line is that males have evolved to be promiscuous and females have evolved to be choosy

This means the stakes of the mating game are much higher for a female, and so she needs to choose her gamble carefully. Meanwhile, the male has sperm to spare, letting him take a gamble wherever he chooses.

The female's investment is even greater if she has to spend time gestating and rearing the young, so she needs to make sure she chooses a mate who will give her young the best genes and the best chances of survival.

"The bottom line is that males have evolved to be promiscuous and females have evolved to be choosy – they should only mate with the best male," says Tang-Martinez.

Some of the first evidence came from an experiment Bateman conducted on fruit flies in 1948. He found that males had a better chance of passing on their genes if they mated with many different partners, whereas the females did not produce any more offspring after their initial mating.

Just like peacocks, female pipefish have evolved bright, colourful markings as a result of sexual selection

The same kind of logic has since explained the behaviours of many different species, from dragonflies and grouse to baboons and elephant seals. Indeed, a seminal 1972 paper on the subject by Robert Trivers has now been cited more than 11,000 times, making it one of the most influential ideas in evolutionary biology.

True, there were always some exceptions. For instance, in certain species of pipefish the female actively courts the male, before "gluing" her eggs to her chosen mate. While she can swim off to find another partner, he spends time nourishing the growing young.

In this case the male invests more in the young than the female does. But such cases of "sex role reversal" were generally considered to be rare

They were also thought to be exceptions that proved the rule. Just like peacocks, female pipefish have evolved bright, colourful markings as a result of sexual selection. These females are also larger than the males, and form hierarchies of dominance determining who can access the "harem".

Still, in the vast majority of species, males were assumed to play the jock while the females waited patiently on the sidelines. This assumption is now under attack by some biologists, who wonder whether it has been shaped by prevailing cultural preferences.

The arguments are particularly troubling when sexual selection theory is used to explain human behaviours.

Even that very first study of fruit flies has come under scrutiny

For instance, some researchers had argued that men are naturally funnier than women, with humour acting as a sexual display akin to bright, colourful plumage – even though any apparent sex differences could easily be the result of sexist stereotyping rather than evolutionary history.

Perhaps biologists just have not looked hard enough to truly understand the complex ways that males and females may interact.

"We haven't really asked any questions about how sexual selection may be acting on females," says Patricia Gowaty at the University of California, Los Angeles. "We know barely anything about what's going on in competitive arenas of females… and the people that have asked seem to think the only way it might act is the same way it does on males."

Even that very first study of fruit flies – the cornerstone of parental investment theory – has come under scrutiny. When Gowaty tried to replicate the results in 2012, she failed to find convincing evidence that the males benefited from being more promiscuous than the females.

In a paper published in April 2016, Tang-Martinez describes many examples in which females do not play by the rules laid down by sexual selection theory.

Female lionesses may mate 100 times a day with a string of different partners

For instance, the females of many bird species had been thought to be exclusively monogamous, with the female faithfully sticking with her chosen partner.

In fact, this could not be further from the truth. Female birds often have dalliances even when in a stable partnership. Among the fairy wren, for instance, just 5% of the clutches will have been fathered by a single mate.

As further evidence, Tang-Martinez points out that female lionesses may mate 100 times a day with a string of different partners. The same seemingly-indiscriminate lust can be seen in many species of primates: not just the famously sexually-active bonobos, but langurs, lemurs and capuchin monkeys. That's not to mention countless studies of beetles, crickets, salamanders, snakes, geckos and house mice.

In all these cases, the females simply do not sit around waiting for Prince Charming, as Bateman had proposed. But the idea that this overthrows Bateman's ideas is rather controversial.

While Trivers says he was surprised by some of the findings in songbirds, he argues that the balance of evidence still hangs in favour of parental investment theory. "There's no question about it, the general theory is alive and well," he says.

It is dangerous to come up with simple explanations for all species

A study published in February 2016 compared the behaviours of 60 different species, and it supports Trivers. "As far as our data go, it's true for [the] vast majority of species," says co-author Nils Anthes of the University of Tübingen in Germany – although he agrees there are many exceptions.

But even in this comprehensive study, Tang-Martinez points out that the overall differences between the sexes were rather weak: according to one measure, they were not even statistically significant.

Furthermore, the number of species studied was still relatively small, she says. The study also did not fully account for the fact that sex differences may change depending on circumstances – like the ratio of males and females within the population, which could influence how the individuals pair up.

In any case, Tang-Martinez is not suggesting that we should throw out the whole theory. Clearly, it is true for some animals. Instead, she thinks it is time to drop the more sweeping generalisations about male and female behaviour. "It is dangerous to come up with simple explanations for all species," she says.

Joan Roughgarden would firmly agree. Formerly known as Jonathan, she began thinking about the evolution of gender at a Gay Pride march in San Francisco, shortly before her gender transition.

When I got into it I was astonished by just how much variation there is

How, Roughgarden wondered, does biology account for such a huge population, normally considered an unfortunate footnote in scientific theory? "When scientific theory says something is wrong with so many people, perhaps the theory is wrong, not the people," she concluded.

The result was her 2004 book Evolution's Rainbow, which examined the multitudinous ways that sex is expressed in nature. It goes far beyond our black-and-white definitions of "male" and "female".

"As a biologist, you think there may be a couple – maybe as many as a dozen – of cases that depart from heteronormative binary," says Roughgarden. "But when I got into it I was astonished by just how much variation there is."

Scientists generally assume that sex is determined by the presence or absence of certain chromosomes. In humans, it is the X and Y chromosomes.

An intersex female bear actually mates and gives birth through the tip of her 'penis'

However, the relevant genes can still be expressed in different ways. The result is that, within any species, many individuals will show characteristics of two sexes.

There are plenty of examples of hermaphrodite invertebrates: leopard slugs are one of many. But Roughgarden has also found that intersex individuals are common among mammals, including red kangaroos, tammar wallabies, Vanuatu pigs, and America's black and brown bears.

According to a 1988 study, between 10 to 20% of female bears have a penis-like structure in place of a vagina. "An intersex female bear actually mates and gives birth through the tip of her 'penis'," says Roughgarden.

These are extreme cases. But many other animals cannot be classified simply as "males" and "females", as if members of each sex will look and act according to the same template.

In Evolution's Rainbow, Roughgarden cites many species that could be considered to have three, four or five separate "genders": that is, animals that belong to the same biological sex but that have distinct appearances and sexual behaviours.

Among white-throated sparrows there appear to be two kinds of males and two kinds of females

For instance, the bluegill sunfish has three male genders, each of which reproduces in a different way. The largest, most aggressive males show off a flashy orange breast, and actively court females to lay eggs in their territory. In contrast, the smallest males are duller in colour and have no territory of their own, but will dart into one of the dominant male's territory's to fertilise some of his mate's eggs.

It is the medium-sized males who are the most surprising. They appear to actively court the larger males with a dance in the water. If the big male accepts their advances, they may then form a ménage a trois with an approaching female, with both males both fertilising her eggs.

Why would the larger male team up with the weaker partner in this way? One possibility is that his presence helps to reassure the female that the larger male is not too aggressive. For this reason, Roughgarden describes these medium-sized males as marriage brokers.

In other species, a range of "genders" may offer a greater variety of parenting styles.

For instance, among white-throated sparrows there appear to be two kinds of males and two kinds of females. Each is defined by the colour of the stripes in their feathers, their relative dominance or aggression, and the amount of parental care they offer the young. Noticeable differences can also be found in their brain structure.

A "cross-dressing" male with more feminine features is often described as "deceptive"

The result is a number of different possible couplings, each dividing the responsibilities of parenting – such as feeding and defending the young – in a different way.

Roughgarden's book offers many similar examples among hummingbirds, wrasse and tree lizards, each showing a spectrum of genders.

There is also a growing list of species that engage in homosexual behaviours. With such variation, it begins to make less sense to discuss "male" and "female" behaviour as if it means the same thing for all species, or even all individuals within a species.

Roughgarden says evolutionary theory has not done justice to this broad spectrum.

Even when biologists have noted these exceptions, they tend to describe them in pejorative terms, she says. For instance, a "cross-dressing" male with more feminine features is often described as "deceptive".

The living world is made of rainbows within rainbows within rainbows

"Again, it's this locker-room story – you go to a bar, see this cute-looking girl, and it turns out to be a guy, so you feel fooled and taken advantage of," she says.

Still, Roughgarden thinks attitudes are changing, albeit slowly. Today, homosexual behaviour in animals is attracting more research, and she hopes the same will be true of sex and gender roles more generally. "There's just beginning to be a discussion about non-binary gender variation," she says.

By ignoring this variation, we simplify the story of evolution and neglect some of nature's most astonishing adaptations. As Roughgarden puts it: "The living world is made of rainbows within rainbows within rainbows."

This story is part of our Sexual Revolutions series on our evolving understanding of sex and gender.

Join over five million BBC Earth fans by liking us on Facebook, or follow us on Twitter and Instagram.

If you liked this story, sign up for the weekly bbc.com features newsletter called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.