Charles Ryan has a clinic in San Francisco at which he regularly relieves men of their testosterone. This “chemical castration”, as it is sometimes known, is not a punishment, but a common treatment for prostate cancer. Testosterone doesn’t cause the disease (currently the third most deadly cancer in the UK), but it fuels it, so oncologists use drugs to reduce the amount produced by the testicles.

Ryan gets to know his patients well over the years, listening to their concerns and observing changes in them as their testosterone levels fall. Because it involves the so-called “male hormone”, the therapy poses existential challenges to many of those he treats. They know that every day, millions of people – from bodybuilders and cheating athletes to menopausal women – enhance their natural levels of testosterone with the aim of boosting their libido, muscle mass, confidence and energy. So what happens when production is suppressed? Might they lose their sex drive? Their strength? Their will to win?

The fears are not always groundless. Side-effects can also include fatigue and weight gain. But Ryan has witnessed positives, too. As professor of medicine and urology at the University of California, he has noticed that the medical students who have passed through his clinic in the 18 years that he has been treating prostate cancer invariably comment: “Dr Ryan, your patients are so nice.” He replies, jokingly: “It’s because they don’t have any testosterone. They can’t be mean.”

Could there be some truth in that glib reply? Ryan knew his patients hadn’t always been so kind. Before being robbed of their testosterone, they might have been personable and adept at small talk, but they weren’t nearly as interested in other people. He could feel a hypothesis coming on: that as men’s testosterone levels lower, their capacity for empathy will rise. In his new book, The Virility Paradox, he argues that “the fact that reducing testosterone in these ageing men may lead to increased empathy, more emotional engagement in relationships and a softening of aggression could be something of a silver lining”.

Ryan started measuring his patients’ “empathy quotients”, using a survey developed for studying autism. It’s too early to release detailed results, he says, but “we do see increases in the empathy scores in many patients on the treatment”.

He also dived into the literature on testosterone, attempting to understand what exactly was happening to them. Try as he might, however, he found little conclusive evidence for many of the claims made about testosterone, such as a link between hormone levels and risk-taking or sexual violence. “There’s so much ambiguity in the science,” he says. Many of the studies had been carried out on disappointingly small numbers of people.

Ryan is one of several researchers who are questioning the accepted wisdom about testosterone. It is often wheeled out as an excuse for patriarchal society, in arguments along the lines of: women, with their lower testosterone levels, have evolved to nurture and multitask in the domestic sphere, while men are hardwired to take risks, compete and furnish as many women as possible with sperm, thus ensuring the future of the species. But, as Ryan points out, “obviously behaviour and cognition are extraordinarily complex and don’t pivot on one molecule”.

The psychologist Cordelia Fine makes a compelling case that it is our culture rather than our hormones that most influences gendered behaviours. As she writes in Testosterone Rex (winner of the Royal Society’s science book prize for 2017) testosterone has been blamed for the financial crash of 2007-08, yet studies show that, although women have lower levels than men, they can have a higher appetite for risk – even when it comes to financial decisions. She uncovered similar stories when it came to the evolutionary need for more sexual partners (more babies get made if women sleep around, too) and competition for status.

Fine’s pluck in challenging the scientific status quo could itself be viewed as classic testosterone-fuelled behaviour. She has cojones, you might say. She asserts that many typically female behaviours, such as deciding to have babies, are riddled with risk, only women’s risks don’t seem to count when it comes to testosterone mythology.

While Ryan comes at the subject from a different angle, both authors highlight how little research there is into testosterone in women. And yet we know it is vital to them (for example, oral contraception reduces testosterone levels, which can lead to low mood and libido). It can also influence sexual orientation, Ryan writes, with studies showing that “self-described lesbians are likely to have [indications of] higher foetal testosterone levels than women who identify as heterosexual”.

The lack of research, meanwhile, hasn’t prevented a fierce debate about testosterone’s role in women’s sports, with high levels seen as conferring an unfair advantage. The athlete Caster Semenya, who won a gold medal in the women’s 800m at the 2016 Olympics, has extremely high natural testosterone levels for a woman. She had to prove her gender, and medically suppress the hormone before competing (although this ruling is currently suspended). Meanwhile, in 2016, the International Olympic Committee ruled that transgender women could compete without having had surgery, on condition that their testosterone levels were no higher than cisgender women’s.

Not that testosterone levels are consistent in anyone. They rise and fall all the time, according to season, health, relationship and parental status, age, time of day (higher in the mornings) and emotional responses. When a man hears a woman cry, his testosterone goes down. When a person cares for their child, the “bonding” or “love” hormone oxytocin rises, while testosterone falls. If a threat to status or territory is perceived, testosterone rises again. It’s the situations, the culture even, that seem to pull the hormone’s strings. Testosterone, in both men and women, also works in a “feed-forward” system: when you win at something, you get a spike in testosterone that as well as making you feel dominant and confident, increases your sensitivity to the hormone – encouraging further swagger and quests to win.

Another of the hazards when studying testosterone is that there are three significant measures of how strong its force is in you. You can check levels in the bloodstream, but we already know how they fluctuate. The second measure is the number and sensitivity of androgen receptors, which vary significantly from person to person. (Testosterone is one of three hormones known as androgens, and receptors are what allow them to act on the cells in our bodies.) Third is the amount of testosterone to which we were exposed in the womb, most of which is produced by the foetus itself. This exposure is harder to gauge, although the difference between the lengths of the index and ring fingers is often used as a marker. The smaller the difference, the theory goes, the greater that foetal exposure.

This complex web, says Ryan, means that responses to hormonal suppression therapy “are highly variable, based on [individuals’] intrinsic biology. I have patients whose testosterone I take away and they don’t have any [unwanted] side-effects. In fact, they say: ‘I feel better. My brain is less clouded with intrusive thoughts about sex and things like that.’”

In a sort of mirror-image experiment, the writer Ann Mallen recently told how she accidentally rubbed testosterone cream into her skin every day for a month due to a mixup at the pharmacy. She wrote in the Washington Post that her sexual appetite became a constant distraction, as did her new persistent bouts of “irrational anger”. She concluded that “underneath the high-pitched whine of our sex hormones, we are neither [male nor female].”

Because women are more responsive than men to supplemental testosterone, they were used in one of the key studies into how testosterone essentially removes the burden of empathy from moral decision-making. It’s known as the “trolley car experiment”. Picture a runaway tram hurtling down the tracks towards five unsuspecting workers. There’s a lever that would divert the tram to another track, but there’s someone working on that track, too. “You have to kill somebody to save five others,” says Ryan, and you have to act fast.

The researchers at Utrecht University gave some of the subjects a shot of testosterone the night before presenting them with the dilemma. “The number of respondents who were willing to kill in order to save people, and their confidence in carrying out the act were enhanced,” says Ryan. “And the equivocation they demonstrated was significantly reduced.”

This isn’t to say that empathetic people can’t make tough decisions. Hormones are a bit-part in a complex cognitive picture. Aaron, a high-flying lawyer treated by Ryan, was adept at suppressing his empathy in order to win a case. But as his testosterone dissipated, he grew more caring and started asking Ryan about his family. At one appointment he asked whether “getting emotional” was a side-effect of his treatment, after he had wept at the end of a long-distance visit to his elderly mother. “Like many patients,” writes Ryan, “Aaron regards these developments with a measure of surprise. Hormonal therapy hasn’t been as bad as he expected, and he admits he has actually come to appreciate some of the effects it has had on him.”

However, this outcome posed one worry for Ryan. “A major case is heading to trial and Aaron is the lead attorney. Will having a testosterone level at 10% of normal affect his performance?” he writes. The answer, it turns out, is no: Aaron had not lost his killer instinct in the courtroom.

You get the sense that Ryan sees toning down testosterone as a force for social good. Take his patient Marcus, an octogenarian who is still a keen runner. When his cancer risk was sufficiently low, he came off hormone suppression therapy and started taking supplemental testosterone to counter its effects. “He would come in and talk about his half-marathon, weightlifting, his younger girlfriend,” says Ryan. “He never talked about anybody but himself.” Eventually, he had to quit the supplements because his markers for cancer rose again. “He disappears for more than a year, and comes back and is now taking care of his daughter, picking up his grandkids and being a nice grandpa. I think it is misguided for ageing men to think they should necessarily want to have high testosterone levels, because they may pay a price for that in terms of their relationships. They may be more self-centred, lack empathy.”

But again, it’s complicated and depends on the individual. “Many men, as they age, feel sluggish and lose muscle mass, lose their self-esteem, so I don’t say we shouldn’t ever use supplemental testosterone.”

It’s estimated that one in 10 men aged over 40 in the UK have low testosterone levels, which is in a large part related to obesity. “Fat tissues will produce an excess of oestrogen,” says Ryan, which leads to reductions in testosterone. Artificially boosting the latter could help them lose the weight, but any other benefits, Ryan warns, could be transient. “A study published in the New England Journal of Medicine … found that while [their participants on supplemental testosterone] felt good at first and their libidos went up, there weren’t long-term beneficial effects.”

And, of course, they may end up impairing their capacity for empathetic relationships. But there are non-medical ways to boost empathy. In Testosterone Rex, Fine cites a 10-year US study targeting boys at high risk of behaving antisocially later in their lives. Some of them were given coaching to improve their emotional resilience, relationships and educational performance, while their parents were trained to manage their children’s behaviour. The goal was to enable the boys “to respond more calmly and less vociferously to provocation”. Years later, when the participants had reached their mid-20s, about 70 were deliberately provoked by someone stealing points from them in a game. Not only were the group who had been given coaching as boys less likely to retaliate; their testosterone levels rose less.

Another way, according to Ryan, is to do more childcare. Testosterone levels are 33% lower in fathers of newborns than in non-fathers, making way for a good 25% more oxytocin. This hormone, says Ryan, induces men to spend more time with their children and respond more quickly to their needs. It enables fathers to play more closely with their children, and get less rattled if they cry. (One of Ryan’s patients started getting down on the floor to play with his grandkids for the first time during hormone suppression therapy.) Romantic love, friendship and pet ownership open the floodgates to oxytocin, too (even a dog’s oxytocin rises when it stares into its human’s eyes). “Less testosterone, more oxytocin, more bonding,” says Ryan. That’s another, perhaps more fulfilling, feed-forward system.