Not long before Christmas, newspapers, radio stations and TV networks were abuzz with some exciting news: the male pill could soon be a reality. Researchers at Monash University in Melbourne had discovered a way to block a couple of proteins that controlled the movement of sperm. The result was complete but temporary infertility – there would be no long-term effects on the "viability of sperm or the sexual or general health of men", assured Dr Sabatino Ventura, the head researcher.

The medication would be taken orally, just like the female contraceptive pill, and that was not the only attractive feature. Crucially – for men at least – taking it would have no impact on libido or sexual behaviour. "If anything, their performance should improve with the strategy we are using," Ventura, who looks like a handsome, rugged version of the Count from Sesame Street, told Australian TV.

The response to the research was not quite hysteria, but just a couple of notches below. In the Guardian, Daisy Buchanan rejoiced: "Women: throw your knickers in the air… No more doctor's appointments, repeat prescriptions, injections, mood swings and irritability for us." The Daily Mail launched a debate on whether women would ever trust men to take it. But despite any niggling reservations, everyone was in agreement that a significant development in the equality of the sexes was set to take place: contraception could at last become a fully shared responsibility.

I reach Ventura on the telephone and he comes across as somewhere between elated and overwhelmed. He first had the idea for the treatment as a postgraduate researcher 25 years ago – he's 49 now – and since then he had been "waiting for technology to catch up with me". He never fully believed the day would come when he would be fielding calls from across the globe; Colombian reporters, for some reason, had taken a special interest. "I thought I'd get local media because the research was done in Melbourne," he admits. "But I didn't expect so much from around the world. That was quite a shock really."

Male contraception is tricky, Ventura explains, because men make about 1,000 sperm every second and you need to stop them all. Women produce one egg a month, so interfering with that is a simpler medical challenge. "Remember," he notes, "if just one of those sperm gets through to the egg, you've lost your contraception."

Ventura's research was praised by fellow scientists for offering an elegant solution to this longstanding problem. I ask him for the most unscientific explanation for how his pill would work. He tells me to picture an aeroplane: the plane taking off is ejaculate leaving a male body, but before that, it has to be loaded up with passengers, ie sperm. His medication silences the announcement in the airport terminal telling people to board. "So the plane's taking off," he says, "but it's got no passengers on it."

It does sound ingenious. Here is contraception that is non-hormonal, does not require surgery, and is fully reversible. And because it would be targeting the two proteins with vasodilators – medicines that allow increased blood flow to the penis like, say, Viagra does – it could help boost erectile function. It is a win-win for men and women: one of health's most enduring holy grails was close to being found.

But then I speak with Elaine Lissner, founder of a San Francisco-based non-profit group called the Male Contraception Information Project. Lissner, a fast-talking – and sometimes jaw-droppingly frank – woman in her 40s who traces her interest in male contraception to her student days at Stanford University in the late 1980s and "horror stories" of female friends taking the pill, is keen to highlight some of the hurdles that Monash University's male pill would face before eventually one day coming to market.

The Australian research was conducted on mice who'd had their DNA tweaked: what the scientists have to do now is create a human equivalent that relies on a chemical component rather than genetic tampering. This would be a significant undertaking that could take a well-funded team at least a decade.

"This is very early-stage work," Lissner warns. "When you hear 'five to 10 years' you might as well figure 'maybe never'. If we actually want a male contraceptive, we have to finish the job on what we've already started."

Lissner is not entirely impartial. She has invested heavily in procuring the international rights for a male contraceptive treatment developed in India called Vasalgel. Nevertheless it's hard to dispute her circumspection. The news agenda loves the story of a "new male pill": every couple of years, almost without fail, there is a huge rush of interest in a scientific paper that hits the headlines. But then it disappears, never to be heard of again. A cursory online search finds a graveyard of such treatments, all of which were once hailed as earth-shattering advances.

Immediately a flurry of questions present themselves. Half a century after the pill, why has it proved so complex to come up with a male equivalent? Why is there so much media interest in male contraception and so little funding? And if we're honest, is there any point in developing a male treatment when birth control will always ultimately be a physiological concern for women, not men? The answers turn out to be surprising and often counterintuitive.

Male contraception has been a preoccupation for longer than you might imagine. In the 5th century BC, Hippocrates had some success with heating a man's testicles in a hot bath. The father of western medicine might not have had proof that sperm is much happier several degrees below a man's core body temperature – we learned that in the 20th century – but the "heat method" has become an enduring and often successful way of preventing conception. Dr Marthe Voegeli, a Swiss doctor working in India from the 1930s to the 1950s, advocated that men put their testicles in a shallow bath warmed to 46.5C for 45 minutes a day for three weeks. Her tests – on hundreds of Indian men and a small handful of English, American and Scottish volunteers – indicated that this would lead to around six months of sterility.

Other esoteric methods have shown potential, too. Pedanius Dioscorides, a Greek physician who worked as a Roman army doctor in the 1st century, detailed in his five-volume De Materia Medica – which would become a health bible for the next 1,500 years – that drinking juice made from cannabis would reduce sexual potency. Recent tests, on rats, have shown that marijuana can reduce sperm counts by half. Another age-old treatment is papaya seeds, which at high doses have rendered langur monkeys azoospermic – that is, there's no sperm in their ejaculate – before a full recovery in less than five months.

All clear: Elaine Lissner of the Male Contraception Information Project with a vial of Vasalgel. Photograph: Male Contraception Information Project

At this point you may be wondering – especially if you are a woman – why no progress has been made with research that in some cases is a couple of millennia old. Sure, sitting with your balls in a scalding bowl of water doesn't sound like a lark, but then neither are the known side-effects of the pill. Perhaps the most enduring issue with the heat method is that it is hard to monetise it: you can't patent the procedure, only come up with a creative way of applying it, which drastically reduces the appeal to pharmaceutical companies. Elaine Lissner has also wondered aloud in the past whether male researchers are less inclined to trumpet an uncomfortable, awkward-sounding treatment for men than they would be for women.

A further problem with the heat method is that academics working in the area have traditionally been the butt of jokes from their peers. "Research on it is not prestigious," states the Male Contraception Information Project's website, "and often earns scientists derision from their colleagues." There's a suggestion that a dismissive attitude still persists to this day, according to Ventura.

"You've already got a good female contraception, so people don't think we need a male equivalent," he says. "It's more like: 'Oh, that would be handy, but we could probably do with curing a whole lot of other diseases beforehand.' If you are writing a grant proposal to develop drugs to cure cancer or other chronic illnesses, you have a better argument because you are saving people's lives."

Back in November 2006 Dr Nnaemeka Amobi, a physiologist at King's College London, was the hot-shot scientist who had cracked the male pill – the product of 17 years in development. Heading a team alongside his King's College colleague Dr Chris Smith and Professor John Guillebaud, an expert in reproductive health from University College London, Amobi received an almost-identical response to the clamour that Dr Ventura recently experienced.

Dr Amobi's research was also at a very early stage – it had yet to be tested on animals even – but once more there was an attractive theoretical purity behind his idea. It derived from research published in 1968 about thioridazine, a drug used to treat schizophrenia. Thioridazine had fallen out of favour as a mental-health treatment – it could cause an irregular heartbeat that led to sudden death and was withdrawn from use in the UK in 2005 – but it did have a surprising side-effect: male patients who took it typically experienced their orgasms without any ejaculate.

Guillebaud, a lively 73-year-old who was born in Burundi and now lives in Oxford, explains: "So this man went to a psychiatrist and said: 'This stuff you've given me, I don't know how it's helping my mental case, but when I masturbate' – only he used a shorter word beginning with 'w' – 'I'm dry.' And yet he said: 'I enjoy doing it, so I'm still doing it, but I get nothing out.'"

Amobi heard this story and he also discovered research from the early 1980s that there was another, very different drug with a similar side-effect: phenoxybenzamine, a medication for high-blood pressure. For some reason, these two treatments disrupted the so-called Mexican wave that a man experiences during sex to produce a "dry orgasm". All Amobi and his team had to do was to find other compounds that have the same effect on ejaculation without side-effects of their own (thioridazine caused drowsiness and phenoxybenzamine could lead to dizziness) and without affecting other muscles in the body. In news reports at the time, he suggested that this might take them five years.

Eight years on, Amobi's male pill has not come to fruition – and in fact its only meaningful progress since 2006 has been a prototype test on rams. The stumbling blocks are recurring ones, at least in the world of male contraception. Despite all the publicity that research received, from reports in most British newspapers to coverage on CBS News in America, Amobi and Guillebaud have found it impossible to secure grants to develop the treatment. They are currently asking for around £200,000 and cannot even raise that. This is despite evidence that their pill could have benefits beyond contraception, most notably that it could reduce the transmission of HIV.

Numerous explanations can be found for the failure of Amobi's "clean-sheets pill" to get off the ground, but probably the most devastating blow is studies hinting that around half of men simply could not imagine a semen-free orgasm. There is speculation over whether this is tied to the increased popularity of internet porn or with ejaculation being inextricably linked to a man's sense of masculinity, but whatever the reason, it has been damning for further investment.

Lissner, for one, is apoplectic that this might be the barrier. She comments: "I'd like a policy-maker to read this in their morning paper and say: 'I'm not interested in [funding] this method because 50% of men will say: 'Sorry, no money shot, no dice.' I'd like somebody to make a conscious decision: 'OK, we are turning down a potential HIV transmission-reduction pill for the want of £200,000 because we think all men need to see their cum come out.'"

I ask Guillebaud what he and Amobi do these days without financial support. "We can stay knowledgeable and up-to-date in the field," he says, "but really we're just twiddling our thumbs because there's no money at all."

All clear: Brian MacDonald, one of the first men in the UK to trial a male pill. Photograph: Murdo Macleod for the Guardian

The question remains, however: why is male contraception important? When women can choose from pills, patches, injections, more than a dozen methods – do we really need something for men? A recurring issue in clinical tests for a new male pill has been uncomfortable side-effects. So far, the majority of attempts to find a solution have been hormonal – though not Ventura's or Amobi's – typically using some combination of progesterone, to inhibit a man's sperm, and testosterone, to ensure he retains his inherent manliness. This has proved problematic, leading to mood changes, depression and weight gain – and also to an increased libido.

Hormonal trials have an extensive history of foundering, too. In 1997 a research team at Edinburgh Royal Infirmary, led by Dr Cameron Martin, embarked on final tests for a progesterone pill that were closely tracked in news-papers. Dr Martin predicted it would be available for men as early as the new century – obviously he was an optimist – but warned that we should never expect a silver bullet. "The male pill will have some side effects," he told the Guardian. "It won't be a wonder medicine."

Again, women who have taken the pill for years might well respond: "So what?" Or even: "About time." But the recurring – somewhat shaky – riposte is that there is a substantive, even ethical, difference between the sexes here: the side-effects of women's contraception need to be set against the positive health benefits of avoiding an unintended pregnancy (the same cannot be said for men directly, making it more complex to justify the negative symptoms of hormonal treatments). A cynical person might speculate there is an innate chauvinism involved, but it has been enough to disrupt one of the largest trials, sponsored by the World Health Organisation in 2010, which was forced to end prematurely on health grounds.

Right now there are three options for men who would rather not produce offspring – condoms, withdrawal and sterilisation – and none of them is exactly flawless. A vasectomy, although incredibly reliable, needs to be considered a permanent procedure, while withdrawal and condoms are both problematic. With perfect use, the failure rate of condoms is 2%, and that figure rises to 18% for couples who rely on them exclusively. According to estimates, there are 80m unintended pregnancies in the world each year, which is – by coincidence – the same figure as the global population increase. A large proportion of these babies is in the developing world, but it is also a first-world concern: last year the Guttmacher Institute found that a staggering 49% of the 6.7m pregnancies per year in the United States are unintended. Contraception scientists suggest that if we want to reduce overpopulation of the planet, unplanned newborns are somewhere to start.

The usual rejoinder is that men don't want to take control of contraception, but that view is not reinforced by the studies. According to a 2005 survey of 9,000 men in four continents, 55% said they would be interested in a male pill or something similar. An increasingly prevalent factor in developed nations is that men are now financially responsible for any children they father – in the UK up to the age of 17, while in the US it can translate to 40% of a man's salary for 18 years. "If you don't have much money, that's a lot of money, and if you do have money that's even more money!" notes Elaine Lissner. "As a man, you're pretty vulnerable. People forget pills, and it's tricky because men can't know whether women were subject to human nature or had other motivations. But the result is the same either way."

Lissner would like to see a range of options for men that compares to the "contraceptive supermarket" that now exists for women. Men looking for a one-night stand would want a pill that works quickly and wears off just as fast, such as those treatments proposed by Amobi and Ventura, while others in a stable relationship might be tempted by a longer-lasting option that they wouldn't have to worry about forgetting to take and which would work out as better value in the longer run.

Male contraception is tricky because men make about 1,000 sperm every second and you need to stop them all. Photograph: London Scientific Films/Getty Images

One such treatment is the non-hormonal Vasalgel, Lissner's preferred method, which is essentially a reversible vasectomy whereby a polymer gel is injected into a man's vas deferens (another injection flushes the gel out). Pioneered in India by an eccentric scientist called Sujoy Guha, it has been used – on humans – since 1989 and has been claimed to have been 100% successful to date. Lissner changed the name (it was formerly known as Risug) and her team tweaked the recipe, recently conducting experiments on rabbits. This month they will start tests with baboons, and she hopes it could be available in 2016.

Much as there is competition to become the first successful male contraceptive, there is also a sense of camaraderie among practitioners in the field. Many have been pursuing a solution for decades – the leading experts are mostly at retirement age or beyond – and would be happy to see any of the rival methods cross the finish line. "I don't feel like it's a race," says Ventura. "If someone else came up with one first and got it to clinical trials or even to market, that would help everyone else. The other pharmaceutical companies might start saying: 'Hang on, we want to get into this, too.'

"Will it happen in our lifetime?" he ponders. "I wouldn't outlay a lot of money on it, but I'm reasonably confident."

While the international media fixates on treatments that may be at least a decade away, a different form of male pill could, in fact, be a reality in the very near future. In Indonesia, tribesmen in the province of Papua have known for years of the magical properties of a shrub called gandarusa. If a man chews its leaves he can have sex and not make his partner pregnant. Indonesian researchers began looking into the properties of gandarusa in the 1980s and conducted animal and human trials through the 1990s. The active ingredient of the plant was isolated, chemically synthesised and has now been turned into pill form. In the most recent tests, 350 couples took the pill for 30 days and there was a 99.96% success rate. Normal fertility returned three days after the men stopped taking the pills.

In Indonesia – unlike everywhere else – the scientists enjoy significant government support. Overpopulation there is a major concern: the archipelago was home to 246 million in 2012, up from 145 million in 1980. It is now the most populous nation in southeast Asia and the fourth-largest on the planet. It also has a very low incidence of male involvement in family planning: just 1.8% of men use condoms – they have a negative association with casual sex – and 0.2% of men have a vasectomy.

All of which means that gandarusa is taken very seriously by Indonesia's Health Ministry. Antara News, the national news agency, reported in August last year that a purified version of the plant extract has been handed over to Indofarma, a state-run pharmaceutical company, for the final phases of production and distribution licensing. Because gandarusa is likely to be classified as a herbal medicine, the process could be expedited faster than normal. It may even be available to Indonesians on prescription later this year.

Of course it would be many years before gandarusa satisfies the necessary approvals in the west, but Lissner believes that some eager men might skip that and find ways to import the pills direct from Indonesia. Perhaps then pharmaceutical companies over here will realise there is a demand for – and money to be made from – male contraception after all.