In the world's largest democracy, homosexual activity is still a crime punishable by life imprisonment. A few weeks ago, the Indian Parliament saw MPs of both major parties scurrying for the exits to evade a debate on this anachronistic ban, introduced by the British Raj in the 19th century. The occasion was a failed attempt by a maverick backbencher to introduce a private member's bill legalising homosexuality. Indian politicians are in no hurry to debate a reform that would annoy religious extremists. And yet India, always a land of contradictions, allows Gay Pride marches in most major cities, has vibrant gay pressure groups and publications and officially accepts people who are transgender.

When I grew up in India's most cosmopolitan city, Bombay, in the 1960s, the very mention of homosexuality was taboo, and absolutely no one was “out”. I remember the frisson when, as college students, we discovered and staged plays with gay allusions and characters by authors such as Tennessee Williams. My own Westernised parents were sexually liberated and tolerant, but they were very much the exception.

Last month, I made a personal journey home for a BBC Radio 4 documentary, to explore how dramatically India's sexual landscape has changed in recent years.

Bombay is India's Big Apple and, like London or New York, is now a magnet for young people wanting to live exciting new lives free of their families. The northern suburbs are home to many of the thriving call centres we love to hate; and the anonymity and freedom they offer has been particularly valuable for young gay jobseekers. “How you look doesn't matter in call centres,” explains Vikram Doctor, a senior, openly gay journalist with the The Times of India. “You can be a butch lesbian or a camp, effeminate gay man, all that matters is how you speak. It's meant that youngsters just out of college can get well paid jobs free of parental pressure. Suddenly you have all these young people earning as much or more than their parents, and that's been a lifeline for lots of young queer people in cities like Bombay and Bangalore.” Vikram also points to the liberating impact of the internet, with its dating sites and social media replacing the hazards of cruising and the risks of extortion gangs and police entrapment.

During my trip, I took part in Bombay's annual book festival. I was on a panel about equal rights, along with the prominent human rights lawyer Anand Grover. He explained the muddled legal backdrop to homosexuality – indeed, all sexuality – in India. “How many people here have had oral sex?” Grover asked our amused audience. “Put your hands up! You're all criminals under Section 377.” That's the provision of the Indian Penal Code which forbids not just gay sex, but any sexual act other than penile-vaginal intercourse. Grover led a legal battle in the Delhi High Court to get Section 377 declared unconstitutional, because it violates citizens' rights to equality, privacy and dignity guaranteed in India's written constitution.

Zareer Masani at Cathedral School in Bombay where he suffered homophobic bullying

The case dragged on for almost a decade and had a profoundly radicalising impact on gay Indians and on public attitudes to them. “When the case began,” Grover tells me, “no one was willing to come forward openly to say they were gay. Today, because of that case, things have changed completely. The media coverage became ever more sympathetic to the cause of LGBT rights. Debates were being held on television, where mothers and fathers and sisters would say, 'I have a family member who's gay, and that's perfectly OK'.” The court judgment in 2009, which lifted the ban for consenting adults, was strongly influenced by arguments that the existing law was a blackmailers' charter and that HIV treatment was being impeded by fear and secrecy among gay people. The judgment was accepted by the then Congress government, and gay Indians across the country celebrated the end of the ban with marches and parties. But it was a short-lived victory, because a motley collection of religious fundamentalists appealed to the Supreme Court, which overruled the Delhi High Court and decreed that only Parliament could change the law.

The position now is that the Supreme Court is mulling over a “curative” petition to reconsider its own judgment, a process which could take years. Meanwhile, are gay Indians being forced back into a twilight zone of secrecy, fear and extortion? Justice Ajit Shah, the courageous judge who presided over the Delhi High Court case, is confident that the genie is out of the bottle once and for all. “Section 377 has no place in modern India,” he declares, “and it can and must be replaced. Many people present in court when I gave my judgment broke down and wept. The middle class has stopped joking about homosexuals, and the language of homophobia is ebbing away.”

As well as evolving middle-class attitudes, global media has had a huge impact on social attitudes across large swathes of middle India. One touching example was a father and son who I met in Bombay. Sushant, aged 24, was chosen as Mr Gay India 2014 in a beauty pageant that's now in its third year. Tall, slim and androgynous, clad in the latest fashions, he would look more at home in Soho than on the streets of Bombay. But he has no qualms about dressing and looking as he pleases. “I wear what I want, and I'm the way I want,” he declares defiantly. “Why should I be scared? I don't want to be a straight-acting gay man. I'm a very flamboyant individual, and when I travel abroad they say: 'Here comes the curry queen'. So what?”

Sushant's father, Pradeep, is a middle-aged business executive. His response to his son's sexuality was anything but conventional. “When he told me he was gay,” he says with a laugh, “I told him: 'I love you even more'. After all, he's my child, and I brought him into this world. I always say: 'He's gay, and I'm happy'”.

That's a remarkably warm parental welcome, even by Western standards, and a reminder that India's close family structure can at times be supportive, rather than just oppressive. When it comes to family acceptance of homosexuality, it's wrong to generalise about Indian religious or class differences, says Parmesh Shahani, author of an openly gay autobiography and founder of the Gay Bombay pressure group. He's now director of the Godrej Culture Lab, a large centre funded by one of India's biggest business corporations, which hosts many gay events, including Bombay's annual Kashish Queer Film Festival.

“My first boyfriend came from a really conservative Muslim family,” Parmesh (a Hindu) tells me, “and his parents outed him. They sat him down and asked: 'Are you gay, and are you dating Parmesh?' He said 'Yes' to both, at which they said: 'Well, why don't you call him over for dinner?' And so we had this incredible relationship where I would go to his home, have dinner with his parents, and then we'd go out on a date, which was something that most of our heterosexual friends couldn't do.”

The city’s gay and lesbian parade takes place every year (Getty)

Evidence of changing attitudes is very noticeable at my alma mater, the Cathedral School, founded in Bombay by Anglican missionaries in the 1860s, around the same time that the notorious Section 377 was enacted. As a child there, I was a loner, suffered homophobic bullying, avoided games and escaped into a private world of books, art and history. Today it's a very different world, as I step into the bright and cheerful office of the principal, Mrs Isaacs, who bears no resemblance to the apoplectic English headmaster who caned us when we misbehaved. I expect her to be embarrassed when I ask if homosexuality is discussed in the classroom. “Of course it is!” she laughs. “When I was growing up, we never spoke of such things, but today it's in your face, and children now are very open about it. In fact, some of our children wanted to go out and demonstrate against the Supreme Court judgment.”

The Cathedral was and still is Bombay's most elite school. “Prima in Indis, gateway of India, door of the East with its face to the West,” the opening lines of the school song proudly remind one that this cosmopolitan city has been India's intellectual and commercial hub for almost two centuries. But how representative of Bombay, or indeed of India, is the affluent, internet-savvy, sexually liberated world of the city's Westernised middle class? And can we assume that things can only get better for a younger generation of gay Indians?

Deep in Bombay's less affluent northern suburbs, I find a group of radical, lesbian feminists, called Labia, which describes itself as “a queer, feminist, lesbian, bisexual and transgender people's collective”. Much of their language and perspective reminds me of militant gay liberation groups in Britain in the late 1960s. Labia argues that what they call “privileged voices for gay people” are just seeking assimilation into “a Hindu, upper-caste hetero-patriarchal” status quo. For them, the most oppressive threat to young gay people comes, not just from the state, but from their own family.

I discuss this with Chayanika, a middle-aged lesbian, and her transgender partner Shals. They argue that the traditional Indian family structure results in abuse, torture and forced marriage, especially for young women, who are not even recognised as sexual beings. They point out that traditional family control is being strengthened by new market forces that are making affordable housing impossible to find in most Indian cities. “The big difference with the UK,” says Shals, “is that an abused teenager who runs away from home here has nowhere to go, because there's no entitlement to social housing.”

Groups on the left such as Labia argue that Hindutva, the Hindu nationalist ideology of India's ruling party, the BJP, is threatening all Indian minorities, including gays. Other more mainstream gay organisations prefer to remain politically neutral. The largest gay organisation in India is the Humsafar Trust, with a staff of almost 300, located in the Bombay suburb of Santa Cruz. Up a dingy staircase in a rundown tenement, you enter a large open-plan area, partitioned into a dozen smaller cubicles. These are the treatment and counselling rooms where Humsafar (“travelling companion” in Hindi) provides a wide range of free services from HIV testing and treatment to psychological counselling and legal advocacy.

Humsafar's CEO, Vivek Anand, shows me round and explains how the organisation began in the 1980s as a gay dating club called Bombay Dost (friend) and then expanded through its HIV work, which eventually won it substantial government funding. When I ask if gay causes are being squeezed under the present BJP government, which swept to power in 2014, Vivek points to the gay film festivals and pride marches that the government continues to license and the HIV funding that Humsafar continues to get. “But let's be clear,” he concludes about the politicians, “no one really wants us!”

Later I talk to Sonal Gyani, who's in charge of Humsafar's legal advocacy service. She has hands-on experience of representing gay people in police stations and argues that both extortion rackets and police harassment have got worse since the court cases about Section 377. The lifting of the ban, followed by its reinstatement, and all the publicity around it, has made potential blackmailers more aware that homosexuality is a criminal offence. A typical scenario, says Sonal, would be for someone to arrange to meet you for sex, then arrange for their accomplices to turn up and catch you in the act, take photos and threaten to show them to your family, employers or the police. The police themselves, she adds, sometimes join in the blackmail and take their cut.

The day I'm visiting Humsafar is the second anniversary of the Supreme Court judgment reinstating Section 377. There's a community evening to mark the event, with Bollywood-style singing and dancing and individuals taking the mic to share their experiences of being gay as well as to remember how they felt when they heard the court's shock judgment. The crowd of about 100 brings together mostly young gay men from across Bombay's less privileged, northern suburbs.

Among them I also spot a couple of hijras, the very distinctive Indian eunuchs and transsexuals who dress as women. It's a reminder that, in 2014, the same Supreme Court that reintroduced the gay ban reached a remarkably enlightened judgment recognising the right of transsexuals to be treated as male, female or “third gender”, according to their own individual choice. The court also ruled that transgender people are entitled to positive discrimination by the state in education and jobs.

The legal anomaly that accepts gender change but bans homosexuality cries out for reform, as most of the Indian media has been demanding. One of the most vocal advocates of change has been the veteran television journalist, Karan Thapar, fondly called India's Jeremy Paxman because of his talent for grilling politicians on the popular shows he presents almost every day of the week. “I don't think the media has a huge impact on the attitude of politicians,” he complains. “They're more concerned about their unwillingness to offend voters, and most of those voters continue to be conservative-minded. For politicians to muster both the moral conviction and the political courage to tackle this issue looks unlikely in the foreseeable future.”

Looking back on my own decision to settle in Britain as a sort of sexual migrant in the 1970s, I know it would have been impossible for me then to live an open and fulfilled life as a gay man in Bombay. That situation has changed very dramatically for the better, whether in the affluent, downtown Bombay bubble or in the upwardly mobile call centres of the suburbs. The infamous Section 377, banning homosexuality, continues unchanged, and I doubt if Indian politicians will have the nerve to repeal it in my lifetime. But I'm confident that the law is becoming a dead letter. There have been no prosecutions for the past 20 years; police harassment is being reduced; and the scope for blackmail, too, is shrinking as Indian families and employers join the growing global acceptance of gay rights.