It was on the return flight from Rio aboard the papal plane known to Vatican-watchers as Shepherd One that Pope Francis reopened the most urgent issue of his reign. Relaxing in the glow of adulation from a tour of Brazil that had culminated in the celebration of mass with three million pilgrims on Copacabana beach, the pontiff wandered to the back of the papal plane, where he spoke freely to an astonished press corps about the vexed question of the Vatican and homosexuality.

He first dispensed with his predecessors' distaste for the very word "gay". "Who am I to judge," he said, "if someone is gay and he searches for the Lord with goodwill?" Gay people should not be marginalised from society, he said, before tackling head-on the rumoured, much-discussed existence of a powerful "gay lobby" in the Vatican. If such a thing existed, it was not the most important form of corruption: "The problem is not having this orientation. We must be brothers. The problem is lobbying by this orientation, or lobbies of greedy people, political lobbies, Masonic lobbies, so many lobbies. This is the worse problem."

To understand the significance of homosexuality to the Vatican, one needs to know that a large minority of Catholic priests are thought to be gay and these priests know all too well the catechism's teaching that "homosexual acts are intrinsically disordered". Add this to the Vatican tradition of discrediting an enemy by accusing him of being gay, and the result is a sizeable number of closeted men in positions of authority with deep and potentially damaging secrets. By making same-sex acts a sin, the church moved homosexuality from being a simple matter of sexual orientation into the realm of conspiracy and politics.

The problem is compounded by a document drawn up by Pope Benedict XVI when he was Cardinal Ratzinger, which appears to mandate a life in the closet for the gay priest. Although it condemns homosexuality as an attraction towards "an intrinsic moral evil", men who have managed three years of celibacy are assumed to have no "deep-rooted" homosexual orientation. In such an atmosphere, it's no wonder that the phrase "gay lobby" has become so powerful and that almost everyone is suspected of membership.

A recent and high-profile target of suspicion was the pope's new personal representative at the troubled Vatican Bank, Monsignor Battista Ricca, who was accused of living openly with his male lover when he was a Vatican diplomat in Uruguay at the turn of the century. The pope, on the flight from Rio, said the allegations had been found to be groundless and anyway only concerned a sin, which God might forgive, and not a crime. This was not only an assertion of his authority over the backbiters within the Vatican but also marked an attempt to deal with gay people realistically, as people.

These two strategies are closely linked in his struggle to reform the church, and tame the Vatican's bureaucracy, known as the Curia.

The English theologian James Alison, himself openly gay, reckons more than 40% of the Catholic clergy today are gay, but that very few are comfortable or honest about it. Other experienced observers concur with this estimate, though few inside the church will speak on the record.

"The notion of a gay lobby is complicated," said Alison. "There are so many uses of the term." Inside the Vatican, the term typically refers to all people outside who claim being gay is normal. The second use, said Alison, was as shorthand for delusional thinking, such as "When people say things like: 'This so-called scientific teaching is merely the result of a powerful gay lobby.'" This shows, said Alison, that the biggest and most successful gay lobby in the Vatican is the closeted one.

The veteran Vatican correspondent John L Allen of the National Catholic Reporter believes the Italianised term "lobby gay" has a different nuance to the English phrase. "When you say 'gay lobby' to the typical English-speaker, what they're going to think of is … an interest group advancing an agenda.

"That really is not what Italians mean by the term 'lobby gay'. What they mean is this clandestine network of people in the Vatican who have skeletons in their closets who are looking out for one another, and as far as Italians are concerned those skeletons don't even have anything to do with sex in some cases.

"If the question is 'Are there gays in the Vatican?', yes, of course there are. But if the question is 'Is there some kind of organised network of gays in the Vatican who are protecting one other and advancing their own interests', all I can tell you is that in 15 years of covering the place I've never seen any particular evidence of that."

According to Alison, even the notion of a closeted gay lobby turns out to be complicated. "It's a honeycomb of closets. Not everyone knows everybody else. Everybody knows somebody who knows someone else. So there is a …game of blackmail going on.

"The people with the strongest motivation to keep the current system are those people who – maybe for the best of motives – opted to 'sacrifice' that part of themselves for what they thought was the glory of God. They found themselves constantly having to re-enact that sacrifice for other people, as though the annihilation of who one is was actually what Our Lord meant. Their sacrifice has been not only in vain, but has been a monumental act of self-destruction. This destruction is independent of whether the person has or has not got partners."

Another perspective is supplied by the German theologian David Berger, who for 20 years was part of the closeted traditionalist scene in Germany (Alison notes wryly that the smallest possible Catholic society would be a gathering of straight traditionalist priests). Berger was denounced and sacked from his teaching job when he came out in 2010 and now edits a gay magazine. "In Rome I experienced that these [gay] networks exist but they're not about power grabbing. Nepotism exists in the Vatican anyway, based on friendships. The main aim of these circles is simply to gain access to sex in an uncomplicated way. There is also a lot of paid sex but much unpaid sex as well. There's no gay conspiracy in the Vatican."

Berger has large files of letters from priests, largely from German-speaking countries, in which the writers have told him graphic and often tortured stories about their experiences of being gay and in the priesthood. One priest was so distraught by the attempts to blackmail him as well as his sense of guilt that he covered himself in petrol in front of the other priest with whom he was in a relationship and set himself alight. He died, whilst the surviving priest lives with a huge psychological burden.

Some observers identify an older generation of gay priests as being less conflicted. One highly placed English Catholic said: "I have known quite a few gay clergy of that generation, who accepted the church's teaching on homosexuality without having to fight any self-hatred. They did not see their homosexuality as being at the very core of their identity. They happened to be gay. But what is at the centre of their being is that they can love, and it just happens that some of their most profound experiences of love were of other men."

This adjustment to a celibate life – where sex is entirely unimportant and uninteresting when compared with love that seems to need no sexual expression – does certainly still exist, and was acknowledged by everyone the Guardian talked to. But it is hard in the modern age. "Rome is one of the last places on earth where 'don't ask don't tell' actually means that," said Alison. "It is a traditional monosexual culture, in the same way that the British army would have been in 1890. Women and indeed sex are simply irrelevant. It didn't matter what you did so long as you weren't caught and caused no scandal.

"It's a remarkable cultural survival of a pre-modern world. But the people inhabiting it are modern people. So you get cognitive dissonance. There are various ways of surviving. You can live a double life, with all the pain that will lead to. You can choose to shut down your emotional life and become career-minded."

It will be a test of his reign whether Pope Francis can negotiate this politically charged issue, one consequence of which is that accusations of homosexuality remain one of the commonest and most effective forms of attack.

Sometimes this is entirely deranged. A traditionalist blogger has denounced the archbishop of Vienna, Cardinal Christoph Schönborn, as "a homosexual" because he had said something sympathetic about his own gay clergy.

Similarly, a letter denouncing by name a number of prominent German and Austrian Catholic clergy, inside and outside the Vatican, was circulated to all German-speaking bishops last year and to the media including the Guardian. The men are widely regarded to be moving in gay networks. Some are accused of complicity in the manoeuvrings around the Vatileaks scandal last year, when Pope Benedict's butler was convicted of passing on detailed and confidential information to journalists. It seems likely that this letter formed part of a dossier presented to Pope Benedict in February by three cardinals charged with investigating the Vatileaks affair.

As his remarks on the plane imply, an opening towards honesty in Catholic attitudes to homosexuality and gay people must form a part of what Francis was elected by the cardinals to do: reform the Vatican. It is a huge task.