He may once have been a bouncer, but the pope no longer looks like the kind of guy who can handle himself in a fight. Most opponents would fancy their chances of downing him with a knee to the pontifical plexus. Be in no doubt that, whatever his physical condition, the pope is still up for it. He will bellow the Latin equivalent of “come on then, if you think you’re hard enough” – “agite tentateque si fortiores vos putatis”, our classical correspondent tells me – and it will all kick off.

After saying that he believed in freedom of speech, as everyone does, Pope Francis added the inevitable “but,” as so many do. If a friend “says a curse word against my mother, he can expect a punch,” he told reporters. Insults to religion invite the same violent response as insults to his manly pride and family honour. “It’s normal. You cannot provoke. You cannot insult the faith of others. You cannot make fun of the faith of others.” Or to put it another way, the Parisian satirists had it coming.

Why supposed liberals continue to believe Pope Francis is one of them is inexplicable to me. But then supposed liberals are becoming ever harder to understand.

After the Paris attacks, the novelist Will Self claimed moral equivalence. Those who say “freedom of speech is an absolute right” – no one does, incidentally – have “a religious point of view”. Mehdi Hasan, political director of the Huffington Post, agreed that freedom was fanaticism. He condemned “the hypocrisy of free-speech fundamentalists” and cited a thought experiment of an Oxford philosopher called Brian Klug. If an Islamist had joined the free speech rallies in Paris and applauded the murderers, Klug mused on the basis of no evidence whatsoever, he “would have been lucky to get away with his life”. The demonstrators had their limits on free expression too. “They just didn’t know it.”

Contemptuous laughter is the easy response. Dr Klug can play with his thought games for as long as he likes, but as he must know, no one has been murdered for criticising free speech. The supposed “free-speech fundamentalists” are not the killing type. Clerics and intellectuals run no risks when they criticise them. They don’t have to worry about the safety of their families and colleagues, as the critics of actual fundamentalists do. The example of Self makes my point for me. When he denounced the freethinking George Orwell, I shrugged. Self was a conventional political thinker, who hid behind an obscure style, while Orwell expressed his dislike of the conventional left of his day in the plainest possible English.

I could not expect Self to admire Orwell. In our most fevered dreams, however, neither I nor anyone one else thought that Self was putting himself at risk; that he would have to go on the run from “free-speech fundamentalists” who would make him pay for his “offence” in blood or force him to spend the rest of his life looking over his shoulder

The moral equivalence on offer here is immoral and bogus. But it is popular and potentially powerful. George Osborne says, if the Conservatives are re-elected, they will not only allow the police to arrest those who incite violence against others, as the police always should. The criminal justice system will also ban those who “spread hate” based on gender, disability, religion or sexual orientation, “but do not break the law”. When the papacy, the PC and the Tories go into alliance you need more than contemptuous laughter to repel them.

So let me concede acres of ground that are not worth defending. Yes, yes and obviously, there is no stupider cliche than sticks and stones will break my bones but words will never hurt me. Most of us can remember words that cut deeper than wounds. The parent who says you are a failure. The lover who admits betrayal. The women who laugh at you. The men who humiliate you. The employers who dismiss your dearest ambitions with a snort. They leave scars that may never heal.

Likewise, religious belief can be so much a part of your identity that an assault on it is an assault on everything that makes you who you are. If Observer readers find religious offence hard to understand, ask: have you ever found criticism of the left from the right or the sight of a confident Conservative leader so unbearable you were physically repelled, as I was by the sight of Margaret Thatcher?

But understanding is not excusing in either the personal or the political. Think before you go along with the pope’s argument that violence is the “normal” response to insults to family honour. Once the law accepted it was. A husband could beat a wife, who failed to stroke his ego and confirm his superiority and the police would dismiss the case as a “domestic”. A man could kill a woman who had betrayed his honour and the courts would dismiss it as a crime of passion.

Everyone tells white lies and spares the feelings of others. Absolute freedom of speech is impossible to imagine because no one could blurt out every thought that came into his or her head and expect to live among their fellow social mammals.

But we do not now say that the woman who refused to lie to please her husband deserved to die. Nor do we call on the law to prosecute her for demeaning his macho pride with “hate speech”. Move from the personal, and the attempt to say that those who refuse to tell white lies about religion are asking for whatever punishment the faithful gives them is sinister.

Do I need to remind you that insulting the gods, the pope or the synagogue were the charges the faithful levelled against Socrates, Galileo and Spinoza? Or that insulting religion is everywhere the favourite charge of fanatics?

In Pakistan, Saudi Arabia and Iran, where fanaticism has real force, they always use it against religious minorities and the secular and, indeed, against women demanding equal rights. Religion is a form of power. We do not have absolute freedom of speech, but we must protect our limited freedom to criticise power.

Give up on that, dismiss it as rudeness or say that supporters of hard-won freedom are just as “fundamentalist” as their religious opponents and you abandon every inch of ground that has to be defended without so much as a fight.