Illustration: David Foldvari

Last August, in this newspaper, I responded as follows to the fraught question of whether or not prisoners should have the vote: "I think it's fine either way… To be honest, I don't give a damn and I don't think I would if I were in prison, either."

How horribly naive. It wasn't long ago but I do hope I've matured as a columnist. I see now how irresponsible and profligate that was for somebody in my line of work. The fact that something doesn't matter is no excuse for not arguing hotly on one side of the debate or the other. No need to waste time thinking, you can just tumble merrily on to whichever side of the fence the unknowable undulations of your buttocks seem momentarily to favour, and start enthusiastically hurling shit on to the other. A columnist refusing to form an opinion merely because a subject doesn't seem important or interesting is like a hairdresser refusing to cut the hair of a woman he doesn't find attractive. It's an offensive dereliction of professional duty.

My former indifference was based largely on the fact that I hadn't known that prisoners couldn't already vote. It would have been unreasonable, I felt, to get furious about the European court of human rights changing a state of affairs about which I was previously unaware. It would be like bursting into tears at the announcement of the death of someone I hadn't been sure was still alive. I doubt I was alone in my ignorance. Hypothetically, if on 1 April 2011 a newspaper had run a spoof story saying that some European institution was insisting British prisoners' time-honoured voting rights be revoked, I'm sure plenty of people would have been up in arms in defence of a British tradition that had never in fact existed. I was determined not to be the sort of hypocrite who could fall into a trap like that.

I've since realised that they're not hypocrites at all. In fact, I think they're right. It doesn't matter whether the European court of human rights is telling us that prisoners must have the vote, or mustn't, or that some of them can be disallowed, but most aren't allowed not to be permitted. What's wrong is that it's characterising as a human right something that obviously isn't – or not an inalienable one, at any rate. It's definitely alienable – I've seen it aliened.

In general, in a free society, people should be allowed to vote – I get that. But they should also be allowed to wander around the streets. If you're convicted of a crime serious enough to get sent to prison, you lose some of your rights but not all of them. You're not allowed to pop to the shops but you are allowed water. You can read books but you can't go on holiday. You might get searched but you shouldn't get tortured. The court has decided to lump the voting franchise in with "water and no torture" when it's clearly a lot closer to "no shopping or holidays". I'm not saying a society that lets prisoners vote is as inept as one that lets rapists speed date – but neither is Britain a repressive state for saying you can't cast your vote until you're also allowed to set foot in a normal polling station.

So why is the court seeking to interfere? Does it really think that the UK's fairly solid reputation as a free country is compromised by its refusal to change policy on this point? Surely not. It seems clear that stopping prisoners voting is perfectly consonant with a basic respect for freedom and democracy. But, like all those anti-foxhunters who sought to conceal their resentment of an upper-class sport as concern for an abundant carnivore's welfare, the court is pretending to champion prisoners' rights largely so that it can tell the British what to do.

Am I being xenophobic? I worry that I am. The court's loudest detractors are a fairly unprepossessing bunch to find myself agreeing with. But what if the court is also being xenophobic – trying to get revenge on the troublesome independent-minded UK? My assumption that all Europeans are as placid, mature and even-handed as the Swedish royal family has been rather shaken lately by all this euro chaos. I've begun to wonder whether we British don't quite have the continental monopoly on vindictiveness and incompetence that my liberal leanings had previously led me to assume. Why can't we carry on doing what we've always done when it's what our parliament has consistently voted for?

Maybe because, as Angela Patrick of the Justice campaign group puts it, "We should welcome any prisoner who wants to participate in the democratic process." Perhaps we need to be more understanding of the politics of criminals. Being vaguely left wing, I've naturally always hoped that murderers and paedophiles voted Conservative. But what if I'm wrong? What if some of them are Lib Dems?

But I'm already misrepresenting the prison population. It's not just free marketeer kiddie-fiddlers and One Nation stranglers: there are also handsome jewel thieves looking for an outlet for their liberal consciences; disgraced peers like Lord Archer, who aren't allowed to vote even when they're not in prison; and of course that lookalike of the Queen who has to wear an iron mask.

So perhaps we should let prisoners vote but withhold the secret ballot. Then we'd know which party was benefiting from the reform – I'd like to see them claim they were tough on law and order then. Alternatively, give prisoners any votes not used by the rest of the population. If you're genuinely too busy, confused or apathetic to vote, you can console yourself with the thought that an armed robber with a lot of time on his hands will get your say instead.

But if I ever sympathised with ambitions to influence the electoral process from those we've decided aren't safe to have walking among us, some bossy European lawyers saying we're infringing such people's rights has cured me of it. Miscarriages of justice aside, you don't get sent to a British prison without infringing a right or two yourself. So I say, sod them.

I don't believe that David Cameron would genuinely be "sick in the stomach" if prisoners were granted the vote. It wouldn't make much difference. But I reckon he's thrilled to be able to appeal to little Englanders by defying the court. Well, that doesn't make him wrong. And, in the spirit of the jubilee, I'm happy to sit back and wave my flag.

• The headline on this piece was changed on 27 May 2012 in order to reflect the content more accurately. The former headline was 'When wrongdoers forfeit human rights'.