The vote on Britain’s membership of the European Union may still be months away, but already the scare stories have started. One that is particularly revealing is the question about who should be allowed to vote in the in/out referendum. The Mail on Sunday reported Tory concerns that one million Europeans who are not UK passport holders could be allowed to cast their vote. Conservative MP Philip Davies told the paper that there was “massive concern that the referendum could be rigged to deliver a desired outcome. But it would be unjustifiable if EU nationals were allowed to take part in this vote”.

Such is the concern that Eurosceptic Tories have challenged David Lidington, the minister for Europe, to make sure these European foreigners are not allowed to vote. At present, EU citizens living in this country can vote in the local and European elections but not in the general election. Tory rebels want the referendum to be run under general election rules. What could be fairer?

What these MPs neglect to mention is that even now you do not necessarily need to be a UK passport holder to vote in a general election. Indeed, for decades the UK has allowed citizens from other countries the right to select members of parliament, a right that even extends to citizens of three EU countries. They are part of a much larger group of 72 countries that includes all Commonwealth territories, British overseas territories and British crown dependencies. Fiji and Zimbabwe may be suspended from the Commonwealth but their citizens resident here have not lost their right to vote in UK elections. The three special EU countries are Ireland, Cyprus and Malta. They enjoy this privilege because while they may now be part of the EU, they once had an older allegiance to a much greater union: the British empire. The sun set on the empire long ago, but its legacy lives on.

What makes all this fascinating is that while Eurosceptics are happy to raise all sorts of scare stories about the EU, these other voters are an issue they are reluctant to discuss. Indeed, as far as the UK electoral franchise is concerned, this is now the great elephant in the room, as I was made well aware during the recent election. At one husting, I had the chance to raise this issue with three panellists from the main parties: Michael Gove for the Tories, Ivan Lewis for Labour and Baroness Kramer for the Lib Dems. Lewis disapproved of my even raising the issue. Baroness Kramer, who did not seem to know that non-citizens could vote, justified it on the grounds that this was a wonderful example of British eccentricity. Gove just said that he did not want to see any change in the franchise.

More fascinating was how Ukip reacted. Some weeks before the election, at a British Future event, Douglas Carswell, now the only Ukip member of parliament, made a very reasoned speech to show that Ukip was not an anti-immigrant party. But when I raised this issue, he made it clear that this was not a question Ukip would touch, remarking that the British system was so complex that to lift the carpet would mean all sorts of things would crawl out. How strange to hear this from a party whose leader, Nigel Farage, makes so much of the fact that he is prepared to go where no other politician dares.

As far as the UK electoral franchise is concerned, this is now the great elephant in the room

Indeed, if Farage were the revolutionary he claimed to be, he could turn to the country whose points-based immigration system he is always advising us to copy: Australia. Before 1984, British citizens living in Australia were allowed to vote in its elections. After that you had to be an Australian citizen to do so, although British citizens who were entitled to vote before 1984 retained their right.

I am not saying we should change our electoral system. I am merely intrigued that the issue is not discussed. It is all very well to say this is part of Britain’s great imperial legacy. But in recent years the question of non-doms, another imperial legacy, has been widely debated with Labour proposing to abolish it. So why not this? In refusing to even have a debate, we are in danger of allowing Ukip and other Eurosceptics to raise scare stories of how foreigners are usurping British rights when, in fact, reflecting this country’s complex history, it already extends all sorts of rights and privileges to foreigners. It seems to depend on who the foreigners are.