Being a City Brexiteer might be assumed to be a lonely life. After all, aren’t we told repeatedly that “the City backs Remain” as though this were, like Newton’s laws, an immutable fact of physics?

In fact, life isn’t lonely. Daily, people from the City sidle up to me much like Cold War dissidents. Carefully pausing to look left and right like characters in a John Le Carre novel, they admit sotto voce they have “doubts” not only about the EU but about the institutional regime within the City that continues to back it.

I am in the lucky position of being able to speak without being dragged off to financial Siberia. So on behalf of those who can’t, here’s what those doubts sound like.

The idea that the City is exposed by leaving the EU is one based on the notion that somehow the EU has the interests of London at heart. It doesn’t.

Read more: The City will prosper in or out of the European Union

Paris and Frankfurt, cities at the heart of the Franco-German axis which so dominates European decision-making, do enviously eye the Square Mile as undoubtedly the most successful financial centre in Europe, arguably the world.

But the City has achieved this position through being a huge global market, as capable of dealing simultaneously with New York, Chicago, Hong Kong, Tokyo, Beijing and Zurich as it is with its EU counterparts. In other words, it is open where the EU is not. The EU looks within its own borders to what the historian Niall Ferguson calls an im-pire. That confines it and restricts its ability to match London’s success.

The City also works in harness with a critical mass of support services built up over hundreds of years. It is a simple nonsense to suggest that our European rivals will be in a position to develop a similar environment in short order while adding the advantages of time zone and the English language.

Read more: Osborne’s Brexit Budget: Blackmail based on fundamentally flawed economics

Paradoxically, while Europe – to use that all-encompassing term – regards London as something to envy, it also regards it as inimical, an Anglo-Saxon construct whose ideals and purpose it inherently distrusts.

For the City’s love of markets, read Europe’s love of dirigisme. For the City’s love of open trade and innovation, read Europe’s love of regulation and protectionism.

It was the Anglo-Dutch and later American love of sea-borne mercantile trade, the raising of private rather than state finance, that created the modern financial world and the sense of independent endeavour and democratic accountability which have defined those countries. It is no wonder then that the City is a bête noire for Europe, for whom internal taxation was always an easier financial alternative than risk-based endeavour.

Read more: Why Britain really joined the EU and should now vote to leave it

We must also take into account the “ever closer union” which is the raison d’être of the EU and which manifests itself increasingly through its institutions. As the Continent and those institutions become increasingly bound into the remorseless and disastrous logic of the Eurozone and a European Central Bank, the advantage moves ever further to London as an independent entity capable of acting accordingly and as markets, not politics, demand. For how long can that be tolerated by a union riven with its own economic woes and apparently incapable of curing them?

And so we are left with two options. The first being that we remain and eventually surrender the City of London to the dead-handed and self-interested embrace of its institutional and national rivals. Or, as the Remainers would have it, we stay for fear of a dreadful revenge which will banish City influence and operations from European Union territories.

Read more: Losing the EU financial services “passport” would be a disaster for the UK

Even if the latter were achievable or even desirable, are those really the terms on which we should embrace Europe? Not as a benign and helpful partner but as a glowering and threatening master, stick raised over a cowering London, its own interest always in mind and mistrust at its heart.

As Bernard Connolly, former European Commission economist and author of The Rotten Heart of Europe, has written: “there are three things the EU does not like about Anglo-Saxon financial markets, they are Anglo-Saxon, they are financial and they are markets!”

Those, to my mind, are no choices at all and, as an increasing number of my City colleagues secretly confirm, for few others either.