The sun is setting on the City of London Charles Clark/Business Insider LONDON — The loss of passporting rights following Brexit is one of the biggest fears in the City of London, but it will not lead to an exodus of London firms moving to other major cities in the European Union.

That is according to Sam Bowman, the executive director of one of the world's prominent think tanks, the Adam Smith Institute, who spoke at a local Conservative party conference in East Croydon that Business Insider attended on Saturday.

Bowman said: "Frankfurt is too small and Paris is a regulatory nightmare for multinational companies, and tax is too high. London has enormous benefits and the UK regulatory system for finance is very good in relative terms [to other major EU cities]."

"They are barking up the wrong tree if they think they're going to steal business."

The loss of passporting rights seems almost a certainty under Prime Minister Theresa May's "hard Brexit" plan — the loss of membership of the EU's Single Market in exchange for complete control over immigration.

If the passport is taken away, London could cease to be the most important financial centre in Europe, costing the UK thousands of jobs and billions in revenue. Around 5,500 firms registered in the UK rely on the European Union’s passporting rights for the financial services sector, and they turn over about £9 billion in revenue.

Global investment banks, with banks such as Citigroup, HSBC and JP Morgan signalling job moves to continental Europe from London. This week, Lloyd's of London said it would open a subsidiary in the heart of the EU, the day after Prime Minister Theresa May began talks to take the UK out of the 28-state trading bloc.

In March, Lloyd's of London — the city's 329-year-old insurance market — said it would open a subsidiary in the heart of the European Union to counter Brexit effects.

Bowman says though just because there will not be a mass exodus in London jobs and firms, it does not mean that processes will not become more difficult at least for the short-term nor that there will not be job losses.

"I think we have to accept that business is going to be more difficult for the City and some things are going to change. Euro-denominated trading probably will leave the city. I don't believe it is a threat [to mass relocation of companies]. There is a reason why London has done European central clearing for 60 years for a reason. It is useful to have a single centralised place for industry and work. If they were to move enmasse, where would they move?"