LONDON — Four major U.S. banks — Goldman Sachs, JP Morgan, Morgan Stanley and Citigroup — have plotted out their exit plans after Brexit and already have one foot out the door.

"Citi is planning to move people to Dublin, while the other three are going to Frankfurt," one person familiar with the matter said. "At this stage the discussion isn't about moving thousands of people, just hundreds; numbers will grow if single market access doesn't happen." The source said the first to be moved will likely be the banks' derivatives trading teams.

Bankers say they love London and don't want to move.

"London is an optimal choice for us. Any other city would be sub-optimal," a banker at one major U.S. firm said. However, those familiar with the ongoing talks between the City of London, the British government and EU regulators insist things will change sooner rather than later.

"Some jobs will go, others will move and others will be created in Frankfurt or Dublin instead of London — and it's likely to happen between the first and second quarter of next year," the source said.

Given the skin financial institutions have in the game and the persisting Brexit uncertainty, none of these firms wants to be portrayed as rushing out the door — especially if they can still lobby the government to get a good enough deal for them to stay.

“Nobody wants to make a bad decision by moving quickly with the amount of uncertainty around. We will be ready to make a permanent decision once we have a better idea of the direction of travel [a hard or a soft Brexit],” Daniel Pinto, JP Morgan's head of corporate and investment banking, said in an interview.

Bank of England Governor Mark Carney said Tuesday that lenders could start relocating 18 months before the actual British departure from the EU. That would mean September 2017, if Prime Minister Theresa May's plans proceed on track and Article 50 is triggered early next year.

But according to people close to the banks' Brexit discussions, relocations could start as early as next March, or June at the latest. Citi did not respond to requests for comment while Goldman Sachs and Morgan Stanley declined to elaborate on their plans.

A spokesperson for JP Morgan said no final decision had been taken, but in the first quarter of 2017 the bank will start "renovating and improving their technology and infrastructure across offices in Europe." The U.S. banking giant, which was ranked the world's top investment bank by fees in 2015, has offices in 15 EU countries apart from the U.K.

Likewise, a Goldman Sachs representative said they are still working through the implications of the June 23 vote. "There remain numerous uncertainties as to what the Brexit negotiations will yield in terms of an operating framework for the banking industry. As a result we have not taken any decisions as to what our eventual response will be,” the spokesperson said.

Sources say Goldman is trying to move its balance sheet from the U.K. to Germany before moving its staff, but supervisors are reluctant to take on the risk of operations carried out elsewhere. The ECB declined to comment. Regardless, Frankfurt would be a logical alternative destination for Goldman given it is already its second EU hub, employing 300 people. Goldman has 5,000 employees in London.

For Citigroup, the best option is Dublin where it already employs 2,500 people and has the needed regulatory approvals to carry out business, a source said. The group has denied the claim, according to an Irish news report.

Other EU financial hubs, including Frankfurt, Paris and Milan, are courting the banking behemoths.

“Several elements play in Frankfurt’s favor, and not only the proximity to the European Central Bank and its banking authority ... its central position in Europe, an effective infrastructure, a stable political and economic environment, an efficient administration, regulatory stability, an excellent education system and research facilities as well as a good standard of living,” a spokesperson for the German finance ministry said.

On Monday, Thomas Steffen, a senior official at the ministry, said that after the EU referendum they had registered an increasing number of queries from U.K.-based banks. Milan Mayor Giuseppe Sala also said that local authorities are working with the government to create incentives and improve the business environment for banks to set up their headquarters in Italy.

One senior U.S. banker pointed out that both Frankfurt and Milan have significant limitations. "Aside from office space challenges, how do you move entire families, perhaps where the other spouse doesn't work in banking and doesn't speak German?" the banker said. Italy's slow legal system, high taxation and complex labor market doesn't even put the country on many firms' radars.

Hortense Goulard contributed to this article.