French Finance Minister Emmanuel Macron warned the U.K. that a possible exit from the European Union (EU) could endanger an agreement that allows British border police to operate in Calais in northern France. Macron made the comments while speaking to the Financial Times before security and migration talks between British Prime Minister David Cameron and French President François Hollande began Thursday in Amiens, France.

A referendum is set to be held on June 23 that will have British citizens vote on whether they want to stay with the EU or split form the bloc. Marcon told FT if the votes favor an exit, known as Brexit, it will bring an end to the Le Touquet agreement between France and the U.K., which allows British authorities to conduct border controls on the French side of the English Channel, the Guardian reported. About 4,000 refugees are currently gathered in Calais in hopes to cross over to the U.K. soon. The refugee crisis is also expected to be one of the key topics discussed between Hollande and Cameron Thursday.

“The day this relationship unravels, migrants will no longer be in Calais,” Macron told FT, according to France 24, adding that “there would be nothing to stop thousands of people crossing the Channel overnight.”

Macron also said, according to BBC, that a country leaving the single market would “not be able to secure the same terms,” adding that the EU’s “collective energy would be spent on unwinding existing links, not re-creating new ones."

“The EU as a whole would be weakened as a military, diplomatic and economic powerhouse if the U.K. left,” he said, adding: “The EU has no choice but to become a true military and diplomatic power, something it has been always reluctant to be, mostly by lack of ambition.”

Cameron, who backs staying with the EU, had warned earlier that a Brexit would result in refugees from the “Jungle” camp of Calais moving into Kent. Xavier Bertrand, the recently re-elected president of the Nord-Pas-de-Calais-Picardie region, said, according to the Guardian: “If Britain leaves Europe, right away the border will leave Calais and go to Dover. We will not continue to guard the border for Britain if it’s no longer in the European Union.”

While Cameron has also suggested that Le Touquet agreement could be torn up in the event of a Brexit, critics have dismissed such fears, saying that the agreement is unrelated to the membership to the bloc. London’s Mayor Boris Johnson, who supports an exit from the EU, urged voters to “ignore scaremongers,” according to the BBC.

Big businesses have also sounded caution over a potential Brexit. Rolls Royce CEO Warren East wrote a note to workers saying, according to the Telegraph, that a Brexit would lead to a rise in costs and prices, in turn affecting the “employment base.” The Society of Motor Manufacturers and Traders, which is based out of the U.K. also said, according to the Telegraph, that 59 percent of its members believe that Brexit will have a negative impact on their business in the long term while 18 percent were unsure.

Tony Walker, deputy managing director at Toyota Motor Manufacturing U.K., said, according to the Telegraph: “Our U.K. operations are fully integrated within our European business — exporting nearly 90 per cent of all U.K.-built vehicles,” adding: “After considered review, we believe that continued membership of the European Union is best for our business and for our competitiveness in the longer term.”

Rory Harvey, managing director and chairman of Vauxhall, said, according to Telegraph: “We are part of a fully integrated European company where we benefit from the free movement of goods and people, and we believe not to be part of the EU would be undesirable for our business and the sector as a whole.”