A Union flag flies next to the flag of the European Union in Westminster, London, Britain June 24, 2016. REUTERS/Toby Melville/File Photo

BRUSSELS (Reuters) - Most British expatriates in the European Union are worried that Brexit will limit their rights in their country of residence, according to a survey published on Wednesday.

About 83 percent of respondents said they were “very concerned” about the impact Britain’s departure from the EU could have on them, while only 3.8 percent answered “not at all”.

The survey of 5,000 Britons living in EU countries or Iceland, Norway and Switzerland was carried out by the European branch of the British Liberal Democrat party, which campaigned for Britain to stay in the EU.

Only 2 percent of respondents voted to leave the EU and 57.6 percent voted to remain in the Brexit referendum. Most of the rest were unable to vote. Over 58 percent said they did not plan to return to Britain.

Respondents said they wanted to retain the rights that came with EU membership, such as to reside in their countries of choice without permission, freedom of movement and health care.

The majority of the survey group live in France or Spain and are 55-74 years old.