The EU believes that there will have to be a “standalone” deal on Northern Ireland’s border with the Republic of Ireland, paving the way for a special status for the province after Brexit.

Senior sources at the European commission said a tailor-made solution that works for Ireland must be found.

When the third round of Brexit talks ended on Wednesday, officials made it clear that the British proposal for a customs waiver on the border was a non-starter.

It said the plan was predicated on the EU suspending the application of its laws, namely checks on goods from the UK coming into Ireland, which will remain a member state.

Dublin believes the best solution for Northern Ireland is for the UK to remain in the customs union.

Sinn Féin, which represents the nationalist community in Northern Ireland, has called for Northern Ireland to be granted special status, but this stance is opposed by the Democratic Unionist party.

It was also made clear in Brussels that talks on the Northern Ireland border have been kicked into the second phase of discussions, because the EU is not prepared to allow it to be a proxy for Britain’s negotiations on its future trading relationship with the EU.

One senior commission source described the British position paper on the matter as an attempt to make Ireland a “hostage” in the talks.

There was agreement, however, on the desire to continue with the common travel area after Brexit. It allows passport-free travel between the islands and has existed since Ireland gained independence in 1922.

It is understood that the UK also moved on its position that after Brexit, the travel area would only apply to British and Irish passport holders, paving the way for mandatory passport controls for other EU citizens travelling between the two countries.

One source said the UK had given guarantees that it would not change the operation of the common travel area “to the detriment of EU citizens”, suggesting the border would remain open.

While ostensibly there is broad political will to maintain the travel area in Europe as well as Britain and Ireland, legal experts have warned that it will not secure the rights of Irish citizens settled in Britain.

Irish citizens are deemed to have separate status to other EU citizens because of the historical and cultural ties between the two countries.

Theresa May has said Irish nationals will not have to apply for the new immigration category of “settled status” that will apply to EU citizens.

The British position on Irish citizens is that employment and social rights will flow from existing agreements under the common travel area arrangements, the 1971 Immigration Act and the 1949 Ireland Act, which ended Ireland’s status as a British dominion and stated that it would not be treated as a foreign country.

Bernard Ryan, a law professor, said there was no legal basis for that position. “There needs to be a special provision in immigration law that says Irish citizens are exempt or for them to be equated to British citizens for the purposes of immigration law, essentially treated the same as British,” said Ryan, who has acted as an expert on the issue for several parliamentary committees.

Such legislation already exists in Ireland, equating British nationals to Irish. Following the Good Friday agreement in 1998, Ireland’s 1935 Aliens Act was amended by the then taoiseach Bertie Ahern to exempt “every person who is a citizen of the United Kingdom of Great Britain and Northern Ireland” from the laws that apply to immigrants.

The British position on EU citizens in the UK after Brexit, outlined by Theresa May in June, exempts Irish people from having to apply for a new immigration status, but Ryan said this needed to be codified in law.

He said the Ireland Act states that Ireland is “not a foreign country” for the purpose of “any law in force in any part of the United Kingdom”.

Ryan said the concept of “foreign” was outdated legally, with immigration categories defined as EU or non-EU, and therefore the concept was anachronistic.

“My issue is not what is going to happen in practice, it is the lack of legal foundation [for rights],” he said.

Ryan said legal clarification was also needed for Irish passport holders living in Britain; for when they enter from other countries outside the common travel area, for example after a holiday.

His position was supported by other experts including David Phinnemore, a professor of European politics at Queen’s University Belfast. Phinnemore said the fact that the travel area was included as a discussion point in the British position paper on Northern Ireland and Ireland suggested “there are issues there”.

“People have assumed there’s a whole set of agreements which have legal force, but if we want to have the [common travel area] continue after Brexit, it would be good to have some clarity and legal underpinning,” he said.