Northern Ireland could veto its exit from the European Union, a lawyer for anti-Brexit campaigners from the region has told the high court in Belfast.

A senior barrister argued that Brexit could not be imposed on Northern Ireland and that the Good Friday agreement, ratified by a referendum in 1998, meant the province had some control over such constitutional changes.

Leaving the EU would undermine gains made during the peace process, he told the court on the first day of a legal challenge by political leaders and human rights campaigners against the EU referendum result.

Ronan Lavery QC said on Tuesday: “Sovereignty over constitutional affairs has been ceded [by the UK]. It is not the relationship, as it might once have been, between a dominant partner and a submissive partner.

Brexit secretary: no return to 'hard' border in Ireland Read more

“The people of Northern Ireland have control over constitutional change, it cannot be imposed upon the people of Northern Ireland. If that means that Northern Ireland could exercise a veto over withdrawal then I am [asserting] that is what Britain signed up to when it signed the Good Friday agreement.”

The court challenge comes as an increasingly anxious Irish government prepares to apply to the EU for a special status that would prevent a hard border being re-established between the UK and the Republic of Ireland.

Charles Flanagan, the Irish foreign minister, said the UK and Irish governments would seek special legal status for Ireland, even though an open border between Ireland and the UK could theoretically allow EU citizens to travel to the UK.

The Irish government will announce on Tuesday that it is setting up a civic dialogue to be held in Dublin on 2 November. It will involve political parties, business groups and non-governmental organisations from Northern Ireland and the Republic.

One aim will be to demonstrate to Brussels that Ireland deserves special legal status. Flanagan also stressed the need for Brexit negotiations to take account of the Good Friday agreement, under which all citizens in the north are entitled to an Irish and therefore an EU passport.

Martin McGuinness, the deputy first minister and Sinn Féin leader, spoke to David Davis, the UK Brexit secretary, on Sunday to discuss government plans after it emerged that the cabinet is leaning towards leaving the EU single market and the customs union.

McGuinness said the UK was facing a “head-on collision” with the EU over Brexit, and suggested Northern Ireland was likely to be “collateral damage”.

Sinn Féin is one of the parties to the legal challenge to Brexit, alongside the Social Democratic and Labour party, the Greens, a former head of Northern Ireland’s equality commission, the former NI justice minister David Ford and Raymond McCord, a campaigner for victims of paramilitary violence.

About 56% of Northern Irish voters backed remain on 23 June, but some unionist-dominated areas supported leave. The DUP insists the overall UK result to leave has to be respected.

The Irish government has been trying to talk down the prospect of passport or border checks for those visiting Northern Ireland, but the prospect of the UK leaving the EU customs union without a new trade agreement with the bloc has created anxiety in Ireland.

If the UK leaves the customs union, the EU could demand a hard border in Ireland to prevent goods flowing in and out of the EU from Northern Ireland without paying required tariffs or facing checks on rules of origin.

If the UK reverted to World Trade Organisation tariffs, this could also spell serious trouble for agriculture in the north as tariffs are especially high for this sector.

A “hard Brexit”, including a return to UK border controls, also makes it more likely that the common travel area between the Republic and the north will be abandoned, hitting the Northern Ireland economy and one of the main tenets of the peace process.

Speaking on RTÉ’s Morning Ireland, McGuinness said: “We have all been concerned for some time at the direction the government is going to take. It is very disturbing.” He predicted that the UK government’s demands in Europe would not be met, making it less likely that Brussels will let the open border remain.

The tone of speeches by leading Brexiters has also caused unease in Dublin, no matter how much the Northern Ireland secretary, James Brokenshire, has insisted a deal can still be struck on maintaining a common travel area.

Scotland, Wales and N Ireland could demand vote on Brexit terms Read more

Flanagan has previously said he could not see how a hard border could be enforceable. He said: “Ultimately, the matter of the border is a decision that won’t be determined by the UK and Irish governments, irrespective of how we feel about the issue, but this will be a matter for the remaining 27 EU countries, one of which of course will be Ireland.”

Nick Clegg, the former deputy prime minister and a former European trade negotiator, has warned of the implications of leaving the customs union for the border. “Leaving will require the introduction of new controls [a hard border] between Northern Ireland and the Republic. There is no way round that.” That was also the position of the UK government during the referendum campaign.

If there were no EU-UK agreement on free trade in goods, there would be some British taxes on imports from Ireland, and vice versa.

UK withdrawal from the customs union would make goods exported across the border subject to various forms of customs controls with duties determined according to complex rules of origin on each individual piece of a product assembled in Europe.

A third of Northern Ireland’s exports in 2015 (£2.1bn out of £6.3bn) were to the Republic (trade with Great Britain is not considered an export), while only 1.6% of the Republic’s exports of €111bn (£97bn) were to Northern Ireland.