The government’s post-Brexit immigration system would cost employers more than £1bn in administration costs over five years, harm the higher education sector by putting off EU students and risk a new Windrush scandal, according to an analysis of the plans.

The report by Global Future, a thinktank that supports more open immigration, says the NHS would also face almost £120m a year in additional costs to recruit overseas staff under the system laid out in the immigration white paper.

The long-delayed plan imposes the same £30,000-a-year minimum salary threshold for EU citizens looking to work in the UK that already applies to those outside the bloc, as well as various visa fees and other charges.

Using Home Office modelling for the number of people expected to apply for visas of different lengths, and a calculated average cost to employers of almost £12,500 per EU national, the study comes to a five-year total of £1.14bn, with £337m faced by the public sector.

The study separately estimates the cost to the NHS by using figures for the likely number of EU workers needed under the service’s long-term plan and fees and other charges they would incur. This would, it said, total £118m a year.

Quick Guide Why extend the Brexit transition period? Show Will the proposal solve anything? The mooted extension to the transition period is a new idea being put forward by the EU to help Theresa May square the circle created by the written agreement last December and the draft withdrawal agreement in March.

That committed the UK and the EU to ensuring there was no divergence between Northern Ireland and the Republic of Ireland.

But it also, after an intervention by the Democratic Unionist party, committed the UK (not the EU) not to have any trading differences between Northern Ireland and Great Britain.

The problem is that these are two irreconcilable agreements. They also impinge on the legally binding Good Friday agreement, which brought peace to Northern Ireland and in some senses pooled sovereignty of Northern Ireland giving people a birthright to be Irish or British or both.

If the UK leaves the EU along with the customs union and the single market then the border in Ireland becomes the only land border between the UK and the EU forcing customs, tax and regulatory controls. The backstop is one of three options agreed by the EU and the UK in December and would only come into play if option A (overall agreement) or option B (a tailor-made solution) cannot be agreed by the end of transition. The Irish have likened it to an insurance policy. The new EU idea is to extend the transition period to allow time to get to option A or B.

But an extension is problematic for Brexiters and leave voters, who want the UK to get out of the EU as soon as possible.

The Irish and the EU will also still need the backstop in the withdrawal agreement, which must be signed before the business of the trade deal can get under way. Otherwise it is a no-deal Brexit.

Extending the transition into 2021 would mean another year of paying into the EU budget. Britain would have to negotiate this but it has been estimated at anywhere between £10bn and £17bn. Staying in the EU for another year would also mean continued freedom of movement and being under the European court of justice, which Brexiters would oppose.

It also says the £30,000 threshold could eventually leave as many as 100,000 jobs social care and nursing jobs unfilled and cause difficulty in many other sectors.

Students from the EU who decide to base themselves in the UK would incur visa and health surcharge costs of almost £1,300 over a three-year degree, the thinktank says, making a total of £80m a year across the sector as a whole.

The report also says that in requiring people based in the UK under freedom of movement to suddenly prove their status, the scheme risks mirroring the Windrush scandal, in which UK nationals of Caribbean origin who had been in the UK for decades were denied rights or even deported because of a lack of documentation.

This could happen again through a lack of awareness, Home Office errors and technicalities such as people having spent time out of the UK.

Fergus Peace, a researcher at Global Future, who wrote the report, said the white paper “represents an unambiguous shift towards a more complex and burdensome immigration system that will damage our country’s prospects”.

“The government presents ending free movement as the great prize at the centre of its Brexit strategy, but ultimately this white paper is a plan to close ourselves off from the world. That’s not something to celebrate. It’s a dreadful mistake.,” he said.

A Home Office spokesman said the report’s estimated costs “are entirely speculative”.

“To attract the talented workers we need we are bringing forward a new skills-based immigration system, which is designed to drive up wages and productivity across the UK,” he said.

“This will include supporting businesses by stripping out bureaucracy through streamlining and digitising the system and committing to process the vast majority of work visas within three weeks.”

The scheme to give EU nationals settled status in the UK was “simple and straightforward”, he added.