Leave supporters among the British working class threw a tantrum in the Brexit referendum about immigration, and often complained about Europeans taking the jobs they themselves had refused to do, the Europe minister, Sir Alan Duncan, has said.

He also warned that labour shortages after Brexit “could cause us a lot of damage”, especially in the National Health Service.

Duncan said he believed there should have been a turnout threshold or super-majority required for the referendum, as during the UK devolution referendums conducted in 1979.

A remain supporter, Duncan was speaking at the Chicago Council of Global Affairs, where he twice complained about the way in which the leave campaign, which was fronted by his boss, Boris Johnson, sought to stir up prejudice about immigration.

He said it was “undeniably true” that young people had overwhelmingly voted against Brexit, but had to face its consequences over the next 20-30 years.

Q&A What are the two phases of the Brexit talks? Show The EU27’s negotiating guidelines for the two-year Brexit talks stipulate that they must take place in two phases: separation and “orderly withdrawal”, followed by future relationship. Only when the EU27 decide “sufficient progress” has been made on phase one can phase two begin.

Broadly, phase one is about providing “clarity and certainty” to people and businesses on Brexit’s consequences and agreeing a sum covering the commitments the UK made as an EU member: avoiding a legal vacuum, protecting citizens’ rights, solving the Irish border, and reaching a financial settlement. Phase two of the talks will then focus on agreeing the “framework” of the future trading relationship between the UK and the EU. A transition period can also be agreed as part of this second stage, but the detail of the future relationship can only be worked out once the UK has left. Britain wants to move to stage two fast, but in order to keep as much leverage as possible in talks on the future relationship aims to delay agreeing the financial settlement as long as possible. The EU27 are adamant that all phase one issues must be addressed to their satisfaction before any talk of the future relationship.

Speaking in a question-and-answer session in Chicago, Duncan displayed a sharp contrast with Johnson’s position, as he strayed from the official script that Britain is going to be a larger global player once it has left the EU, explaining instead the task ahead was to convert something that could prove a disaster into an opportunity.

Duncan hinted the UK might not seek an early free trade deal with the US at all, insisting the first trade deal the UK needed was with its closest trading partners in the European Union.

Asked why Boston in Lincolnshire voted so heavily for Brexit when it was so dependent on Polish immigrant labour for its agriculture, he replied: “That is one of the paradoxes. Lincolnshire folk said ‘ooh, they are coming to pinch our jobs’. Well, they would not do the jobs themselves anyway so it was a rather artificial anger.

“It is less worrying to our agriculture than it is to our health service and our care sector. So many of the labour problems that might follow from this could cause us quite a lot of damage.”

Discussing the validity of the Brexit referendum itself, Duncan said: “I agree with you in principle that constitutional changes should have thresholds of some sort either in turnout or percentage voting. It is a very very strong argument.”

He pointed out the devolution referendum in 1979 had a threshold since it had to be supported by at least 40% of those entitled to vote, adding: “That is why it did not go through in Wales”.

Discussing the Brexit referendum, he said: “The manner in which the campaign was fought stirred up a lot of sentiment amongst people that are not habitual voters, particularly on the issue of immigration. You could feel in the last 10 days of the campaign, traditional blue-collar urban Labour opinion going viral for leave. They were stirred up by an image of immigration, which made them angry and throw a bit of a tantrum. That was part of the chemistry that explains the result.”

He added: “The remain campaign was badly conducted in that the then government led by David Cameron and George Osborne hogged the campaign limelight and it did not look as though the remain argument put a vision of optimism. Nor did it appear a cross-party endeavour and that made elements of voting opinion treat it as a verdict on the government of the day”.

He admitted he had been surprised by the result. “I was a remainer. I thought, as most people that had political antennae, that they thought worked that it would be 52 remain to 48 leave. It turned out to be the other way round.”

Asked about the place of the UK in a free trade deal with the US, Duncan said: “I would like to think there is no queue.”

Discussing a deal with the US, he said the UK already conducted a lot of business across the Atlantic. “Whether we need a free trade deal or not will become clear. There is not one at the moment with the EU. So business is happening in the absence of a free trade agreement.

“A free trade agreement is not the answer to everything but where one can be designed and agreed in a way that is mutually advantageous to both sides that will be of benefit. We do hope to have free trade agreements across the world but the one that will matter to us enormously is the one we have with the 27 countries in the EU itself.”