The UK is not about to “suddenly shut the door” on low-skilled EU migrants, Brexit secretary David Davis has reiterated.

Davis said in a press conference in Riga, Latvia, that Britain wanted control over immigration but that it would only restrict free movement of people when it was in the “national interest”.

He said that it was not plausible that British citizens would immediately take jobs in the agriculture, social care and hospitality industry once the UK had left the EU and repeated comments made in Estonia on Monday that immigration restrictions would be phased in.

He said: “It will be a gradual process. That will take some time; yesterday I said it will take years.

“Don’t expect just because we’re changing who makes the decision on the policy, the door will suddenly shut: it won’t,” Bloomberg quoted Mr Davis as saying in the Estonian capital of Tallinn on Monday.

Davis said migrants had helped make the UK a strong economy. “We’re a successful economy, largely or partly at least because we have clever people, talented people come to Britain,” he said. “Even on the wider area, where we’ve got less well-paid people who have come to live and work in Britain, that will take time.”

The remarks look likely to provoke questions from some hardline leave supporters after the prime minister, Theresa May, insisted that taking back control of immigration would be a key change after Brexit.

The government says reducing net migration to below 100,000 a year remains its target. Referring to Davis’s comments, Iain Duncan Smith told the Times: “My sense is that it is going to happen quicker than that.”

May has made it clear that transitional arrangements could need to be imposed on certain parts of the economy after Britain formally withdraws from the EU.