A Conservative minister has accused remain campaigners of attempting to silence the debate on immigration, warning that failure to address people’s concerns would simply fuel the emergence of a “nasty politics” on the far-right.

In an interview with the Guardian, Dominic Raab hit out at Tory colleagues including Lord Michael Heseltine and Sir John Major, who he said were starting to sound like “the elite saying we can’t talk about immigration”.

In response to Major’s claim that Tories campaigning for Brexit were morphing into Ukip, Raab said: “I’m proud in this country we’ve virtually eliminated the BNP. The risk if we don’t take people’s concerns seriously is not that we start to look like Ukip, the risk is that the really nasty politics emerges.”

The intervention intensifies the clash between the remain and out camps on the issue of immigration, as the referendum campaign enters its final month. On Sunday David Cameron accused his own defence minister Penny Mordaunt of misleading the public about the prospect of Turkey joining the EU.

The prime minister said Mordaunt was “absolutely wrong” to suggest that Britain had no veto over the accession of new countries. “Let me be clear, Britain and every other country in the EU has a veto on another country joining. That is a fact,” Cameron said in an interview on ITV’s Peston on Sunday.

Earlier during a BBC interview, Andrew Marr asked Mordaunt whether Britain had a veto, and she replied: “No it doesn’t.” She then argued: “I do not think that the EU is going to keep Turkey out. I think it is going to join. I think the migrant crisis is pushing it more that way.”

Mordaunt told the Guardian that her intention had been to highlight the prime minister’s own support for Turkey joining the EU, and to point out that the British public would not be given a referendum if the decision was taken in the future.

Campaigners on both sides of the arguments jumped on the interviews to highlight past records of their opponents. Brexiters pointed to Cameron’s statement in 2010 that he wanted to “pave the road from Ankara to Brussels”, while Remainers pointed to Boris Johnson’s claim that Turkey’s accession was not likely to happen.

Raab said he was determined not to enter into “tit for tat”. On immigration, he said he was the son of a refugee and married to a woman from Latin America. “I’m not a little Englander, but I want the British people to have confidence in our immigration system.”

He said that wasn’t possible while the UK was in the EU because there was no control of numbers, no way to make sure immigrants were self-sufficient and no ability to remove people who had committed serious offences. He talked about the emergence of the far right in France, Austria and the Netherlands, and said the same could happen in the UK.

Vote Leave embroiled in race row over Turkey security threat claims Read more

The justice minister also said it was difficult to stop convicted criminals entering Britain. “You are importing risk, and the EU has forced us to import risk, and tied our hands in trying to deal with that risk. There is no doubt that that is the case. You can take a view that that is a price worth paying. I don’t think it is,” he said.

The result, according to Raab, is that the terror risk to Britain is higher than it might be post-Brexit. The prime minister has made the opposite argument, saying that cross border co-operation within the EU keeps Britain safer.

The question of immigration is at the heart of the Vote Leave campaign, and the subject of part of its referendum broadcast to be shown on Monday. It has a mother and daughter going to A&E and waiting frustratedly to be seen because of pressures resulting from immigration. The video then shows an alternative picture in which the smiling pair leave the hospital quickly in a post-Brexit world.

An internal Labour report by Jon Cruddas has said the party risks becoming “irrelevant” to working people if it fails to tackle their concerns on immigration. Cruddas has also contributed to a book being edited by the backbencher Tristram Hunt on why Labour lost the last election.