A Tory MP leading the opposition to lifting controls on Romania and Bulgaria has called for Vince Cable to step down from the cabinet after the business secretary warned that Conservative rhetoric on immigration was reminiscent of Enoch Powell's "rivers of blood" speech.

Nigel Mills said: "It would be very hard for him to sit around the cabinet table having effectively compared his Conservative colleagues to Enoch Powell, which is an utterly ridiculous thing to have done. Mr Cable's always had a rather creative interpretation of what collective responsibility ought to look like but these comments, coming on the back of some would say completely sensible policy announcements by the prime minister to restrict welfare for people newly arrived here [who] can't claim until they've paid in – it just looks completely out of touch with the sentiments of most of the British people.

"What he said yesterday was ridiculously over the top and ill-judged remarks. We've tried to conduct this debate in a sensible manner especially at a time when the economy's still pretty weak and for him to use such intemperate language really is unacceptable."

The extent of the bitter personal poison seeping into some of the coalition's key policy disputes was revealed on Sunday when Cable accused the Conservatives of grubbing for Ukip votes with irresponsible and populist rhetoric.

He said the Tories were creating a panic in Britain about the scale of migration from the European Union, saying it was the duty of political leaders to provide reassurance during times of national anxiety.

He also revealed he was concerned for the country's social fabric because of the scale of public spending cuts and for the first time warned there may have to be an increase in interest rates to ease a "raging housing boom" in the south-east.

He said if no action were taken, there was a risk that in parts of London housing would be too expensive for anyone but "foreigners and bankers".

Cable's remarks will infuriate Tory backbenchers, who claim they are merely voicing the fears of their constituents over the likely scale of migration when transitional controls on Romanians and Bulgarians are lifted on 1 January.

Cable also repeated Nick Clegg's dismissive rejection of Tory plans to place a cap on the number of EU migrants allowed into Britain, saying it was "illegal and had no chance of being implemented".

But it was Cable's branding of Cameron as irresponsible in his pandering for Ukip votes and likening the rhetoric to Enoch Powell's "rivers of blood" speech in 1968 that will cause most friction as the coalition seeks to remain a functioning government before the 2015 general election. Powell was sacked as shadow defence secretary after the speech, which was widely regarded as racist.

Speaking on the BBC's Andrew Marr Show, Cable said: "I think there's a bigger picture here. We periodically get these immigration panics, I remember going back to Enoch Powell and 'rivers of blood' and all that, and if you go back a century there were panics over Jewish immigrants.

"The responsibility of politicians in this situation when people are getting anxious is to try to reassure them and give them facts and not panic and resort to populist measures that do harm."

He added: "What's happening here is the Conservatives are in a bit of panic because of Ukip, reacting the way they are. It's not going to help them politically but it's doing a great deal of damage.

"The simple point is there is very little evidence of benefit tourism from people coming from eastern Europe. All the evidence suggests they put far more into the economy in terms of tax than they take out in benefits. It was right to stop abuse of the benefits system … but freedom of movement, albeit constrained as it is under the European agreements, is an absolutely basic principle and a lot of British people take advantage of it."

He also laid into "ridiculous rules" developed by the Home Office that prevented Indian and Chinese business people from visiting Britain.

He added that he feared for the UK's social fabric, saying "pressure on public spending is getting very severe – actually, some very good services are now being seriously affected".

Downing Street was unapologetic about its immigration policy, saying: "Vince is a member of the government and supports government policy. The words he chooses to do that are up to him."

But Cable again expressed doubts about macroeconomic policy, saying the imbalance of the recovery was helping to fuel "a raging housing boom in London and the south-east". He pointed out that last week Moritz Kraemer, head analyst at rating agency Standard & Poor's, had said that the surge in the UK housing market meant questions about sustainability remained.

Cable has repeatedly called for the Help to Buy scheme introduced by the chancellor to be discontinued, but Conservative housing minister Kris Hopkins rejected Cable's analysis, saying 3.4m property transactions were expected during the existence of Help to Buy and less than 2% of those would be through the scheme.

"This is not fuelling a housing boom. This is facilitating some 18,000 people get their foot on the housing ladder," he said.

Meanwhile the Bulgarian president has warned David Cameron he risks being judged by history as a prime minister who has isolated the UK and damaged its reputation. Rosen Plevneliev said Bulgarians were watching Britain's immigration debate unfold and raising questions about the "democratic, tolerant and humane British society".