Iain Duncan Smith, the work and pensions secretary, has called on firms to give the British unemployed a try rather than hiring a foreign worker.

The Conservative cabinet minister also defended the comments of home office minister James Brokenshire, who said a "wealthy metropolitan elite" were benefiting from cheap labour such as cleaners and nannies at the expense of "ordinary, hard-working people".

Speaking to the House magazine, Duncan Smith aligned himself with people who believe foreign workers are depressing wages and taking up low-paid jobs that otherwise could be carried out by Britons.

He said: "I was talking to a plumber and a carpenter; trained, qualified individuals. And I asked them about the Olympic sites. And they said, 'They're all or mostly people coming in from eastern Europe who undercut us because they all hotbed in these areas and then they're gone.

"I can't compete with them on this because I've got a house, I've got kids, I've got commitments, there's only so far I can go on salary, otherwise it's worse for me than being on benefits."

Duncan Smith added: "Now more jobs are going to British workers than went originally. So that is the beginning of a change, but I'm very keen to thrash that out so try one of our unemployed people first before you go anywhere else."

He also said Brokenshire was merely making the point that richer people tend to benefit more from immigration than low earners. "He was saying the benefits of this, such as they are, tended to be in this group but not down at the bottom end, where they've tended to see the problem," he said.

"He's right in that by the way. That is the issue. People that I deal with every day, they feel very squeezed by people who come in, undercut them in price."

A politically sensitive report on immigration and job prospects was published earlier this month which suggested the impact of such immigration is less than some ministers have claimed.

It said: "There is relatively little evidence that migration has caused statistically significant displacement of UK natives from the labour market in periods when the economy is strong.

"However, in line with some recent studies, there is evidence of some labour market displacement, particularly by non-EU migrants in recent years when the economy was in recession."