Ed Miliband has signalled a change in Labour's immigration policy by disclosing he wants to change the economic rules to do more to help people already living and working in Britain.

In an interview with the Guardian, he concedes that immigration is being discussed in "every kitchen" and that the Labour party has been too quick to dismisses the concerns of ordinary people as "prejudice".

He says the government should strengthen the law so that employment agencies cannot – even informally – favour foreign workers.

"In sectors where there is a problem, every medium and large employer that has more than 25% foreign workers – double the average share of migrants in the population – should have to notify Jobcentre Plus."

Miliband said he wanted to reframe the debate on immigration by pledging to reform a "brutish labour market" that encourages the excessive use of low-paid immigrants and allows British companies to shirk their responsibilities to train workers. But he distanced himself from remarks made by the former prime minister Gordon Brown, suggesting that it would be wrong to promise too much. "We cannot tell people things we cannot deliver. We cannot say 'British jobs for British workers'."

Speaking on the Today programme on Friday, the former Labour Home Office minister John Denham said the Labour government had been wrongly advised about the number of migrants who would enter the UK after new countries entered the EU, and he had become aware there was a problem in 2005.

"It became clear that the estimates that we had relied on were vastly wrong. We expected 15,000 migrants and 15,000 came to Southampton." The country was put under strain as a result, he added. "There was an impact on wages and public services that people were concerned about," he said.

There was a "debate still to be had" about overall numbers, he said, but added that the UK had benefited "economically, culturally and socially" from migration. A Labour administration would focus on three things to help ease pressure on public services and wages including better enforcement of the minimum wage, cracking down on recruitment agencies who only supplied workers from particular countries and looking at areas and types of jobs where there were large numbers of foreign workers. "We need to make sure there is a level playing field … What [Ed Miliband] is saying is that there has to be a fair chance for everybody," he said.

But the Tory immigration minister Damian Green attacked Denham for not condemning the level of immigration while Labour was in power. "John Denham has blown apart Ed Miliband's so-called apology for immigration under Labour by revealing what Labour really think," he said.

"They still don't think immigration was too high when they were in power and they still won't say that immigration needs to come down. That's why they've opposed every one of the Government's policies to cut immigration, and it's why they cannot be trusted to run Britain's immigration policy."

Speaking to the Guardian, Miliband admitted the Labour government allowed too many immigrants from eastern Europe into the country by lifting controls on EU accession countries such as Poland too quickly, but denied his party lied about immigration, as claimed by his former adviser Lord Glasman.

Miliband argued that immigration should be seen as a class issue, since the evidence shows lower-paid workers and the unskilled suffer disproportionately, especially from the impact of cheap eastern European labour. He said he was entering the debate on immigration in the context of his call for a more responsible capitalism.

"There has been a collision of a large amount of immigration from eastern Europe and a UK labour market that is frankly too often nasty, brutish and short-term," he said.

As part of a policy review, Miliband proposes sanctions against labour agencies that advertise solely for immigrant workers, an early warning system if some industries are employing disproportionately large number of foreign workers, a doubling of fines if employers undercut the minimum wage, and no early lifting of migrant barriers for new EU countries such as Croatia.

He also told the Guardian he would review immigrants' access to benefits and "local connection" rules for council house waiting lists. "There are issues around the pace of change in communities, pressures on public resources and making sure entitlements work fairly," he said.

"There are clearly issues that people have been raising over a number of years. We have to look at where the rules are right and, if they are wrong, what we can do about it. You have to have the right entitlements in place."

Miliband said: "We have got to talk about this issue because the public are talking about it. I am for politics that is relevant to people's lives. We cannot have a debate going on in every kitchen, street and every neighbourhood and the Labour party not talk about it."

He said: "Labour has to change its approach to immigration but you cannot answer people's concerns on immigration unless you change the way your economy works.

"Overall, immigration has benefits, but the thing we did not talk about was its relevance to class, and the issue of where the benefits and burdens lie. If you need a builder, it is good that there are more coming into the country and lowering the price of construction, but if you are a British builder it is less beneficial.

"In government we were not sufficiently alive to the burdens, so when people said they were concerned that their wages were being driven down by people from eastern Europe our response too often was to argue that these people are saying 'stop the world, I want to get off', or at worst 'this is prejudice'. I think we were too starry-eyed about globalisation's benefits.

"We have to confront the fact you cannot address people's concerns about immigration unless you change the way the economy works.

"It is the short-term, fast-buck culture that is at the root of this, so we have to look at what incentives we can give companies so they do not rely on a pool of short-term temporary labour that will come to this country and go away again."

He said he did not advocate ending the free movement of labour within the European Union, or even think it possible.

Miliband said he was not "trying to mischaracterise Gordon Brown" over his controversial speech on British jobs for British workers in 2007. "But the problem is that phrase came across as giving the impression we were promising to do something we cannot deliver." He said fact and anecdote had to be separated in the debate, pointing to evidence that immigration drove down wages mainly in lower-paid sectors, and created unemployment only in some localised markets such as catering, construction and food processing.

Although government evidence shows that 370,000 people who came to the country as foreign nationals are claiming benefits, Miliband pointed out that proportionately fewer migrants were claiming benefits than in the UK labour force.

Polling recently published by Policy Exchange shows that a Labour shift on immigration and welfare would be the single most important issues to win back Labour swing voters.

A Conservative spokeswoman said: "Ed Miliband says he is not going to promise British jobs for British workers, but he seems to have fallen into same trap as Gordon Brown. He still opposes everything the government is going to do to cut and control immigration, and still is not offering a single credible immigration policy of his own."

First steps in Labour immigration policy

• Impose maximum transitional controls for 7 years on the future EU accession countries such as Croatia. No change to free movement of labour within EU.

• Stricter enforcement of minimum wage laws and doubling of fines to £10,000 for those that break the law. Currently fewer than 7 employers have been fined.

• Regulate employment agencies so they cannot operate exclusionary practices, such as advertising they only have Poles on their books.

• An early warning system so that job centres councils and national government can identify sectors where workforce is dominated by low wage labour from other countries. Areas currently include construction, food processing and catering.

• Review immigrants access to key benefits, public services and access to housing lists.

• Review the relevance of government caps on immigration numbers.

Miliband spoke to the Guardian ahead of a speech ahead of an event organised by the Institute of Public Policy Research.