Labour has backed down on claims that Tesco and Next are examples of "unscrupulous employers" over their use of foreign workers.

The party became embroiled in a row after the shadow immigration minister, Chris Bryant, briefed newspapers that he would criticise both firms for favouring foreign workers over Britons. But just hours before he was due to give a keynote speech on the issue, Bryant was forced to retreat from his case as Tesco and Next accused him of making basic errors.

The senior Labour MP insisted he had meant more generally that "sometimes there are negative sides of migration in terms of the labour market".

The controversy has added weight to concerns that Labour is failing to articulate its key policies this summer. It has also overshadowed Bryant's other messages about the exploitation of foreign workers and sham marriages.

Bryant was forced to clarify his position on BBC Radio 4's Today programme, saying the extracts had not conveyed the fullness of his argument.

In the text of his speech, due to be delivered on Monday but leaked over the weekend, Bryant had said: "It is unfair that unscrupulous employers whose only interest seems to be finding labour as cheaply as possible will recruit workers in large numbers in low-wage countries in the EU … Take the case of Tesco … they recently decided to move a distribution centre in the south-east. The new centre is larger and employs more people. But it has been alleged that the staff at the original site were told that they could only move to the new centre if they took a cut in pay. The result? A bigger percentage of the staff at the new centre are from eastern bloc countries."

Speaking later on the BBC, Bryant said he used the word "unscrupulous about another category of people", not Tesco and Next. He said those firms actually "often go the extra mile" to employ British workers, but can find it difficult to get local staff.

"Both Tesco and Next have had to use non-UK nationals in their new operations … in relatively large numbers. My question is, and always was going to be and still remains today, when we have 1 million unemployed youngsters under the age of 24 and a very high level of long-term unemployment, is there a way we can regulate labour markets better so foreign workers are not exploited, as happens in some cases, not specifically Tesco and Next, and also make sure local workers stand a better chance?"

He added: "I don't think it is unscrupulous to employ migrant workers."

Bryant was also pulled up on a mistake in the speech suggesting Tesco had opened a new distribution centre in Kent, when it should have been Dagenham, in Essex.

Challenged on his errors, Bryant told his interviewer: "You're wanting to get into very specific details which are contested and I'm not sure that's that fruitful."

The shadow minister later said any mistakes in the extracts were "entirely my responsibility" but urged people to listen to Labour's argument on immigration in his speech.