The most senior Labour figure in the campaign to leave the EU has warned that the party’s stance on the referendum is a “recruiting agent for UKIP” as new analysis shows the party is struggling to connect with working class voters.

Gisela Stuart, the Labour MP who sits on the board of official Out campaign Vote Leave, said many traditional Labour voters have “legitimate concerns” to which they feel the party is “not even responding”.

Meanwhile, analysis of the recent local elections results by Lewis Baston for the Fabian Society has found that the party’s current appeal seems better with certain middle class areas than the “traditional working class.”

Stuart claims that one of the roots of Labour’s problem is its largely pro-EU position – and the impact on immigration. According to The Guardian, the Edgbaston MP said: “I think in whole swaths of the country, that stance is the biggest recruiting agent for UKIP I can think of,” she says. “They feel there are legitimate concerns they have, and Labour are not even responding to it.”

Stuart, who served on the Foreign Affairs select committee for nine years, argued with the Remain campaign’s claim about the UK’s ability to control immigration while a member of the EU, and that many of those on the pro-EU side were dismissive of Out voters’ concerns as uneducated.

She said the In campaign is “stretching the use of the English language: they say we have ‘control’ of our borders because we’re not part of Schengen, but all we have is the power to ask someone to show a piece of paper.”

She added: “There’s a bit of sneering, which is, if you’re voting for leave, you’re probably the wrong side of 60, you’re probably not very well educated.”

But she also said that it was not just white British working class communities that had issues with immigration, and that the issue was also alive in immigrant communities too: “If you’re an MP in a big city, immigration matters, and it is the first and second generation immigrants who are concerned about immigration.”

“Families of second or third generation immigrants from the Indian subcontinent find it really difficult: they say, why do we have to jump so many hurdles just to bring in relatives for a wedding?”

In the post-election analysis for the Fabians, Lewis Baston said the spread of the vote on May 5 showed Labour was performing unexpectedly well in “classic New Labour territory”, but is seeing its recede in more traditional Labour areas, which includes some swing seats.

“The best Labour results were in some of the most modern bits of England, in London and its hinterland. Swindon, Milton Keynes, Reading and Crawley, and the leafy London suburbs, are what used to be regarded as classic New Labour territory, but now seem oddly fond of new old Labour,” he wrote. “A more traditional socialist appeal seems to go over better with these voters than with the traditional working class.”

He added: “Weakness in crucial types of constituencies in 2016, such as unpretentious Midlands towns (Nuneaton, Cannock) and big city suburbs (Bury, Bolton) is ominous, while stronger showings were in affluent seats that are either already Labour or require large swings to be sustained through to May 2020.”