When she campaigned for remain in the EU referendum, the Labour MP Lisa Nandy was collared outside a pub in her Wigan constituency by a man who was “quite cross” about Brexit. “Look, love, it was better before we were in the EU. I remember it, you don’t,” he told her. The man said he did not believe politicians would go through with Brexit anyway. “We never get to decide anything,” he told her.

Nandy said she had thought the man was wrong, but now, three years on, she has started to believe he was right. “It doesn’t feel, when you look right across the political parties, that there is a seriousness about doing this,” she said on Tuesday. “The amount of democratic harm I think we are doing is really serious.”

In the 48 hours since the European election results, Nandy has watched with growing frustration as several Labour frontbenchers have urged the party to explicitly back a second referendum to halt the exodus of pro-EU voters. The deputy Labour leader, Tom Watson, and the shadow chancellor, John McDonnell, are among the senior figures urging Jeremy Corbyn to take a more unequivocally pro-EU stance.

But in leave-voting towns such as Wigan, said Nandy, such a move would be a “final breach of trust” with working-class voters. “It would leave large swaths of the country politically homeless – not just northern towns like Wigan, or Midlands towns like Bolsover, but also places like Dagenham [east London]. The worry is that vacuum is only filled by populist or far-right parties. It doesn’t leave the door open to any prospect of a Labour government,” she said.

While Wigan has sent a Labour MP to parliament in every general election since 1918, the town rejected the party’s campaign to remain in the EU and voted strongly (63.9%) to leave in 2016. In the European elections last week, a resounding 41% of voters backed the Brexit party, pushing Labour into second place as it shed almost one-fifth of its vote compared with 2014.

In second referendum terms, 50% of people in Wigan voted for no deal-supporting parties or candidates, while only 21% backed the three remain parties – the Liberal Democrats, the Green party and Change UK. Despite turnout being just 28% – the lowest in Greater Manchester – the results support Nandy’s belief that opinions in some parts of the country are shifting towards leave rather than remain.

“Very few people have changed their minds and, if there has been a shift anywhere, it’s been towards no deal,” she said. “The other problem is there’s a very strong feeling among a large minority of people that there’s been no real attempt to implement the result of the first referendum. Now talking about a people’s vote suggests to those people, in towns like mine, that they’re not people and their votes don’t count.”

Facebook Twitter Pinterest Wigan has voted for a Labour MP at every general election since 1918 and is one of the party’s strongholds. Photograph: Mark Waugh/The Guardian

Other senior Labour figures have pointed to the party’s disappointing local election results as evidence against embracing remain. The party shed most of its seats in leave-voting areas in the east Midlands and across the north of England, where it lost control of councils such as Bolton and Darlington for the first time in 40 years.

In Wigan, Labour held on to its decades-long grip on the town hall, but still lost three councillors. James Paul Watson, a former Irish Guards serviceman, was one of the independents who unseated Labour. He said he won scores of votes from lifelong Labour supporters simply because he backed Brexit.

Watson said: “There’s a seismic shift happening where you have lifelong Labour voters – the older generation – who are now suddenly voting [for] independents like myself. If Labour do back a second referendum, they’re going to lose a lot of lifetime voters.”

Taking a break from shopping in the afternoon sun, Frank Picton, 62, said he was a lifelong Labour voter and backed leave in 2016, but could not bring himself to vote in the European elections. “I’m just disappointed really. It shouldn’t have got to that,” he said. “We voted to come out and there’s nobody supporting Theresa May, and I feel sorry for her even though I’m not a Conservative.”

Picton, a retired machine operative, was dismayed at the prospect of a second referendum. “I wouldn’t be happy about it – it shouldn’t go back to that. I’m not happy with Labour with how they’ve gone about it – it’s ripping them apart and they should all be together on this; it’s an important thing,” he said.

Outside the Life Centre, a leisure centre-cum-council office, Janet Berry, 69, another Labour voter, said she backed remain in 2016 but would vote leave in a second referendum. “I think a second referendum is wrong but I would vote leave now – it’s gone too far,” she said.

Her daughter, Susan Berry, 43, did not vote in 2016 but would also back leave in a second poll. “They asked people to vote and they didn’t like the answer – it’s a circus at the minute,” she said.

Bill Peach, 71, said he voted to remain in 2016 but cast his ballot for the Brexit party last week – “even though I can’t stand Nigel Farage”.

He added: “They’re just mucking about because two-thirds of parliament didn’t want to leave.” His wife, Sharon, 58, said she had stopped voting since the EU referendum even though she was as politically engaged as ever. “We set our clocks by the Six O’Clock News, but I’ve stopped voting,” she said. “I just can’t be bothered with it all these days. I know you really should, but I’m just being honest.”