The European elections have polarised the Brexit debate into two camps: those who want to remain and discard the verdict of the 2016 referendum, and those who want a “clean” or “no-deal” Brexit. Labour is urged to pick a side in this ongoing tug-of-war. That is a dead end for the party and for a very divided country.

The pollster Peter Kellner argues that Labour must come out unambiguously for remain. His reasoning is that two-thirds of 2017 Labour voters backed remain in 2016, that a greater proportion of Tory leave voters are now backing Nigel Farage’s Brexit party, and that there is a migration of Labour remain voters to the Liberal Democrats.

For Labour to simply make an electoral calculation based on the apparent short-term interest of the party in these European elections, to scoop up the remain vote and let the Brexit party take the bulk of the leave vote, would be a historic mistake. I’m glad we haven’t succumbed to it, in the face of huge pressure from many of our MPs, some members of the shadow cabinet and several well-funded pressure groups.

Why? First, because it does not add up. Kellner’s argument is based on the premise that the number of working-class leave voters who voted Labour in 2017 is relatively small. Working-class leave voters were far more likely to vote Tory, for smaller parties or not to vote at all in 2017.

But many of them were Labour voters once. Between 2001 and 2015 we lost millions of working-class votes – which, by the last general election, left us with an electoral profile quite unlike anything we held previously. This was a trend highlighted best by Labour’s stunning success in Canterbury, and by the staggering loss of Mansfield, which had voted Labour for generations. To abandon any prospect of representing those voters again is effectively an argument to abandon much of the working class – and with it any prospect of a Labour government.

To win a majority in the country, Labour needs to win seats in the parts of the UK that backed leave. There is no route to 10 Downing Street that doesn’t run through areas that voted heavily to leave and heavily to remain. It is right to acknowledge that, in all these seats, many 2017 Labour voters backed remain; but what of the millions of who voted leave the year before and Labour in 2017?

And what of the millions of former Labour supporters who voted leave, and who see Labour advocates in the media daily calling for their votes to be ignored? While the 2018 local elections saw us pile up votes in places such as Trafford and the London boroughs, we lost councillors in Sunderland, Bolton and Wigan despite eight years of austerity. We cannot win without the trust and support of those towns.

Moreover, calls for a “confirmatory vote” with only Theresa May’s unpopular agreement and remain on the ballot will be seen by many of those voters as dishonest. Significant numbers now want to leave without a deal and would consider this to be a referendum without a genuine leave option at all. For others, this is a straightforward attempt to overturn the result of the first referendum without making any attempt in good faith to implement it.

In areas where we’ve seen decades of decline, where good jobs have been replaced with zero-hours contracts and young people have left in large numbers, there is already a lingering sense that people’s views and experiences are not understood or respected. A rigged referendum will cost us the trust and respect of millions of current and former Labour voters. In a representative democracy trust is the glue that binds us together. Without it the system will not survive.

And the issue goes so much further than electoral maths. I may take a different view to many of my constituents on Brexit, but I like and respect them enough to know that wanting to leave the EU is not the same as being stupid or racist, as they have too often been labelled. They listened to the economic arguments I, and others, put forward in 2016 and took a judgment to vote leave. I am yet to meet more than a handful of people who have changed their minds.

Those working-class men and women come from families who worked in the mining, railway and textile industries to build this country’s prosperity and influence in the world. They have supported the Labour and trade union movement for a century and enabled us collectively to win employment, healthcare and housing rights that have changed this country forever. In recent years Labour has also been building its support in city and metropolitan areas, and I welcome that. But we also have a historic duty to the families in towns outside those big urban centres where we are struggling to retain support. A Labour party that cannot also speak for them doesn’t feel much like a Labour party at all.

We represent the most leave and remain areas of the country, which have been moving apart for decades. But our dilemma is the country’s dilemma, and if ever there was a time to step up to that challenge, it is now. It is the mark of real leadership not to pick a side and wave a flag, but to seek to bring these complex alliances and interests together to find a solution. Nobody else is trying. That is what Labour must redouble its efforts to do.

• Lisa Nandy is Labour MP for Wigan