Here’s the established wisdom on Labour’s Brexit traumas. While the Tories have ably gambled all on reconstituting their coalition of leave voters, Labour’s prevarications and triangulations have left it unable to similarly win back remainers who defected to the Liberal Democrats and Greens in the European elections. While Boris Johnson’s theatrical stunts – from purging his MPs to proroguing parliament – convinced recalcitrant Brexit party voters that the new prime minister was one of their own, Labour failed to press its own reset button with disillusioned remainers. Because the party’s voters mostly voted against Brexit, even in constituencies that mostly voted to leave, Labour’s foot-dragging in shifting towards a new referendum has imperilled its election chances.

It is a narrative with some truths. Labour should have broken off its Brexit negotiations with Theresa May’s crumbling administration sooner than it did. The leadership should have accepted that a pivot towards a referendum was inevitable by the time of the European elections in May. Whatever the costs of this repositioning, better to make the best of this inescapable shift, than look shifty and cynical, incurring the contempt of both sides of the divide. Before the election began, just a third of remain voters stated their intention to vote Labour, leaving the party level-pegging with the Lib Dems. If Labour had failed to embrace a new referendum, it would likely have entered the election campaign behind the Lib Dems, making electoral catastrophe inevitable.

Since the campaign began, Labour’s gradual recovery in the polls has been driven by those previously antagonised opponents of Brexit. While its support among remainers has risen by on average 10 points to 44%, among the Lib Dems that figure has slumped seven points to just 26%. A month ago, Jeremy Corbyn’s net favourability among remainers was a dire -33, now recovering to -4; for Jo Swinson, it has plummeted from +13 to -8. In certain remain seats, particularly in the south, Labour candidates have been surprised at how much their vote has held up.

Remainers who accepted the referendum result – let alone leavers – were conceived of as malign dupes

Here is the rub. For the most passionate remainers, Labour pivoting in their direction was a cost-free exercise. They were baffled by the party’s reluctance to wholeheartedly embrace the “stop Brexit” cause, and believed that a combination of Corbyn’s stubbornness and his Lexit sympathies, shared by his advisers, was to blame. Even now, some denounce Labour for not zealously championing remain, and seem unlikely to be satisfied even if Corbyn tattoos the EU flag on his chest.

Yet there is no question that the chief obstacle to Labour’s electoral ambitions is now on its leave flank. The party’s support has also grown among leavers during this election campaign, but from a derisory level: from 11% to 16%. There is a reason that the Tories believe their chances of securing a decisive majority lie in achieving what May failed to do: sweeping through Labour’s so-called “red wall” of leave-voting constituencies in the north, the Midlands and Wales. Constituency-level polling should be treated with caution, but according to a poll by Survation, Labour’s polling in Grimsby – a seat it has held since 1945 – has collapsed from 49% since 2017 to 31%, almost all to the Brexit party, which would allow the Tories to win through the middle.

Private research suggests that around 80 Labour leave seats are at some risk of being lost to the Tories (although it was conducted before Labour’s more recent polling recovery). In some Midlands seats, up to half of Labour leave voters have left the party’s fold. While it is true that most Labour supporters did not vote for Brexit in 2016, “remainer” is an unhelpful catch-all term. One Labour candidate representing a marginal leave seat tells me of constituents who voted remain angrily suggesting they will no longer vote for the party because it was not honouring the referendum result. This isn’t just about Brexit, though: Labour-leaning leavers tend to be socially conservative and that’s where Tory smears over immigration, national security and lack of patriotism tend to cut through the most.

Remain campaigners must acknowledge their own faults, too. After the referendum defeat, a wise – and dare it be said, a rather obvious – political strategy would have been to seek to persuade leave voters of the case for another referendum. With honourable exceptions, this did not happen. Indeed, Brexit was treated as a self-evident national catastrophe, while remainers who accepted the referendum result – let alone leavers – were conceived of as malign dupes. Some treated the result as illegitimate, stolen through illicit means and foreign meddling. Brexit was therefore a legal problem rather than a political one; and some remainers could hardly resist preaching their contentment with a pre-June 2016 status quo which had caused massive disillusionment among millions who voted both ways in the referendum. Rather than persuade ever more antagonised leave voters of the case for another referendum, the Labour leadership was vigorously denounced, and an existing cohort of remainers became ever angrier. The chief result has been a fracturing of the anti-Tory vote.

In the final two weeks of the campaign, Corbyn’s team still has much work to do to persuade remainers who have defected to the Lib Dems that their only hope is a Labour-led government. But the party’s prospects are doomed without the support of more leave-inclined voters. Domestic policies that are popular among them, from investment in creaking public services to public ownership, are key to winning them round. The threat posed to the NHS by a deal with the Donald Trump administration is critical, too.

Brexit has been a difficult juggling act for Labour. And if the party cannot attract support from voters from across the referendum divide, Johnson will secure his majority and a hard Brexit is just weeks away.

• Owen Jones is a Guardian columnist