John Mann, the pro-Brexit Labour MP, says the party will lose support if it pursues the policy announced by Jeremy Corbyn, and ends up enabling a public vote on the UK’s relationship with the European Union. There are two problems with his argument. One concerns the nature of Labour’s choice, while the second concerns the polling data.

The choice first. In the 2017 general election, Labour won the support of around 8 million remain voters and 4 million leave voters. Plainly, the party wishes to retain the backing of both groups. This is one reason (albeit not the only one) why the party leadership has tried to ride both horses since the election – sympathising with the concerns of voters on both sides.

However, the two-horse option could not last for ever. At some point, Corbyn would have to decide which one to ride – a new public vote, wanted by most remainers, or an enabler of Brexit, wanted by leavers. In pragmatic, electoral terms, the issue is which will win more voters – or, perhaps, lose least. For Mann’s prediction to be right, riding the pro-Brexit horse would have to be more popular than the pro-remain horse.

This brings us to the second problem with Mann’s view: the numbers. In the 2016 referendum, Labour voters divided 67-33% in favour of remain. According to a recent YouGov survey among more than 25,000 voters, they divide 74-26% for remain, if the 2016 referendum choice were to be rerun. If – as seems more likely – a public vote would pitch remain against Theresa May’s withdrawal agreement, the majority among Labour voters widens further, to 82-18% for remain.

The latest YouGov poll, published today, reinforces this point. Just 18% of Labour supporters think the UK was right to vote to leave the EU; 74% say we were wrong. Excluding the don’t knows (8%), the party divides: right 20%, wrong 80%.

One consequence of this is that in almost every Labour seat – and it’s possible that we no longer need the “almost” – most Labour supporters voted remain. In these seats, the bulk of leave voters would prefer a Conservative or Ukip MP. Pro-Brexit Labour MPs cannot assume that a majority of their own party supporters share their hostility to the EU.

All this helps to explain the numbers that should really frighten Labour’s leadership. At the start of the year, well before events that led to the Independent Group being formed, 79% of those who voted Labour in 2017 said they would stay with Labour in a fresh election. Now, a week after the Independent Group’s formation, that proportion is down to 72%. That is an exodus of around 800,000 voters in the past six weeks.

That is not all. When YouGov adds the option of the Independent Group, the proportion of Labour loyalists falls to just 57% – a further loss of around 1.6 million votes. As many as 28% of Labour’s 2017 supporters say they would switch to the Independent Group.

Mid-term polls are, of course, unreliable predictors at the calmest of times; and the times we are living through are far from calm. Nevertheless, the risks to Labour of alienating the pro-remain majority of its supporters are huge. In purely pragmatic terms, the party needs to keep them onside. It is not obvious that the best way to do this is to follow John Mann into the pro-Brexit division lobby in the Commons.

What, though, about the potential for Labour winning, or winning back, leave voters by taking Mann’s advice? Bluntly, these votes don’t exist – or, to be more precise, a more committed pro-Brexit stance would actually alienate more leave voters than it would attract.

In its big new year poll, YouGov explored the views of more than 2,000 people who voted leave in 2016 and Labour in 2017. Asked the standard voting intention question, 76% said they would vote Labour today. But when the same 2,000 were asked how they would vote if a Brexit deal were passed “with the support of most Conservative and most Labour MPs”, that figure actually fell further, to 71%.

What about the 5,700 Conservative leave voters in YouGov’s sample – does Labour offset its leave deserters by making inroads into these? Not at all. When asked the normal voting question, 2% say they vote Labour today – and the proportion that would back a pro-Brexit Labour party is exactly the same.

Again, great caution must be taken with hypothetical polls in such a fluid political context. But if I were Mann I would want some evidence that his Brexit policy would be more popular than a pro-remain stance. Not only does that evidence simply not exist; the data we have suggests precisely the opposite.

• Peter Kellner is a former president of YouGov