On a quiet Monday at the end of February, the most influential voices in Labour’s Brexit policymaking gathered in one of the suite of riverside rooms in Westminster allotted to Jeremy Corbyn as leader of the opposition for a three-hour meeting that could have a profound influence on the party’s future.

How May miscalculated the Brexit numbers game Read more

Seven days earlier, Chuka Umunna and his band of splitters had summoned journalists to a press conference to announce they were leaving the party, blaming Brexit policy as well as a failure to tackle antisemitism.

In attendance were the shadow Brexit secretary, Keir Starmer, and Corbyn’s wily old ally John McDonnell, along with handpicked Corbyn supporters including Diane Abbott and Jon Trickett, plus the shadow international trade secretary, Barry Gardiner, and the Labour peer Shami Chakrabarti.

Later that day, Corbyn was due to address Labour backbenchers for the first time since the breakaway. With little more than a month left until Brexit day, longtime backers of a second referendum were ramping up the pressure on the leadership.

“One of the things people were thinking was that because the splitters – the evacuees – were making a point about Brexit, we should get our position out there,” recalled one shadow minister who was present. “We didn’t want a lot of people to start resigning because of that.”

Starmer certainly felt it was time for a shift and, in the preceding days, he had won the support of McDonnell, who was concerned about keeping onboard hundreds of thousands of overwhelmingly remain-backing activists who had surged into Labour to support Corbyn’s leadership.

Facebook Twitter Pinterest Labour’s stance on Brexit under Corbyn has pitched old allies against each other and opened new fissures within the party. Photograph: Christopher Furlong/Getty Images

The pair had latched on to an amendment being drafted by the Labour MPs Peter Kyle and Phil Wilson aimed at persuading the prime minister to submit her Brexit deal to a “confirmatory vote” by the public in exchange for support for it in the Commons.

Some Labour strategists were dismissive of the idea, regarding it as a backbench boondoggle cooked up to tempt the desperate Tories – one that risked tying Labour in knots by forcing it to tacitly support a Brexit deal the leadership had vehemently rejected. And they were confident that a majority of the wider shadow cabinet, not to mention a majority of MPs. were against a referendum.

But they had reckoned without the shift in political weather that came with the departure of the Tigs, and the persuasive powers of the likes of Starmer and Abbott, who also felt Labour needed to be able to say to its voters that there were certain circumstances in which the party would back a referendum – and if not now, when?

As the fraught meeting wore on, Gardiner and Chakrabarti, both of whom had previously been cautious about a referendum, switched sides. In the end, Trickett was comprehensively outnumbered.

Several sources at the meeting have told the Guardian that the decision appeared to come as a shock to party officials in the room, including Corbyn’s top adviser Seumas Milne. “Seumas looked shaken, to be honest,” said one.

Facebook Twitter Pinterest Chuka Umunna, right, and fellow Independent Group MPs in February. Photograph: Kirsty Wigglesworth/AP

Starmer stayed to sign off a painstakingly drafted press release announcing that the party would now back a public vote to prevent “a damaging Tory Brexit being forced on the country”.

It was a significant moment. Labour’s stance on Brexit has evolved through a hard-fought political tug of war, pitching old allies against each other and opening new fissures that will cast a long shadow over the Corbyn project. The struggle is still being played out in the background as Corbyn and key frontbenchers engage in unprecedented negotiations with the government.

Through conversations with more than a dozen senior Labour decision makers, it is possible to build a picture of an extraordinary few months that have shaped the future of the party and, perhaps, of the country.

Quick guide Labour's key moments in the Brexit process Show Hide August 2016 Making explicit the position he held since the Brexit vote, Jeremy Corbyn says Labour will respect the result of the referendum. "I think we've had a referendum, a decision has been made, you have to respect the decision people made." June 2017 Wary of upsetting voters in leave areas, Labour's manifesto for the 2017 general election declares: "Labour accepts the referendum result and a Labour government will put the national interest first." September 2018 Over five gruelling hours, Labour's internal contradictions over Brexit are distilled into a conference composite motion that makes the party's priority to seek a general election but, failing that, says a referendum should not be ruled out. February 2019 Chuka Umunna and remain MPs from both Labour and the Conservatives quit to form the Independent Group, citing Brexit as a major motivating factor. March 2019 Labour backs attempts by backbenchers to keep the option of a public vote on the table to stop a no-deal Brexit or May's deal. When all options considered by parliament fall, Labour enters into Brexit talks with the government but says May must move on a customs union if they are to succeed.

For Starmer, the announcement of Labour’s qualified support for a “public vote” was a staging post on a journey that had begun more than five months earlier when he chaired a late-night meeting with 300 party delegates that drafted a two-page “composite motion” setting Labour’s Brexit policy.

The meat of the motion said Labour would vote against Theresa May’s deal and push for a general election, but that “if we cannot get a general election Labour must support all options remaining on the table, including campaigning for a public vote.”

Starmer then went further, deftly dropping into his conference speech the view that if another referendum were held, remain would have to be an option on the ballot paper. He won himself a warm round of applause.

Labour’s cadre of bright young advisers immediately grasped the significance of the moment and at one conference party they launched into a refrain of “All options on the ta-ble”, to the tune of the White Stripes’ Seven Nation Army, to which a year earlier at Glastonbury the crowd had roared “Oh Jeremy Corbyn”.

Corbyn’s tone in his end-of-conference speech was rather different. While he praised Starmer, he made a striking promise to May that, if she would compromise, Labour would be willing to vote for a “sensible” Brexit. He also set out what that might mean: a customs union and better protections for workers’ rights and environmental standards.

These two contending visions – a Labour Brexit, and a public vote to reverse the 2016 referendum result and stop Brexit – have continued to shape the debate at the highest levels of the party ever since.

For many of its MPs and the overwhelming majority of members, Labour’s natural home is in what Corbyn’s team waspishly call Remoania. But the Labour leader is a longtime Eurosceptic and he made clear in the early hours of the morning after the Brexit vote that his party would honour the result.

There are strategic imperatives behind the Labour leadership’s hesitation about a referendum, too. A cautionary presentation to the shadow cabinet recently suggested that, as a result of the shift towards supporting a fresh poll, Labour would suffer in May’s local elections, many of which are being held in leave-leaning areas. Projecting the same analysis on to a national scale suggested the party could struggle to win a majority at a future general election.

More broadly, many of those around Corbyn believe their party would benefit if the political debate moved on beyond Brexit, to issues on which Labour is less divided. “We want to get to where it’s the many versus the few, and that’s what politics is again – but we know that to get there we need to keep the coalition together,” said one senior party strategist.

The backlash against the party’s qualified support for a referendum began almost immediately, at the next shadow cabinet meeting, with Trickett’s objections amplified by several others including Ian Lavery, Richard Burgon and Rebecca Long-Bailey, who felt they had been bounced.

Facebook Twitter Pinterest Hundreds of thousands of people attended a Put it to the People march in London last month. Photograph: Paul Marriott/Rex/Shutterstock

Shadow ministers on both sides of the debate note wryly that attendance at decision-making meetings expands and contracts, apparently at the whim of the leader’s office. At the regular Tuesday shadow cabinet meetings, where members are now asked to hand in their mobile phones after a spate of leaks, Brexit discussions have been often perfunctory but occasionally explosive.

One one occasion, Lavery responded angrily to a warning from Abbott that Labour should not give ground to arguments against migration, accusing her of calling him “gammon” – the derisive phrase used by lefty activists to describe middle-aged white men with reactionary views.

Advocates of a referendum frequently complain that advances they believe they have made tend to be spun away – the most obvious example being Corbyn’s February letter to the prime minister setting out his conditions for supporting a Brexit deal.

Starmer drafted the letter, which fleshed out Labour’s policy, and quietly downgraded some aspects of it, including the party’s rejection of state aid rules, in a process insiders dubbed “de-unicorning”.

An early draft included a reference to a public vote, but this was removed in an editing process involving Milne. Starmer signed off the changes on the understanding that, in publicising the letter, the leader’s office would make reference to a referendum – but that didn’t happen.

“It’s obvious that Seumas is the one that unpicks it every time,” said one shadow minister. However, allies of Milne described the idea that he or any other individual was subverting Labour’s policy as “conspiratorial nonsense”.

Whatever the leadership’s reservations, from the moment the composite motion was unanimously agreed, Starmer felt himself to be its solemn guardian and, as the months ground on, it gave him a weapon to nudge and harry the leadership, urging them to “move through the gears”, as he called it.

Their chosen vehicle became an amendment originally conceived by the backbenchers Kyle and Wilson in an attempt to win over the Tories to the idea of a “confirmatory” referendum on May’s deal.

Kyle told May at prime minister’s questions that he was willing to lend support to her deal if she would agree to submit it to a confirmatory vote in a process that echoed the ratification of the Good Friday agreement.

Facebook Twitter Pinterest Brexit divisions have cut across the left-right divide that has traditionally defined who is on or offside in the Corbyn project. Photograph: -/AFP/Getty Images

Even after the announcement on 25 February that Labour would support a Kyle-Wilson-type amendment, Labour avoided parading its divisions in public, as People’s Vote campaigners held back from forcing the issue, judging that the later they tested parliamentary support for the idea, the more votes it would win.

But when the Independent Group’s Sarah Wollaston put down an amendment on 14 March to the government’s motion calling for an article 50 extension, some of Labour’s divisions began to be flushed out into the open.

When backbenchers then seized control of the parliamentary timetable to hold “indicative votes” on alternatives to the prime minister’s deal, those Labour tensions exploded.

On the long day of the first round of voting, which ultimately yielded no majority for anything, there were a series of flare-ups, not least a furious stand-up row between the People’s Vote campaigner Stephen Doughty and the Manchester Central MP, Lucy Powell, the informal whip for “common market 2.0”, a cross-party backbench plan for a soft Brexit.

Before the votes, Corbyn dispatched a letter to all Labour MPs insisting he was only supporting the pro-referendum motion to “keep the option of a public vote on the table in order to stop a disastrous no deal or May’s unacceptable deal”.

Three members of the shadow cabinet – Trickett, Lavery and Andrew Gwynne – ignored the whip and abstained on the referendum motion. Exasperated colleagues waited in vain for them to be disciplined, let alone sacked.

Another 27 Labour MPs voted against the motion, including an informal grouping of backbenchers from leave seats, among them Gareth Snell and Caroline Flint, who have been resolutely against a referendum for many months, believing they would face a ferocious backlash from constituents.

Other Labour MPs also fret about the potential impact of a referendum on British political debate after more than two years of fruitless wrangling in Westminster. Stephen Kinnock, who has worked with Powell’s common market group, said: “We could potentially be tipping over into a culture war.”

Facebook Twitter Pinterest Keir Starmer and Rebecca Long-Bailey leaving the Cabinet Office after cross-party talks on Brexit this month. Photograph: Andy Rain/EPA

Running throughout the debate in the leader’s office and among leftwingers in the shadow cabinet was a dark suspicion that the Brexit issue was being used as a wedge to separate Corbyn from the Labour membership and perhaps open the way for someone else to take over. The deputy leader, Tom Watson, as the left’s favourite bogeyman, is often mentioned.

But Brexit divisions cut across the left-right divide that has traditionally defined who is on or offside in the Corbyn project. Perhaps even more concerning for the leadership is a group of leftwing MPs – including Clive Lewis, Lloyd Russell-Moyle and Rachel Maskell – who are increasingly finding their voice.

Maskell and Lewis set up Love Socialism, Hate Brexit, a Labour grouping backed by the anti-Brexit campaign Best for Britain, which had its own “left block” march alongside the Put it to the People rally last month.

Lewis warned of the risk that if Labour was regarded as complicit, “then for many years to come it will be held responsible by many for every job loss, every poor economic indicator, every nurse, teacher or new medicine we’re told we just can’t afford.”

So alarmed were the leadership about the risks of division and ill-discipline being displayed afresh in a second round of indicative votes that Corbyn’s political director, Amy Jackson, was dispatched to urge Kyle to pull his troublesome motion.

He flatly refused, and it was redrafted and resubmitted for the 1 April votes, again in Margaret Beckett’s name. “Kyle-Wilson became a cipher for people who wanted our policy to be full ‘People’s Vote’ – it became totemic,” one Labour strategist complained.

In the event, it fell just 12 votes short of a majority, as People’s Vote supporters threw everything they had at building support for it, begging, cajoling and persuading their colleagues.

The near-miss gave heart to those, including Starmer, who have suggested any deal now reached – including in the talks between Labour and the government – must be submitted to the public for approval.

Watson has taken that position publicly, and a leaked letter from Emily Thornberry, the shadow foreign secretary, made clear she too felt any deal would have to be put to the public.

But the shadow cabinet as a whole remains far from won over to that view. Many would be satisfied with the offer – mooted in the negotiations with the government – of a free vote on a referendum, separate from any new meaningful vote on a cross-party deal.

Senior Corbyn allies stress that if May signed up to a customs union and the stronger protections for workers’ rights and environmental standards they have demanded, it would no longer be the “damaging Tory Brexit” that their support for a referendum was designed to avert.

But anti-Brexit campaigners on the backbenches, from left and right, are putting their faith in Starmer as their man in the room. “Keir won’t let us down,” said one MP.

Labour is serious about the talks, and Corbyn believes the public would look more kindly on politicians willing to compromise than on ideologues. “The country wants there to be an agreement and an outcome; I don’t think there is any doubt about that,” his spokesman said last week.

Anand Menon, the director of the thinktank Britain in a Changing Europe, said the constructive ambiguity of playing to the audiences of both leavers and remainers, was never going to hold for ever.

“I think Jeremy’s actually played it really, really well, and the 2017 election result was the proof of that,” Menon said. “But I think it was always going to get hard come decision time. He’s just been unlucky, in the sense that decision time has lasted six months rather than a week.”

However grave Labour’s challenges, the prime minister’s appear worse. As one shadow cabinet member put it: “We still think there’s a bit of road left. The Tories could break before we do.”

Tomorrow: Rory Carroll and Lisa O’Carroll on Brexit and the Irish border.