Keir Starmer has expressed doubts that any cross-party Brexit deal lacking a confirmatory referendum could pass parliament, warning up to 150 Labour MPs would reject an agreement that did not include one.

The shadow Brexit secretary said he feared the party risked losing its remain voters after worse than expected losses in the local elections, but he warned Labour remainers tempted to vote for the Liberal Democrats or Change UK that only Jeremy Corbyn’s party could deliver a fresh referendum.

In his first major interview since talks with the government began almost five weeks ago, Starmer:

Suggested a referendum must be part of any package agreed with the government.

Said Labour could call time on the cross-party talks within days if it became clear no new offer was forthcoming.

Warned Theresa May would further damage the prospect of any deal if she set an imminent departure date from No 10.

Said the option was “on the table” to face Nigel Farage or Tommy Robinson in an EU elections debate.

Starmer, a human rights lawyer for 20 years, is to be the key face of Labour’s EU elections campaign, which will be framed as a battle of values.

He said the poll on 23 May would be a fight Labour “cannot sit out” for tolerance and internationalism against populists such as Farage and Robinson.

“If the vote that is progressive is split then all that does is open up the path for the Brexit party and allow it to pretend it represents the majority view in this country,” he said. “On the critical issues, like a close economic relationship with the EU, and a confirmatory vote, only Labour can deliver on those.”

He said he feared the lesson of Labour’s net loss of more than 84 seats in the local elections was that it could shed remain voters as well as leavers.

“There is concern in leave areas about whether some of our voters might vote for other parties, but I think there is an increasing concern that some of the Labour remain voters might not vote Labour,” he said. “It is very important that we learn those lessons.”

An Opinium/Observer survey of voting intentions in the European elections released at the weekend gave Nigel Farage’s Brexit party 34% of the vote, 13 points ahead of Labour on 21%. The Lib Dems were on 12% and the Conservatives in fourth on 11%. Barry Gardiner, the shadow international trade secretary, told Sky News that the surge in support for Farage was from voters “angry that the referendum result has not been delivered on by a incompetent government”.

Starmer has been involved in cross-party talks with the government aimed at agreeing a Brexit deal that can be put to parliament.

Facebook Twitter Pinterest Rebecca Long-Bailey (left) and Starmer have different views on the necessity of another referendum. Photograph: Tolga Akmen/AFP/Getty Images

Ahead of another crunch meeting of both sides on Monday, he expressed serious doubts a compromise plan could command a stable majority without a confirmatory referendum as part of the main package.

“A significant number of Labour MPs, probably 120 if not 150, would not back a deal if it hasn’t got a confirmatory vote,” he said. “If the point of the exercise is to get a sustainable majority, over several weeks or months of delivering on the implementation, you can’t leave a confirmatory vote out of the package.”

Starmer has previously made it clear he believes Labour should back a referendum on any negotiated deal, in contrast to the shadow business secretary, Rebecca Long-Bailey, who has said a referendum may not be necessary if Labour can agree the right deal.

“I’ve made it clear that at this stage, at this 11th hour, any deal that comes through from this government ought to be subject to the lock of a confirmatory vote,” Starmer said.

He said finding a parliamentary majority for any deal, whatever the circumstances, was “very difficult”, and suggested he could not sign up to any agreement if he feared it would fail.

“It has got to be something truly deliverable,” he said. “For many of my colleagues, they have made it clear that they will not vote for a deal without a confirmatory vote attached to it. So if you want that stable majority, that has to be taken into account. And without it, it is impossible to see how the numbers would stack up.”

Starmer said there was seriousness on both sides, but progress was hindered by the substance of the government’s offer on Labour demands, such as a comprehensive customs union, and the threat of a new Conservative leader.

Facebook Twitter Pinterest Starmer said Theresa May setting a firm date for her departure from No 10 would not help matters. Photograph: Will Oliver/EPA

“Is the government actually prepared to change its red lines? The answer is, so far, not really,” he said. “On substance, there is a considerable distance between us.”

He said progress had not been helped by remarks from Tory leadership hopefuls, such as Jeremy Hunt’s suggestion he would oppose a customs union, or Boris Johnson’s comments about tearing up current agreements. “These talks cannot be divorced from the very obvious leadership battle,” Starmer said.

With May under increasing pressure from her party to set a firm date for leaving No 10, Starmer suggested this would damage progress even further. “You don’t normally seek to do a deal with someone about to leave office. Therefore, if she sets a date, it makes things even more tight,” he said.

The shadow Brexit secretary suggested Labour should not be afraid to call time on the talks within days if progress seemed out of reach.

“I think it would be wrong in principle to use up much more time simply exploring each other’s positions,” he said. “I do think we do probably in the coming days need to make that assessment.”

He said he did not believe May had a plan for where to go next if the talks failed, but said Labour would be highly likely to oppose plans to bring forward the EU withdrawal bill without any changes.

Starmer will be at the helm for the Labour campaign in cities and remain-leaning towns in the coming days, though Corbyn launched the party’s European elections campaign in Medway, Kent, which voted for Brexit.

Play Video 0:37 Jeremy Corbyn says second referendum could be 'healing process' – video

He said he was determined to see the polls as “an opportunity to have a fight about the values we actually believe in” and renew Labour’s mission as “an internationalist party”.

“It’s really important we make the case that this is not the country of Nigel Farage and Tommy Robinson,” he said. “That intolerance and hatred and division is not representative of our country. If they win, that is what they are representing.”

Starmer, often tipped as a future leadership contender, is demonstrably keen to have the opportunity to speak about the values he pursued as a barrister. He was joint head of Doughty Street Chambers by the age of 40 and later the director of public prosecutions.

“I was a human rights lawyer for 20 years, I believed those values of dignity, equality and non-discrimination were a given,” he said. “I believed the only question in my lifetime would be – how much further do we extend those values? I did not think in my lifetime we’d actually be having an argument about those values.”

Asked if he would debate Farage or Robinson directly, he said: “I’ll adopt the approach I’ve had for the last few years – all options should be on the table.”

Starmer said he hoped supporters of progressive parties could unite around trying to defeat intolerance and said the elections were more than a proxy referendum. “There needs to be a united progressive fight; this battle of values is much bigger than one political party,” he said.