An insurgent Brexit party and reinvigorated Liberal Democrats have delivered a harrowing night for the Conservatives and Labour at the European elections, prompting profound soul-searching at the top of both major parties.

Nigel Farage’s Brexit party humiliated the Conservatives in their rural heartlands but also made sweeping gains in cities such as Cardiff, Leeds and Sheffield, as well as in Hillingdon, the home of Boris Johnson’s seat, where the Tories were pushed into fourth.

Farage’s success campaigning in favour of a no-deal Brexit is likely to push the Conservative leadership candidates into hardline positions on leaving the EU.

Theresa May said the European elections were a “very disappointing night” for the Tories, adding: “It shows the importance of finding a Brexit deal, and I sincerely hope these results focus minds in parliament.”

Jeremy Hunt, the foreign secretary, warned that the Conservatives were facing an “existential threat”, while Johnson said it was a “crushing rebuke” to the government’s failure to take the UK out of the EU.

The night also confirmed an extraordinary revival of the Lib Dems, who overtook the Tories in Theresa May’s Maidenhead seat and came first in Jeremy Corbyn’s north London home of Islington.

Overnight, the Brexit party gained 28 seats, with the Lib Dems in second on 15 seats. Labour held 10, having lost seven so far, the Green party won seven, a gain of four, and the Tories were languishing in fifth place, with just three seats.

Play Video 9:25 'Immigrants make Britain great': Magid Magid's unlikely journey to Green MEP – video

The results so far show that the hard Brexit vote totalled 36.8% – with the Brexit party on 33.3% and Ukip on 3.5%. The overall total for pro-leave parties was up at 45.6% including the Conservatives on a historically low 8.8%.

European election latest results 2019: across the UK Read more

The pro-remain vote added up to 41.5% – with the Lib Dems on 20.9%, the Greens on 12.5%, the SNP on 3.5%, Change UK on 3.6% and Plaid Cymru on 1%. Labour, which tried to appeal to both sides with a soft Brexit pitch or a possible confirmatory referendum, was on 14.6%.

The punishing result for Labour is likely to prompt Corbyn’s critics to question again the Brexit strategy he and his advisers have pursued, with recriminations likely to spill into the open in the coming days.

The shadow foreign secretary, Emily Thornberry, said Labour must radically readdress its Brexit policy and campaign for a second referendum, and shortly afterwards Corbyn vowed the party would reflect on the results.

The Labour leader said: “After three years of Tory failure to deliver a Brexit that works for the whole country, these elections became a proxy second referendum. Over the coming days we will have conversations across our party and movement, and reflect on these results on both sides of the Brexit divide.”

A triumphant Farage, re-elected to the European parliament in the south-east, said his party had achieved a historic result. The Brexit party won all of the regions in England, apart from London, as well as Wales, and took almost 2 million more votes than their closest rivals, the Lib Dems.

“Never before in British politics has a new party launched just six weeks ago topped the polls in a national election,” Farage said.

“There’s a huge message here, massive message here. The Labour and Conservative parties could learn a big lesson from tonight, though I don’t suppose that they actually will.”

Play Video 0:43 Nigel Farage demands Brexit party seats at the negotiating table - video

Among those elected to the European parliament for the Brexit party were the former Conservative MP Ann Widdecombe and Annunziata Rees-Mogg, the sister of the Tory hard Brexiter Jacob Rees-Mogg.

The Brexit party chair Richard Tice, who won a seat in the east of England, echoed Farage’s demand that his party be included in any forthcoming Brexit negotiations with the EU. He said voters were “sending a clear message back to Westminster that we want a WTO [World Trade Organization] Brexit”.

“What’s so important now is we come in and help the government quickly because it is in paralysis and is navel-gazing about who should be the next leader,” he said.

Among the Tory casualties was the party’s European parliament leader Ashley Fox. The prominent Tory Brexiter MEP Daniel Hannan, one of the key figures in Vote Leave, was narrowly re-elected but said he was concerned the Brexit party could take seats from the Conservatives in a general election.

In Hannan’s south-east region, the Brexit party came top in 27 of 31 areas, with three won by the Lib Dems and one by the Greens. Labour and the Conservatives failed to win a single area.

Hannan said it was “the worst result my party has suffered in its 185-year history” but that he could see why the party had taken a pummelling. “It’s been a very bad night and in way understandably; you don’t need to be any kind of expert on politics – people voted leave and we haven’t left,” he told Sky News.

At the opposite end of the Brexit divide, the Liberal Democrats dominated in London at the expense of Labour, winning areas such as Camden and Lambeth by almost 10,000 more votes than Labour, despite five-figure Labour majorities in most of those parliamentary seats.

Vince Cable’s party took three seats in the capital, with two each for Labour and the Brexit party and another for the Greens.

Cable said voters had backed its “clear, honest, unambiguous message” and delivered the party’s best European elections result, which means they have more MEPs than MPs in the Commons.

“There is a clear lesson for Labour in tonight’s results: get off the fence. In trying to please everybody they have pleased nobody,” Cable said.

Alastair Campbell, Tony Blair’s former press secretary, said he had voted Liberal Democrat. “I didn’t vote Labour for the first time in my life and it was a very, very strange feeling,” he told the BBC.

Thornberry was the first senior shadow cabinet minister to call for a radical overhaul of Labour’s Brexit policy.

“We went into an election where the most important issue was what was our view on leaving the EU, and we were not clear about it,” she said. “We were not clear on the one single thing that people wanted to hear.

“We should have said quite simply that any deal that comes out of this government should be put to a confirmatory referendum, and that remain should be on the ballot paper, and that Labour would campaign to remain. I think that we will after these elections need to look very carefully at why it is that we got this result.”

Recriminations also came from the trade unions. The TSSA general secretary, Manuel Cortes, said the party should adopt a policy on Europe that was more in tune with its pro-European members. “I have knocked on many doors during the campaign and it’s blindingly obvious that our clear-as-mud line on Brexit hurt us deeply,” he said.

Any handbrake turn on Labour’s Brexit policy is likely to meet with resistance in the shadow cabinet. The shadow Cabinet Office minister, Jon Trickett, a close ally of Corbyn, pointed to results in Labour areas where the Brexit party had gained votes, including his own seat.

“Those ‘important people’ who said Wakefield is now for remain need to show some humility; they were wrong,” he said, a veiled dig at his pro-remain neighbouring MP Mary Creagh. “I never doubted my constituents’ determination to have their voice heard and that is why I have stuck to my view to accept the original referendum.”

In Wales, Plaid Cymru took the lion’s share of the pro-remain vote but the Brexit party topped the poll, snatching a seat from the Conservatives and one from Ukip.

In Scotland, the Scottish National party won three of the six seats available, its best European result yet, with Labour failing to win a single seat after enduring its worst Scottish election since 1910. The Lib Dems won back a seat it had lost in 2014, and the Tories narrowly succeeded in holding onto its sole Scottish seat, with the Brexit party supplanting Ukip, coming in second overall.

The night also saw a significant boost for the Green party, a pattern seen across Europe. Molly Scott-Cato, the Green MEP re-elected in the south-west, said it was a “thrilling night” for greens across EU member states.

The results seemed to confirm the disintegration of Ukip, which lost all its seats, including that of its leader, Gerard Batten, in London. Change UK, the nascent remain party led by a breakaway group of Labour and Tory MPs, failed to make any breakthrough.

The far-right independent candidate Tommy Robinson was roundly defeated in the north-west, claiming he had faced a “near impossible task”. Robinson, whose real name is Stephen Yaxley-Lennon, was repeatedly targeted by protesters, including two who threw milkshakes over him.