The health secretary, Matt Hancock, has hinted that the government may be prepared to reach a compromise with Labour on a post-Brexit customs union arrangement with the EU.

He said the local election results on Thursday were a call from voters to “deliver Brexit and then move on”.

The Conservatives lost more than 1,300 seats in Thursday’s elections, their biggest setback since John Major was prime minister. Labour had been expected to make gains, but instead suffered a net loss and also lost control over a string of councils. The remain-supporting Liberal Democrats were the major beneficiaries, taking control of 10 councils.

Hancock said he shared the frustration of voters who were keen for politicians to move on to other pressing issues. “We need to be listening to these results from these local elections which are about ‘deliver Brexit’, not ‘deliver this particular form of Brexit’,” he told Radio 4’s Today programme.

The Conservatives’ poor results have led some MPs to call for a leadership change. The former Brexit secretary David Davis declared his support for Dominic Raab, saying the party needed a leader who could deliver a Brexit “faithful to the demands of the referendum” and present “an optimistic and authentic vision for the future”.

Writing in the Daily Mail, Davis said Raab was “the best-placed Brexit candidate to win the necessary support among MPs and party members and, above all, broaden our appeal to voters”.

He said the European elections on 23 May could be “far worse” for the party than the local elections.

“Delivering on Brexit is a necessary condition for seeing off the threat of a hard-left Labour government under Jeremy Corbyn,” he wrote. “But on its own it won’t be enough. As Conservatives, we must now also offer a more compelling and inspiring vision for addressing the wider challenges the country faces beyond Brexit.”

The justice secretary, David Gauke, described the local election results as punishment for the Conservatives’ handling of Brexit. He told BBC Breakfast: “These are very disappointing election results. What we need to be doing is addressing the big issue in front of us, which is Brexit.”

He said the Tories would have had much better results if they had managed to get Theresa May’s deal through parliament. “I think we can look at those local election results as a punishment for both the Labour party and the Conservative party for failing to find a way through that situation,” he said.

Lisa Nandy, the Labour MP for Wigan, said the poor results for both main parties were a result of “people losing faith with the system as a whole”.

She told Today: “People are really, really frustrated about Brexit, but the major frustration comes from the perceived inability of the two major parties – including Labour – to get our act together and start dealing with the very many and real problems people have got.”