There are grounds for hope that a customs union arrangement may be negotiable with Labour, Jeremy Hunt has said, as he warned that both main parties would continue to be punished by their core voters for as long as they failed to deliver Brexit.

In a Guardian interview, the foreign secretary said: “The ground for hope is that Labour has been clear that they want the benefit of a customs union arrangement – essentially frictionless trade, which is necessary for a Northern Ireland border and for our manufacturing supply chains – and we want that too. So if we can find a solution that brings the benefits of a customs union, that is something that could be a prize.”

His remarks suggest an agreement could limit the UK’s ability to strike trade deals on goods separately from the EU, but could allow it to make agreements on services.

Hunt said the local election results and trends in national polls gave an incentive to both parties to compromise since both were being dragged down equally by Brexit.

“It is actually in both parties’ interests to resolve this because we will both be punished equally hard by our core voters. If we go into an election not having delivered Brexit, the Conservatives will be punished because that is what we promised to deliver in government and Labour will be punished because that is what they obstructed. That’s why it is in both parties’ interest to be statesmanlike at this moment.”

Admitting that compromise was not in the DNA of either party, he said Labour would suffer just as much as the Conservatives by leaving the issue deadlocked.

“All parties that are committed to implementing the Brexit referendum result will have a day of reckoning – it is as simple as that. We promised to deliver that result by 28 March 2019 and if we fail, people are going to be very angry,” he said.

Hunt, a former remainer who is now willing to countenance a no-deal Brexit, said the electorate was angry and bored. “This debate has been going on for three years now. They want it resolved. They want to move on, so anyone thinking of delaying Brexit further needs to remember [that] one of the core reasons why the people of Britain decided in large numbers to give a punch in the face to the British establishment – which was overwhelmingly remain – is because they thought we were not listening to them.

“If, three years on from that vote, we still fail to deliver Brexit, then that anger will only amplify.”

Hunt made clear his reference to a customs union arrangement, one of the main points of discussion in the prolonged private talks between Labour and the Conservatives, did not mean he was willing to support a common external tariff between the UK and the EU.

He said: “I struggle to see how you could have an independent trade policy with a common external tariff, but let us look at what is proposed.”

Asked whether voters in the 2016 referendum had really given any mandate to the government on a customs union, Hunt said: “In the long run I am not convinced that it is sustainable for an economy of the size of the UK to have its trade policy decided by a different trade bloc over which it has no influence. I don’t know of an example of that anywhere else in the world.

“How could the EU be expected to stand up for the rights of Scotch whisky distillers, for example? The model once offered by the EU is subscription to the common external tariff with no say in future negotiations, and I do not think that will work.”

Labour has not yet said whether it is willing to back a customs union deal that applies only to goods, and even the more pro-leave voices around Jeremy Corbyn recognise that being seen as a handmaiden to Brexit poses a serious risk to Labour’s election chances in remain-voting constituencies.

Hunt said the harsh reality was that the government’s lack of a majority left it with only two options: either trying to win over Labour or going back to the DUP and the small group of Conservative hard Brexiters opposed to the deal.

He has repeatedly said the removal of Theresa May as leader at this point would serve no purpose since the arithmetic in the Commons would remain unchanged.