Theresa May has come under attack from Tory colleagues after conceding that the UK may have to remain tied to EU rules and laws beyond December 2020 in an attempt to solve the Irish border issue.

The prime minister told fellow EU leaders on Wednesday evening that she was open to extending the transition period – a proposal mooted during intensive talks over the Irish backstop issue, which broke down without agreement last weekend.

But the idea was quickly rejected back at home. David Davis, the former Brexit secretary, said the idea was “unwise” and it was the wrong time to “take the pressure off” in the negotiations.

Five Conservative former cabinet ministers including Davis and Boris Johnson have signed a letter to May urging her to reject a Northern Ireland backstop and, crucially, an all-UK version.

The letter states: “Talk of either a UK or a Northern Irish backstop is inimical to our status as a sovereign nation state. Both are unnecessary: indeed they are a trap being set by the EU which it is vital we do not fall into.

“Using existing techniques and processes, with political cooperation, we can ensure that trade continues between the Republic of Ireland and Northern Ireland. The necessary procedures can all be implemented within the existing legal and operational frameworks of the EU and the UK.

“Rational and pragmatic approaches can ensure that trade across the border is maintained. There need be no threat to the Good Friday/Belfast agreement.”

The other three former cabinet ministers who have signed the letter are Priti Patel, Iain Duncan Smith and Owen Paterson. Jacob Rees-Mogg, the chair of the European Research Group of pro-Brexit Tory backbenchers, is also a signatory.

They say May should not “engage in a show of resistance and a choreographed argument followed by surrender and collapse into some version of the backstop and Chequers. Instead we urge you to say to the EU at the summit: ‘Let us agree that we need to reset our negotiations. Our objective is a free trade agreement that benefits the UK and EU and millions of our citizens.’”

Another former minister, Nick Boles, described any attempt to extend the transition period as a “desperate last move” and said May was losing the confidence of her party. He told BBC Radio 4’s Today programme on Thursday that the EU was demanding “humiliating concessions”.

Quick Guide Why extend the Brexit transition period? Show Will the proposal solve anything? The mooted extension to the transition period is a new idea being put forward by the EU to help Theresa May square the circle created by the written agreement last December and the draft withdrawal agreement in March.

That committed the UK and the EU to ensuring there was no divergence between Northern Ireland and the Republic of Ireland.

But it also, after an intervention by the Democratic Unionist party, committed the UK (not the EU) not to have any trading differences between Northern Ireland and Great Britain.

The problem is that these are two irreconcilable agreements. They also impinge on the legally binding Good Friday agreement, which brought peace to Northern Ireland and in some senses pooled sovereignty of Northern Ireland giving people a birthright to be Irish or British or both.

If the UK leaves the EU along with the customs union and the single market then the border in Ireland becomes the only land border between the UK and the EU forcing customs, tax and regulatory controls. The backstop is one of three options agreed by the EU and the UK in December and would only come into play if option A (overall agreement) or option B (a tailor-made solution) cannot be agreed by the end of transition. The Irish have likened it to an insurance policy. The new EU idea is to extend the transition period to allow time to get to option A or B.

But an extension is problematic for Brexiters and leave voters, who want the UK to get out of the EU as soon as possible.

The Irish and the EU will also still need the backstop in the withdrawal agreement, which must be signed before the business of the trade deal can get under way. Otherwise it is a no-deal Brexit.

Extending the transition into 2021 would mean another year of paying into the EU budget. Britain would have to negotiate this but it has been estimated at anywhere between £10bn and £17bn. Staying in the EU for another year would also mean continued freedom of movement and being under the European court of justice, which Brexiters would oppose.

“It’s a classic of negotiations that she keeps on thinking that one more concession is going to somehow [succeed], with one bound and she’s free, and she’s not going to be free, she’s getting ever more trapped,” he said.

“I’m afraid she is losing the confidence now of colleagues of all shades of opinion, people who have been supportive of her throughout this process.”

Leave Means Leave, the pro-Brexit campaign group, released a statement from its co-chair Richard Tice accusing May of wanting to remain in the EU. “The original transition was an unnecessary trap created by our weak civil servants who cannot be trusted as they don’t want us to leave. It should be cancelled, not extended. It is increasingly clear the PM doesn’t want to leave either,” he said.

Arriving at the European council in Brussels on Thursday morning, May said an extension of the transition period, by “a matter of months”, could be necessary.

She suggested it could allay concerns about the backstop proposal, in which Northern Ireland would be treated differently to the rest of the UK, by making it less likely that it would need to come into force.

The EU has proposed that Northern Ireland in effect stay in the customs union and single market after December 2020, at the end of the transition period, should there be no other way to avoid a hard border on the island of Ireland.

May said: “The original proposal from the EU was one we could not accept in the UK. It would have created a customs border down the Irish Sea. Earlier in the year we put forward a proposal as to how to deal with this issue; a further idea that has emerged, and it is an idea at this stage, is to create an option to extend the implementation period for a matter of months, and it would be for a matter of months.

“But the point is that this would not be expected to be used because we are working to ensure that we have that future relationship in place by December 2020.”

Back in London, David Lidington, the Cabinet Office minister who is in effect May’s deputy, indicated that the idea of extending the transition period by another year was one worth discussing as a way of “getting over the difficult final hurdles”.

But he acknowledged that staying in the EU’s single market and customs union during 2021 could have financial implications, in the form of further cash contributions made by the UK to Brussels’ coffers.

When asked whether it would cost the UK billions of pounds, Lidington said he did not accept that. “That would be one of the things that would be teased out in the negotiations … There may be other approaches we can take.”

Xavier Bettel, the prime minister of Luxembourg, said the cost to the UK taxpayer of extending the transition period would only be settled once it was clear how much longer the country would remain under EU laws.

He said: “[An extension] is one of the options. I cannot tell you if it is the only way. We are at the beginning of having news of a new idea. We don’t speak about the bill. The bill is at the end … We know that Theresa May is in a tricky situation. She has huge pressure at Westminster.”

Many in Brussels believe May has been hampered by the risk that she could lose key votes on the budget – and potentially then face a vote of no confidence – if the DUP believes she is about to sign up to a compromise on the Irish border that it could not accept.

Anand Menon, director of the thinktank UK in a Changing Europe, said it had long been evident that the future trading relationship could not be settled by the end of 2020, but it would be much better for May to extend the article 50 negotiating period.

“This debate is getting farcical: she keeps calling it an implementation phase but it’s blatantly obvious there’s nothing to implement,” he said. “Jacob Rees-Mogg is right, it is a vassal state thing – we’re going to be under these rules and it’s going to cost us loads of money. The obvious thing to do would be to extend the article 50 period – but politically you can’t call a spade a spade.”

A Labour spokesman said: “Labour has always said the transition should be as short as possible and as long as necessary. But the only reason the prime minister is talking about extending the transition is because she is no closer to having a credible plan for our future relationship with the EU after Brexit.”