Senior Conservative Brexiters have set new red lines for Theresa May in negotiations with the EU, significantly reducing her chances of getting a deal acceptable to the whole of her party.

The former cabinet ministers Iain Duncan Smith, Owen Paterson, Nigel Lawson and John Redwood have said it would be unacceptable for the European court of justice to have any jurisdiction over the UK during the planned two-year transition after Brexit.

Duncan Smith set out his objections in an article in the Sunday Telegraph in which he also said he particularly opposed a plan for the ECJ to have an ongoing role, beyond the transition, adjudicating on the rights of EU nationals in the UK.

Paterson, Lawson and Redwood, along with more than 30 other Brexiters, have signed a letter coordinated by the group Leave Means Leave listing seven conditions they say should apply before the UK makes any divorce payment to the EU.

May and Juncker. The two will meet on Monday for further talks on Brexit. Photograph: Yves Herman/Reuters

On Monday, May will meet Jean-Claude Juncker, the European commission president, in Brussels to firm up what the UK is offering on money, EU citizens’ rights and the Irish border issue in return for agreement at a summit this month to open talks on a future trade deal.

Tory MPs have broadly accepted May’s decision to offer the EU a “Brexit bill settlement” in which the UK would pay in excess of £40bn, but now the role of the ECJ has become a flashpoint with the potential to scupper a deal.

In his Telegraph article, Duncan Smith wrote: “During the implementation phase, the idea that the UK, once it had repealed the 1972 European Communities Act putting it outside the EU, should calmly submit itself to the rulings of the ECJ would be unworkable.”

Q&A What is a hard Brexit? Show A hard Brexit would take Britain out of the EU’s single market and customs union and ends its obligations to respect the four freedoms, make big EU budget payments and accept the jurisdiction of the ECJ: what Brexiters mean by “taking back control” of Britain’s borders, laws and money. It would mean a return of trade tariffs, depending on what (if any) FTA was agreed. See our full Brexit phrasebook.

The Leave Means Leave letter sets out as one of its seven red lines that the ECJ should “cease to have any jurisdiction whatsoever over the UK from 30 March 2019”.

Yet EU leaders have repeatedly said that if the UK wants to carry on as a member of the single market and the customs union during the planned two-year transition, as it says it does, it will have to continue to accept ECJ jurisdiction during that period.

And in a speech in Berlin last month, David Davis, the Brexit secretary, explicitly accepted that. He said the transition would involve “keeping both the rights of a European Union member and the obligations of one, such as the role of the European court of justice”.

Separately, the EU is pushing for the ECJ to have a role guaranteeing the rights of EU nationals in the UK long after Brexit. Ministers are reportedly considering a plan that would involve certain cases being voluntarily referred to the ECJ, but Duncan Smith said this would involve “ceding power to a foreign court on which it has no representation”.

The towers of the ECJ in Luxembourg. The court’s proposed role in the Brexit transition is a thorn in Brexiters’ sides. Photograph: Francois Lenoir/Reuters

The Leave Means Leave letter includes at least three other demands relating to the transition that are incompatible with what EU leaders are demanding. It says the UK must be able to implement as well as sign free trade deals during the transition (the government wants to be able to sign free trade deals during this phase, but accepts implementation would have to wait until the transition is over); free movement must end; and the UK must be exempt from new EU regulations.

Another Leave Means Leave demand is that any payments to the EU must be conditional on the EU offering the UK a free trade deal. In his article, Duncan Smith went further, saying it must be a free trade deal “without tariffs or non-tariff barriers”.

On this point, ministers agree with the Brexiters, although the EU says it does not accept a link between the UK’s financial settlement and a future trade deal.

Quick Guide What is the EU withdrawal bill? Show What is the EU withdrawal bill? The EU Withdrawal Bill – once known as the Great Repeal Bill – is going through the House of Commons to repeal the 1972 European Communities Act and transpose all existing EU legislation into domestic UK law, which will avoid a 'cliff-edge' change on the day after we leave the EU. Parts of the bill have been highly controversial, and MPs have tabled hundreds of amendments to try and change its wording, including a significant number of Conservative rebels. Some of the key controversies include its use of so-called Henry VIII powers, which will give government ministers the power to tweak the wording of laws to make sure they make sense in UK legislation - but those changes could take place without having to go through parliament. MPs have called this a "power grab" by the government. The government estimates around 800 to 1,000 measures called statutory instruments will be required to make sure the bill is applied correctly. Other concerns include the government's decision not to include the EU charter of fundamental rights in the law being transposed. Other amendments are attempts to affect the Brexit process, including legislating for a transitional period and giving MPs a binding meaningful vote on the deal secured by Theresa May, before the deal is finalised.

The other two Leave Means Leave conditions are more acceptable to the EU. The letter says the transition period should last no longer than two years, and that there should be a deal on the Irish border.

Other signatories to the Leave Means Leave letter include David Jones, a former Brexit minister, the Tory backbencher Jacob Rees-Mogg, the former Tory minister Sir Gerald Howarth, the Labour MP Graham Stringer and the former British Chambers of Commerce director general John Longworth.

The letter concludes: “If the EU refuses to agree to these terms by the end of the December council, the UK – having exhausted every avenue – should suspend its participation in the negotiations and inform the EU that, unless they are prepared to talk to us seriously about a future free trade arrangement, we will revert to World Trade Organisation terms from 30 March 2019.”