Up to half a dozen members of the cabinet believe Britain should leave the EU if David Cameron fails to deliver a major reform of Britain’s membership terms, according to senior Tory sources.

As the prime minister prepares to outline his plans to reform the free movement of people within the EU in a long-awaited speech, the Tory sources indicated a growing number of ministers are prepared to speak out.

Oliver Letwin, the Cabinet Office minister, became the latest senior minister to say Britain should be prepared to leave if the negotiations fail following the claim by Sajid Javid, the culture secretary, that Britain has nothing to fear outside the EU. Other ministers prepared to countenance an exit include Iain Duncan Smith, the work and pensions secretary, Chris Grayling, the justice secretary and Theresa Villiers, the Northern Ireland secretary. Michael Gove, the chief whip, was reported by the Mail on Sunday as saying Britain should be prepared to leave unless the EU agrees to restore British sovereignty.

The claims about growing euroscepticism in the cabinet came as Owen Paterson, the former environment secretary who is one of Duncan Smith’s closest allies, called on the prime minister to embark on a British withdrawal from the EU if he wins the election. Paterson said the prime minister should invoke article 50 of the Lisbon treaty which would trigger two years of negotiations on a British exit.

Paterson said the prime minister should hold his referendum as planned in 2017. But voters should be asked to endorse a new UK relationship with Brussels outside the EU rather than being asked whether Britain should remain in or leave the EU. Under Paterson’s plan Britain would follow the example of Norway which is a member of the four-strong European Free Trade Association (Efta) group. Its members are outside the EU but three – Norway, Iceland and Liechtenstein – participate in the EU’s internal market through the European Economic Area (EEA).

In a speech to the Business for Britain group, which is expected to become the main No group in any referendum, Paterson said: “We can leave the political project and enter into a truly economic project with Europe via the European Free Trade Association and the EEA . We would still enjoy the trading benefits of the EU, without the huge cost of the political baggage.

“We need to pick a proven, off-the-shelf plan. However, our participation in the single market is fundamental to protecting the UK’s economic position. This brings us to the only realistic option, which is to stay within the EEA agreement. The EEA is tailor made for this purpose and can be adopted by joining Efta first. This becomes the ‘Norway option’. We have already seen that Norway has more influence in international decision-making than we do as an EU member state. Using the EEA ensures full access to the single market and provides immediate cover for leaving the political arrangements of the EU.”

The changes would allow Britain to gain greater control of its borders because Britain would also leave the European Convention of Human Rights (ECHR). Paterson said: “Outside the EU and freed from the writ of the ECHR, ‘freedom of movement’ within the EEA could be limited to free movement of workers, without having to accept dependents and members of their extended families. This is exactly what David Cameron wants when he said last year that he thought free movement within the EU ‘needed to be returned to the original concept, which was the freedom to be able to go and work in another country’. But, if we are to benefit from the single market, we must at least accept that provision.”

Paterson said his idea to trigger a British exit under article 50 of the Lisbon treaty was better than the prime minister’s plan to renegotiate Britain’s membership terms through a treaty revision. Britain would retain a veto in any treaty revision but would lose its veto the moment it triggers article 50.

The former minister told the Guardian: “For a treaty revision you have to have a convention and then an inter-governmental conference [IGC of all EU leaders]. This [article 50] gives you two years – bang – and a clear process.”