Iain Duncan Smith effectively accused David Cameron of having a “low opinion of the British people” on Sunday as attempts to keep Tory splits over Europe civil appeared to be unravelling.

In remarks clearly aimed at the Prime Minister, the Work and Pensions Secretary hit out at those who warned of the economic impacts of a vote to leave the EU for “pessimistically downsizing” Britain’s role in the world.

Meanwhile George Osborne slapped down Boris Johnson for comparing himself to a man on a mission to rescue Britain from Brussels “baddies”.

“This isn’t some amusing adventure into the unknown – it’s deadly serious,” he said.

The Foreign Secretary, Phillip Hammond, also called the Eurosceptic MP Sir Bill Cash “a total shit” for releasing a Brussels legal report on the EU deal to the media despite an understanding that it was not to be published.

In an interview with the BBC Mr Duncan Smith made little effort to disguise his attack on Mr Cameron, who has made the fear factor a key part of the early campaign.

Lord Howard on David Cameron

“I have never heard such a lot of pessimistic downsizing of Britain’s future outside of the EU,” he said.

“Why would we have such a low opinion of the British people that we go out and talk about leaping into the dark, we talk about profound shocks, we talk about them not being capable, we’re too small.”

David Cameron in Ahoghill, Co Antrim (PA)

He then added pointedly: “I do not know why anybody would want to run a country like this.”

Tory Cabinet members in favour of Leave have become increasingly irritated with Downing Street’s tactics in the referendum. They believe Mr Cameron broke his own ‘no personal attacks’ rule when he obliquely suggested that Boris Johnson might be supporting the leave campaign because of personal ambition rather than long held beliefs.

They are also concerned that Downing Street tactics of warning of the threats posed by Brexit are beginning to resonate with voters and are attempting to counter them.

What has the EU ever done for us? Show all 7 1 /7 What has the EU ever done for us? What has the EU ever done for us? 1. It gives you freedom to live, work and retire anywhere in Europe As a member of the EU, UK citizens benefit from freedom of movement across the continent. Considered one of the so-called four pillars of the European Union, this freedom allows all EU citizens to live, work and travel in other member states. What has the EU ever done for us? 2. It sustains millions of jobs A report by the Centre for Economics and Business Research, released in October 2015, suggested 3.1 million British jobs were linked to the UK’s exports to the EU. What has the EU ever done for us? 3. Your holiday is much easier - and safer Freedom to travel is one of the most exercised benefits of EU membership, with Britons having made 31 million visits to the EU in 2014 alone. But a lot of the benefits of being an EU citizen are either taken for granted or go unnoticed. What has the EU ever done for us? 4. It means you're less likely to get ripped off Consumer protection is a key benefit of the EU’s single market, and ensures members of the British public receive equal consumer rights when shopping anywhere in Europe. What has the EU ever done for us? 5. It offers greater protection from terrorists, paedophiles, people traffickers and cyber-crime Another example of a lesser-known advantage of EU membership is the benefit of cross-country coordination and cooperation in the fight against crime. What has the EU ever done for us? 6. Our businesses depend on it According to 71% of all members of the Confederation of British Influence (CBI), and 67 per cent of small and medium-sized enterprises (SMEs), the EU has had an overall positive impact on their business. What has the EU ever done for us? 7. We have greater influence Robin Niblett, Director of think-tank Chatham House, stated in a report published last year: “For a mid-sized country like the UK, which will never again be economically dominant either globally or regionally, and whose diplomatic and military resources are declining in relative terms, being a major player in a strong regional institution can offer a critical lever for international influence.

Mr Duncan Smith appeared to be responding to an article written by Mr Cameron at the weekend accusing the Leave campaign of not have a proper economic plans for Britain’s future outside the EU.

Mr Cameron challenged them to set out how long the economy would face uncertainty, how joint-security arrangements would be replaced and how Britain’s role and influence in the world would be maintained.

“With so many gaps in the ‘out’ case, the decision is clearly one between the great unknown and a greater Britain,” he said. “A vote to leave is the gamble of the century. And it would be our children’s futures on the table if we were to roll the dice.”

In contrast Mr Duncan Smith said the UK market was so important to other EU businesses that other member states would “have a real requirement to trade with us”.

“The Prime Minister says there is a challenge to know what life outside looks like. My answer to this challenge is very simple: we do a deal with the European Union.

“That is a trade deal, about access to our markets, access to theirs. Part of our red lines will be about us being able to control the borders as we want. We want migration but we want it as controlled migration so we can cope with it. That deal is very doable.”

Addressing the current restrictions on pro-Brexit ministers’ access to Government EU referendum material, Mr Duncan Smith said this couldn’t “possibly apply in the sense of us not knowing what is going on in the department because we are responsible for the department”.

“For example, I will have to work on these proposals which came back from the recent negotiations,” he added.

The Cabinet Secretary, Sir Jeremy Heywood, is due to appear before the Public Administration and Constitutional Affairs Committee tomorrow to be pressed over critics’ allegations that it breaches neutrality rules.Mr Cameron has insisted that while he has given ministers the right to campaign on the opposing side in a personal capacity, the Government has a formal position and the Civil Service must work to deliver that policy.

In a further sign of the tensions between Tory inners and outers the Commons leader Chris Grayling warned Mr Cameron against a “purge” after the referendum.

Mr Grayling said there should be “no recriminations” from either side amid reports that some Tory MPs may try to oust the PM even if the UK votes to remain in the EU.