David Cameron has been told by his cabinet colleague Iain Duncan Smith to scrap his planned shake-up of MPs' constituency boundaries or face a fresh rebellion from Tory backbenchers.

The prime minister met groups of his MPs last week after 81 of his colleagues rebelled against the three-line whip ordering them to vote against a Commons motion in support of a referendum on Britain's future in the EU. But Duncan Smith, the secretary of state for work and pensions, has warned Cameron that the party is teetering on the brink of further mutinies.

According to one Whitehall source, Duncan Smith told Cameron he needed to "kick the whole idea of boundary change into the long grass to avoid further instability".

The Observer understands that Duncan Smith was moved to act after a number of Tory MPs in the Commons tea rooms openly discussed quitting parliament in the new year unless Cameron acted to delay the proposed shake-up of parliamentary boundaries, designed to let the Government cut the number of MPs from 650 to 600.

A Whitehall source said: "Duncan Smith is leading the rearguard action on this one because he knows the instability and upset that it is causing everyone. There has been talk that no one in a seat will lose out and mention of the House of Lords, but that doesn't cut it with many who have left business for this job and quite frankly are thinking of going back to making some money."

A unit inside Conservative CHQ, working under the party's boundary review manager Roger Pratt and former MP Rob Hayward, have been advising those who may lose out in the changes. It is understood they believe that while 20 Labour seats could go, as many as 13 Conservative seats could also be lost in the reforms. Many MPs, however, fear this underestimates the scale of the chaos that will be caused. One senior backbencher said: "The assessment they are offering just looks wrong. They think we will win in the newly formed constituency of West Hull. Hell will freeze over before the Conservatives win in West Hull. It'll be third at best for us there.

"This really is causing lots of grief, especially among the new intake whose careers are suddenly in jeopardy. I have heard more than one MP talk of quitting."