David Cameron's ability to keep his party united behind the coalition with the Liberal Democrats was called into question as nine former Conservative cabinet ministers urged Tory MPs to rise up and defy him over House of Lords reform.

The joint move by Tory grandees – including two former chancellors, Geoffrey Howe and Norman Lamont – in a letter to MPs represents a direct challenge to the prime minister's authority from some of the most senior figures in the party, in advance of a crucial Commons vote on Tuesday.

Cameron, torn between loyalty to his MPs and loyalty to the coalition, is already facing a revolt by between 50 and 100 Tory backbenchers in the vote on the coalition's plan to create a largely elected House of Lords, whose members would be chosen for non-renewable 15-year terms. Despite Cameron ordering a three-line whip, the Tory rebels were confident that – with the help of Labour votes – they could muster 50 or more to inflict the first defeat on the government, on its own business, since the coalition was formed.

The Lords reform plans are seen by Nick Clegg's Lib Dems as central to their purpose in government and constitutional reform is at the heart of their programme. Senior Lib Dem sources threw down a challenge to Cameron, saying his ability to deliver his MPs in sufficient numbers would amount to "a test of David Cameron's leadership".

Many Tory MPs and peers, however, see Lords reform as a sop too far to the Lib Dems, one that will disturb the delicate constitutional balance between the Commons and the Lords, bring no obvious benefit to the way the country is run, and take up huge amounts of parliamentary time before it could reach the statute book. In an attempt to thwart the plans and bolster rebel numbers in the Commons, a cross-party group of 36 peers, which includes the former Commons speaker Betty Boothroyd, has written a letter to MPs arguing that the current House of Lords is "a vast reservoir of talent and experience, which complements the more youthful and vigorous House of Commons without ever being able to threaten it".

Pointing to what they say would be the serious consequences from the reforms, they insist that the election of members of the upper house would blur the age-old distinction between Lords and Commons and "remove the unambiguous democratic mandate the House of Commons currently enjoys".

The other former Tory cabinet ministers who signed the letter are the former Northern Ireland secretary and party chairman Peter Brooke, the former Scottish secretary Michael Forsyth, the former agriculture minister Michael Jopling, the former transport secretary John MacGregor, the former attorney general Patrick Mayhew, the former Scottish secretary Ian Lang and former environment, industry and social security secretary Patrick Jenkin. Another signatory is the former Liberal leader David Steel.

Tensions within the coalition parties over Lords reform reached new heights last week when a senior adviser to Clegg warned that the Lib Dems could scupper plans to redraw constituency boundaries – a move that would benefit the Tories at the next election – unless the changes are agreed. Some Tories say the suggestion has hardened their attitude against the Lib Dems and Lords reform rather than persuaded them to think again.

Two votes will take place on Tuesday evening: the first to give the Lords reform bill a second reading. This looks certain to be passed because Labour, which backs an elected Lords, will support it. But Labour will join Tory rebels in opposing a motion that would place a time limit on debate of the bill. If this is defeated, it would mean the bill would be open to filibustering by MPs determined to prevent its passage.

Tory MPs insisted that they had the resolve to talk the bill out even if that meant using delaying tactics well into next year. Bernard Jenkin, the Tory chairman of the public administration select committee, said that opposition in the party to Lords reform was growing. "As it dawns on people that this massive change is being made to the constitution as a result of a political deal, rather than because of its own merits, people find it harder and harder to support."

A defeat on the so-called "programme motion" would be a huge blow to Cameron's authority on a bill that was a centrepiece of the Queen's speech and which he has said he will deliver for the Liberal Democrats.

As the battle lines were being drawn for Tuesday's vote, a cross-party group of supporters of Lords reform, including former Tory health secretary Stephen Dorrell and the former Labour Welsh secretary Peter Hain, issued a pamphlet making the case that modernisation of the Lords is long overdue and would assist the functioning of democracy. "Parliament should agree reform now and at last bring closure to a debate which has raged for more than a century," they say. The pamphlet was compiled with the backing of several young Tory MPs, including members of the joint committee of both Houses that drew up the reform blueprint.

Senior Lib Dems insisted that Lords reform was a key part of the coalition agreement and hinted that, if the Tories did not back it, they might be less willing to hold to their side of the bargain.

"The coalition agreement is not some kind of a la carte menu. It is not about having a pick and mix approach. You cannot have the parts you like and reject the parts you don't. A deal is a deal and it up to them deliver that deal," said a senior party figure.