Nick Clegg is ratcheting up the pressure on David Cameron to persuade scores of Tory rebels to support House of Lords reform by saying that a crucial vote on Tuesday is a test of the prime minister's leadership.

As a cross-party group of MPs and peers urged the Commons to support the deputy prime minister's reform plans, Liberal Democrat sources warned the prime minister that the coalition will enter "uncharted territory" if the vote is lost.

"A defeat would be unprecedented because the coalition has not been defeated before," a senior Lib Dem source said. "It would take us into uncharted territory.

"Each time Nick has been asked to deliver a vote he has delivered it, including in difficult areas like the NHS and welfare reforms. This is a test of David Cameron's leadership."

The Lords reform bill will receive its second reading on Tuesday night because Ed Miliband, who favours reform, has instructed his MPs to support it at this stage. The bill would introduce a mainly elected second chamber, 80% of whose members would be elected on a regional list by 2025.

The deputy prime minister, who will open a two-day debate on Monday afternoon, fears the bill may run into serious trouble in a second vote on Tuesday night when up to 110 Tory rebels join forces with Labour to vote against a "programme motion". The Lib Dems believe that without such a motion, which sets a timetable for the bill throughout its Commons stages, the measure could be killed off because it would clog up the government's parliamentary schedule.

Richard Reeves, Clegg's former strategy adviser, warned last week that the chances of the bill becoming law would be "vanishingly small" if the timetable motion is rejected.

The Tory rebels are adamant that they have the numbers to defeat the programme motion. This has persuaded the Lib Dems that the fate of the bill, seen as the best chance to introduce House of Lords reform in a century, lies in the hands of Labour.

Reformers are given a boost today when Peter Hain, Labour's veteran campaigner, says that the bill offers a "now or maybe never" chance of change. In a foreword to a cross-party report by members of the joint committee of MPs and peers, which examined the bill in draft reform, Hain writes: "The Labour case for House of Lords reform goes back to when over 100 years our first leader Keir Hardie demanded a democratically elected second chamber. Whatever amendments on detail that will likely be needed to the bill, it is now or maybe never for Lords reform."

But Hain said he supports Miliband's decision to vote against the programme motion because he regards that as a good opportunity to block the coalition's "rightwing" legislative agenda.

"I am very comfortable with [voting against the programme motion]," Hain told the Guardian. "Within the rest of the legislative programme are loads of rightwing bills which will damage people in Britain. So I don't think it is any part of our responsibility to try and get those bills into statute. So I will happily vote against the programme motion but I will vote for the second reading and I will support the bill thereafter, though I will back some amendments."

Stephen Dorrell, the former Tory cabinet minister, makes an appeal to party rebels to support the bill on the grounds that it will tackle what the late Lord Hailsham, the former lord chancellor, described as the tendency of Britain's parliamentary system to produce an "elective dictatorship". Dorrell writes in a foreword to the report: "Those Conservatives who take a sceptical view of legislation as an instrument of human perfectability, and who believe that the executive should be subject to more effective scrutiny, should welcome steps to strengthen the voice of parliament. The key issue in the debate over reform is not the balance between lords and commons; it is the balance between parliament as a whole and the executive. That is why, along with many Conservatives, I am a longstanding supporter of an elected second chamber and welcome this pamphlet as a contribution to that debate."

The report is supported by some members of the 2010 intake of Conservative MPs, showing that the rebels do not have a complete hold over the party. The pro-reform new Tory MPs are Gavin Barwell, Laura Sandys, Daniel Poulter and John Stevenson."

The Lib Dems turned their fire on the Tory rebels who accused Clegg of treachery after Clegg's adviser Richard Reeves warned that the party might block Cameron's plans to reduce the size of the commons if the bill falls. One party source said: "There are hysterical cries of betrayal and treachery from serial rebels on the Tory backbenches. All we are asking them to do is support something that was in the last three Conservative general election manifestos, was in the coalition agreement, was in the Queen's speech and was agreed by the cabinet. To be accused of treachery by backbench Tory rebels is like being accused of being over-theatrical by Mario Balotelli."

Vince Cable denied that the Lib Dems had issued threats to scupper Cameron's plans to reduce the size of the House of Commons. He told the Andrew Marr Show on BBC1: "We are not threatening, we are working in a businesslike way with our Conservative colleagues."

Asked whether there was a link between Lords reform and reducing the size of the Commons, he said: "They have got to be judged on their individual merits. We will vote on those issues on their individual merits when the time comes."