Image copyright Getty Images Image caption The Liberal Democrats have a strong representation in the House of Lords

It was part-way through the Queen's Speech, just after she mentioned the EU referendum, so you might have missed it - but when she promised that "my ministers will uphold the sovereignty of Parliament and the primacy of the House of Commons", she was signalling a retreat.

Just a few weeks ago, arguing that the primacy of the Commons was being undermined by the House of Lords, ministers were considering curbing the power of peers.

A new law would stop the Lords from blocking what is called secondary legislation. When the Queen's Speech was published on Wednesday, however, there was no bill.

During the parliamentary session just ended, the Lords inflicted 60 defeats. Of those, the most humiliating was over tax credits.

Here the government had argued the Lords was breaking a convention - an unwritten rule - that secondary legislation isn't rejected, because Parliament has already scrutinised and approved the principle behind it by passing the primary legislation.

Peers denied there was such a convention, pointing out they had thrown out secondary legislation on five previous occasions.

Lord Strathclyde, a former Leader of the Lords, was asked by the prime minister what should be done, and he recommended legislation to stop it.

Government to review Lords powers after tax credits defeat

Limiting Lords powers 'calls its future into question'

It isn't just Conservatives who are unhappy with the Lords.

"We're defeating the government more often than is good for us," a Labour peer told me recently - but isn't that the job of the opposition?

House of Lords 807 Peers 247 Conservative

211 Labour

175 Crossbench

109 Liberal Democrat Getty

Yet it is not so simple as that if you expect your party to be in power again one day. Then, you worry about the House of Lords doing the same to your legislation.

My Labour source blames the Liberal Democrats, as well as David Cameron for allowing his coalition partner to nominate so many new peers in the last Parliament.

It is the Lib Dems, he says, who insist on inflicting so many defeats. Labour peers don't like it, but fear if they don't go along with it, Lib Dems will say they have become the only real opposition to the Tories.

So what are the Lib Dems up to? Lord Newby, the chief whip, cheerfully admits he is up for a fight.

House of Commons 650 Members of Parliament 330 Conservative

229 Labour

54 SNP

8 Liberal Democrat PA

"There's no point in us existing," he told me, "if we don't vote for the things we believe in."

That's another thing, says my Labour contact: they hardly exist at all in the Commons, most of their MPs were rejected by the voters, yet in the Lords it is as if the general election never happened.

In the Commons, just eight of 650 MPs are Lib Dems; in the Lords, they have 109 peers out of 807, about half of the number of Labour, with the Conservatives on 247.

Image copyright Lord Newby Image caption Lord Newby denies claims the Lib Dems are acting undemocratically

Lord Newby denies it is undemocratic for his party to wield so much influence when the party was almost wiped out at the general election.

"The only suggestion I can make is get used to it or reform the House of Lords," he says.

Labour peers believe this is the real strategy, provoking a constitutional crisis; after all, the Lib Dems have no love for the Lords, and want a wholly elected second chamber in its place.

"It's true we don't have a vested interest in the mechanism as it is," says Lord Newby, though he denies wanting to precipitate reform by creating a crisis.

He thinks the real vested interest is that of the two biggest parties. He accuses Labour of backing a "spurious" Conservative claim that governments get their way in the Lords 70% of the time.

He even accuses them of deliberately engineering some unnecessary votes when there aren't enough opposition peers around, so the government wins, helping keep up that average of government successes.

Cameron's Commons problem

I suspect the real reason David Cameron backed off from a confrontation from the Lords actually lies elsewhere: in the House of Commons.

Earlier this month, MPs on the Public Administration and Constitutional Affairs Committee, said removing the Lords power to veto secondary legislation would be a mistake.

Image copyright Conservative Party Image caption Bernard Jenkin's committee report said the Lords was unbalanced

The committee, chaired by Bernard Jenkin, one of Mr Cameron's most trenchant Conservative critics, was clear.

"We think the issue of the excessive size and unbalanced composition of the Lords is a much more pressing issue, and it is wrong to consider the powers and functions of the House of Lords in isolation, on the basis of one wholly exceptional and highly political event," he said.

David Cameron's real problem isn't peers, Lib Dem or otherwise.

It is the increasing difficulty he has with Conservative MPs. On tax credits and making all secondary schools become academies, to give just two examples, it was Tory MPs who rebelled.

Loyalty, it used to be said, was the party's secret weapon. Today, a significant number of MPs no longer see it as a virtue.

As the measures announced in the Queen's Speech work their way through the parliamentary machine in the coming months, don't be misled by the number of defeats MPs inflict on the government.

It is the number of measures withdrawn before any vote is taken, or never proposed at all, which is the real measure of the government's problem.