The Liberal Democrats may be able to use their disproportionate power in the Lords to ensure that 16 and 17-year-olds get to vote in the EU referendum, senior party figures have said.

A Labour amendment to the European Union referendum bill, which would have lowered the voting age to 16 in David Cameron’s promised poll on Britain’s membership of the EU, was voted down in the Commons by 310 votes to 265 on Thursday.

Despite losing 48 of their 56 MPs at the last election, leaving them as the fourth biggest party in the Commons with only eight MPs, the Lib Dems have 101 members in the Lords. The Conservative party has 228 members, Labour has 212 and there are 178 cross benchers.

Speaking ahead of the Commons vote, the Lib Dem chief whip and one of the party’s eight surviving MPs, Tom Brake, said lowering the voting age could be a “crunch issue” in the second chamber.

“When the Lords are pursuing issues, they will look at issues in the Commons where a head of steam has built up,” he said.

Lord Jim Wallace, the Lib Dem leader in the Lords, said: “As far as we are concerned [votes for 16 and 17-year-olds] is party policy.

“The Labour party seem to have also come on board with this and said they would support it too, so there’s a fair chance that it could actually carry in the Lords.”

It is the first time in modern political history that a Conservative government has not also dominated the second chamber. The House of Lords Act 1999 removed the majority of hereditary peers, many of whom were Conservative supporters.

Lord Wallace insisted that the party’s peers would reserve the right to say no to elements of government legislation, though he acknowledged that the function of the Lords was to revise legislation, rather than to stop it outright. “If we retain the right to say no, it actually brings the government to the negotiating table,” he said.

Those 40 years younger than me are going to have to live with the consequences of this. Why should they be denied? Baroness Smith of Basildon

Wallace described the Salisbury Convention, which dictates that the Lords should not vote down a governing party’s manifesto promises, as out of date. The convention was established in the aftermath of Labour’s landslide victory in 1945, when the party only had 16 peers.

Lord Richard Newby, the party’s chief whip in the Lords, agreed with Wallace’s comments. “People say you can’t say no, but it’s just rubbish. It’s just because they don’t want us to say no,” he said. “Unless [the government is] prepared to face concessions, they will – as every government does – face defeat.”

The Lib Dems in the Lords are likely to use their power to frustrate the government’s attempts to repeal the Human Rights Act, and make significant amendments to the proposed Psychoactive Substances bill and the proposals to extend the Right to Buy scheme to housing association homes.

Labour’s leader in the Lords, Baroness Smith of Basildon, said that if Cameron insisted on resisting a change to the bill to allow votes for 16- and 17-year-olds, it was “almost certain” that he would lose the vote in the Lords.

“Those who are 40 years younger than me are going to have to live with the consequences of this referendum for much longer than I will,” she said. “Why should they be denied a say?”