The Labour shadow cabinet has decided to vote against a bill introducing reform to the voting system, raising the prospect of a Commons defeat for one of the governing coalition's flagship policies.

The decision, taken last night, followed two lengthy shadow cabinet discussions.

It could herald a backbench Tory-Labour alliance designed to derail the bill either at its second reading or by rejecting the proposal that the referendum be held on the same day as elections in Scotland, Wales and parts of England next May.

A total of 50 Tory MPs are opposing next May as the date for the referendum, and are coming under intense pressure from Tory whips to pull back from that stance.

The shadow cabinet had an earlier discussion at which – on the advice of John Denham, the shadow communities and local government secretary – it proposed abstaining on the bill.

But Denham, a strong advocate of voting reform, backed moves at yesterday's shadow cabinet, led by its justice spokesman, Jack Straw, to oppose the bill outright.

Straw argued that the bill introducing the referendum was being coupled unnecessarily with boundary changes which he described as gerrymandering.

The shadow cabinet agreed yesterday that it still supported the referendum the Alternative Vote (AV) system – but, in a new reasoned amendment, will say it is entirely wrong that this reform, on Conservative insistence, is being bound up with plans to reduce the number of MPs and introduce widespread boundary changes.

Labour claims the boundary reforms would benefit the Tories so much that the Labour party would find it impossible to win a general election again.

Denham's decision to oppose the coalition bill is significant because he is one of the most long standing supporters of AV in the Labour party.

Under the AV system, voters rank candidates in order of preference until one of them reaches 50% or more of first-choice votes and is elected.

Supporters claim the system will end the culture of safe seats and lazy MPs, but there is no agreement about which party will benefit most.

There is fury in Labour ranks that the previous process of independent public inquiries has been removed from the boundary review process.

Labour MPs also claim a government minister will be given powers in the bill to change boundary review recommendations without an appeal mechanism.

The coalition claims the period of consultation is being extended, but Tory MPs yesterday expressed strong concern that the pre-eminent requirement to equalise the size of constituencies would see old countryside boundaries, such as Somerset, ignored.

However, the coalition insists the boundary review – due to be completed before the next election – is necessary to cut the number of MPs to 600 and attempt to equalise the size of constituencies.

Mark Harper, the constitutional affairs minister, hit back at Straw's complaints, saying: "All this bluster simply highlights the fact that Labour MPs do not believe in seats of equal size and votes counting equally across the whole of the United Kingdom."

The potential parliamentary roadblock to a referendum stems from a deal the deputy prime minister, Nick Clegg, struck with David Cameron as part of the coalition talks.

The two agreed a referendum on AV would go ahead, but only on the condition that the boundary review was completed in time for the next election. Both proposals have been included in a single bill.

Many Labour and Tory MPs are angry that the referendum is set to be held on the same day as the Scottish, Welsh and English local elections.

A total of 45 disaffected Conservative MPs, led by Bernard Jenkin, John Redwood, Edward Leigh, Sir Malcolm Rifkind and David Davis, have signed an early day motion saying it is wrong to hold the referendum and elections on the same day.

They could combine with Labour to inflict the first serious defeat for the government on an issue of acute sensitivity for the coalition's internal political balance.

One shadow cabinet member who supports AV said yesterday: "I am going to be put in the impossible position next May of campaigning to remove the Liberal Democrats on my local council three days of the week, and then join forces with the same Liberal Democrats to call for a change in the voting system for the Commons for the remaining days of the week. It is totally incoherent."

The chairwoman of the electoral commission, Jenny Watson, has said she will rule in September on whether she thinks the referendum and local elections should take place on the same day, but indicated she did not believe it was an insurmountable problem and said she had given the proposal an amber light.

But in a sign of anger on the Tory benches, the rightwing backbencher Edward Leigh warned Clegg in the Commons: "If we had had the alternative vote in 1997, the Conservative party would have been reduced to a pathetic rump of 65 MPs.

"Does he not think that, precisely because AV is not proportional, it raises complicated questions? It is extraordinarily dangerous, therefore, to have the referendum on the same day as other elections, namely the Scottish elections. We need a proper debate on the issue."

Clegg repled: "About 39 million people will be invited to vote next May, and it seems to me that, instead of asking people to constantly go back to polling booths to cast separate votes, it is perfectly right to invite them to have their say on a simple yes/no issue on the same day – at, by the way, a lower cost to the exchequer. It will save about £17m."

• This article was amended on 28 July 2010. The original referred to Jack Straw as justice secretary, albeit in the context of the shadow cabinet. This has been clarified.