The coalition government is to test its unity by announcing that it plans to stage a referendum on voting reform next May, amid signs that Labour enthusiasm for the reform is wavering owing to the party's growing hostility to the Liberal Democrats.

Nick Clegg, the deputy prime minister, is expected to make the referendum announcement on Tuesday as part of a constitutional package including an equalisation in the size of parliamentary constituencies, a move that will probably favour the Tories.

Andy Burnham, the shadow health secretary and Labour leadership contender, told the Guardian that voting reform was "a peripheral issue" and added: "It is not my party's job to prop up the Liberal Democrats by helping them win a referendum that is important to them."

Burnham, a long-time sceptic about voting reform, said he was leaning towards reform, but the party could not officially take sides. He said: "The party nationally couldn't campaign for any one position – you know, it really couldn't. Those who are calling for retention of first past the post are making an incredibly important and legitimate argument."

He added: "Let's not get obsessed by this issue, because it really is irrelevant. It's a kind of fringe pursuit for Guardian-reading classes."

Ed Miliband, one of those contesting the leadership with Burnham, has committed to Labour backing the AV referendum vote if he becomes leader.

The referendum commitment was probably Clegg's single biggest prize in the coalition deal, and defeat in May would be destabilising, as Lib Dem MPs question whether the price of joining the government has been too high.

The referendum will be a straight choice between first past the post and the alternative vote, whereby voters rank candidates in order of preference. May is seen as the preferred date as it would coincide with elections to the Scottish parliament, Welsh assembly and English councils, so boosting turnout.

Critics of AV believe it leads to negative voting, makes coalitions more likely and can produce results that exacerbate pre-existing voting trends.

David Cameron is thought to believe that the Conservatives would probably have lost an extra 20 seats in this year's election if had been held under AV. Should voters back AV in a referendum, he could still hold a 2015 election under first past the post if a boundary review equalising constituencies is not complete.

Some Labour figures judge that the emergence of the coalition could radically change the impact of introducing AV. Nick Palmer, a former Labour MP, said last week: "In the worst case, the outcome could be deadly: an AV system with two parties locked in anti-Labour embrace could shut us out for decades."

In the past it has been assumed that Lib Dem second preferences in a general election held under AV would split two to one to Labour, but the experience of the coalition could change that dynamic.

Supporters of the planned yes campaign intend to distance it from the Lib Dems to try to broaden support and sidestep the growing wider political tensions between Lib Dems and Labour.

They are looking for leading business figures such as the former CBI bosses Lord Jones and Lord Turner to front the campaign, as part of an attempt to make it non- party political and prevent the coalition becoming the issue in the referendum.

In a sign of deepening tension between Labour and the Lib Dems Harriet Harman, the interim Labour leader, told the Guardian she was launching a campaign targeted at the 57 Lib Dem MPs over their party's decision to back the rise in VAT.

She said that Labour whips would ensure there is another vote on the issue in the Commons before the summer recess.