Labour is playing bad politics. The leadership campaign is turning into a tin-ear, foot-in-mouth competition about who can be nastiest to the Liberal Democrats. As candidates desperately try to prove themselves more true Labour, more tribal than the next guy, they are in danger of missing the big picture about our changing politics. They could end up wrecking their party's position for the next generation, which is their own.

Part of this is about an underestimated and under-discussed quality in politics: tone. The biggest tone change in politics has been the transition from the raw warfare of the latter days of New Labour towards the apparently collegiate and good-humoured attitude of the coalition. It's true this is already fraying at the edges, with reports of a vicious bust-up between Iain Duncan Smith and George Osborne over welfare reform; but the crucial thing is that the public seem to like the spirit of co-operation.

The Australian election ought to have been a sharp reminder for Labour in Britain of the perils of ruthless, even nasty, behaviour. There, Labor's Julia Gillard, who brutally ousted Kevin Rudd two months ago, is now trying to patch together a coalition after taking a pasting from opposition rightwinger Tony Abbott. Commentators claim the reason for Gillard's failure at the polls was distaste for the brutality she displayed in despatching her predecessor.

So Labour's leadership candidates have to think hard and keep asking themselves if they have understood the voters' clear dislike of aggressive and over-cocky politicians. It was never "all about Gordon"; it was about the demeanour of ministers who seemed to feel they had a right to rule, and thought their opponents contemptible. People won't have that any more, a truth that will remain even when the coalition becomes seriously unpopular.

Humour, self-deprecation, owning up to mistakes and being flexible in your thinking – all in. What is definitely out is Ed Miliband's pledge to the Kilmarnock Labour party that "we have to make the Lib Dems an endangered species – and then extinct". He says it was a throwaway remark taken out of context. Good. But it was bad politics.

This is not an abstract or medium-term matter, because the possibility of a future Labour-Lib Dem realignment is still on the cards, and may be decided this winter. If Labour goes into all-out war against Clegg and colleagues, betting on the Lib Dems breaking up, and loses the bet, they may well find British politics has realigned very differently. Put it another way: the behaviour of Labour now may decide whether its future is as an opposition party or the next government.

The "tee-hee" business of trying to flatter Charles Kennedy, who has had a rough time lately, into defecting to Labour, or talk of exterminating the Lib Dems, is tone-deaf politics. It's too obviously opportunistic, too tribal – too early. After the election result Labour politicians don't have the right or the authority to talk about making other parties extinct.

Yet Ed Miliband is right to suggest to Lib Dems that they have much more in common with Labour than with the Conservatives: indeed this is something they are already discovering for themselves. There will be a fascinating, if agonising, debate at their conference in Liverpool over whether the Lib Dems should leave themselves open to a Labour alliance in future.

Privately, this is something Labour politicians are cheerfully thinking about too. For the fact remains that on a range of issues from taxation to social mobility, from tuition fees to Trident, Lib Dem and Labour voters are usually coming from the same place.

For many Lib Dem activists, the price of winning the alternative vote is not worth the pain of helping deliver a Tory-dominated cuts agenda, slashing public services and increasing inequality. Is that what the long tradition of social radicalism the party represents amounts to? The grumblings have started, with Simon Hughes demanding a veto on policy, complaints about Michael Gove's "free schools" and disquiet over the appointment of Sir Philip Green, whose wife lives in a tax haven in Monaco. It can only be a matter of time until more cracks in the coalition appear.

Labour's proper response is to keep acknowledging past failings and stay politely aloof. Everyone is still watching and waiting: David Miliband has clearly had a bad week. His advice about phoning friends and putting out nibbles for a local party sounded like the kind of thing John Redwood might have said, a well-meaning stab at earth-speak. If it was meant as a joke, so much the worse.

Had brother Ed stuck with his latest idea, to subsidise companies who want to pay a living wage, he would have enjoyed a leap ahead. It's a good idea, radical and unexpected. But I come back to those words about making Lib Dems endangered and then extinct. In the real world, people don't much like the idea of endangering species. It isn't nice. Extinction? Rarely something to gloat about. (And not very Ed Miliband, either.)

Both brothers should reflect on Nick Clegg's life inside the coalition – an amiable and intelligent politician who has left his own comfort zone and is being carried along, a willing passenger, in the most radical and fiscally extreme experiment since Geoffrey Howe arrived in the Treasury in 1979. He knows how high the stakes are, and that if the economic experiment fails (likely) or he fails to deliver PR (likely), then his political career will be going nowhere.

Clegg ought to look ashen-faced, shaky, uncertain and deeply worried. Instead, he seems ebullient, perky and as optimistic as ever. So why would that be? He has been using a magic facial elixir, rubbed on each morning after he shaves. It's called Power. It's the energising effect of being able to do things, make real choices and directly affect the country you live in. He has it because, towards the end, Labour got the tone wrong. Isn't that a lesson to remember?