Self-interest can have surprising consequences. Just as capitalism is premised on the idea that the enrichment of the few will solve the poverty of the masses, Kevin Rudd’s efforts to permanently install himself as ALP leader may have saved the party by making it far more democratic.

We are only halfway through the Coalition government’s first term and its leadership is already in turmoil. The Greens have lost a leader and elected a new one. Clive Palmer has lost two of his three senators. The speaker of the House of Representatives has been forced to resign.

Amid the chaos, Bill Shorten’s leadership appears safer than that of any Labor leader in modern memory. And for that, the ALP has Kevin to thank. One of Rudd’s “conditions” for returning to the prime ministership in the dying days of the previous government was that the party membership have a 50% say in the election of the ALP leader. (It used to be a 100% caucus decision.) The result is a significant change, not just in the internal dynamics of the Labor Party but also in the terrain on which the battle between the major parties takes place.

The job of first-term Opposition leader is usually the worst job in politics. It demands you to be simultaneously proud of what the previous government achieved and apologetic about all the things voters disliked about it.

Before you can get on with “rebuilding” you have to first spend a year “listening” – ostensibly to the public, but in practice mainly to the new government gloating or to leadership rivals leaking against you. And then, just as the next election finally creates an opportunity to spell out a positive vision, you are often dumped in order to give the party a “fresh voice” that can “cut through”. Just ask Kim Beazley or Brendan Nelson how much fun it is to take over a political party after a defeat.

Ironically, while Kevin Rudd is almost certainly the ALP’s most divisive leader in recent history, his new rules for selecting a Labor leader destroyed the incentive for the kind of backroom jockeying and leaking that won him his party’s leadership. While it wasn’t his goal, Rudd created a structure that united his party around the task of winning government.

The need for a ballot of the members meant that Shorten and Anthony Albanese were back on the hustings straight after the ALP’s 2013 drubbing, talking positively about policy, party building and, perhaps most significantly, each other.

The art of leaking against political opponents in order to undermine them is as widely practised as it is denied, and while such a tactic can be effective, both ALP leadership candidates decided they would win more votes from party members by treating each other with respect. The strategy left both men looking impressive, and the biggest winner was the Labor brand.

When Rudd won caucus support for his reforms to the leadership-selection process, it was viewed as the manoeuvring of a Labor leader seeking to strengthen his own position. Rudd lacked both factional and union backing, and the only numbers he thought he could bank on were those of the membership. Whatever his motivations, his reforms made Labor the first Australian parliamentary party to give its members a real say in its leadership since the Australian Democrats self-combusted.

At this point, I have a confession to make. I have been a bit of a weathervane on the best way for a political party to select its leader. Once upon a time, I thought it was obvious that the party’s membership should choose its leader.

Back in 2001, I was chief of staff to the then leader of the Democrats, Natasha Stott Despoja. Although the Democrats won 37 Senate seats between 1977 and 2004, a rift between its membership (which had the right to select the parliamentary leader and set policy) and the party room (which wanted those rights for itself) ultimately destroyed the party.

After watching at close range as a group of Democrat senators undermined a leader chosen by a membership they felt distant from, I changed my mind. I then decided only a leader chosen by a group of parliamentarians could demand the respect of those parliamentarians. Thanks to Rudd’s success in white-anting Julia Gillard, however, I changed my mind about that model too. My first reaction when I heard of Rudd’s plan to give the membership a say in the party’s leadership was to dismiss it as a cynical attempt to protect his own back. My second thought was that running a ballot would be clumsy and slow, and the process would make life a misery for the ALP in the post-election environment. In hindsight I think my first reaction was spot on, but my second was completely wrong.

Despite how hard some have tried, it has been impossible to get a half-decent rumour running that Shorten’s leadership is in trouble, for the simple reason that it clearly is not. Starved of opportunities to discuss the dangers to the Opposition leader, the media has spent 18 months analysing the performance, and the prospects, of Tony Abbott and his Coalition cabinet. The results haven’t been pretty.

The ALP has been ahead of the government in each of the past 27 Newspoll results. The Coalition leadership is under pressure, and Shorten is not.

Rudd’s gift to the ALP is delivering far more than short-term leadership stability. And, as with any significant rule change, it has led to a significant change in strategy among prime-ministerial aspirants.

Until 2013, the only way for an ambitious member of the ALP to become prime minister was to secure the support of a majority of their parliamentary colleagues. In such an environment, leaking against rivals is core business, as it simultaneously harms opponents while building up a “favour bank” with the journalists who play a central role in later campaigns for the top job.

Chasing the leadership of a major Australian political party was, by design, an insider’s game. Promises about who would be treasurer counted as much as the principles by which the economy should be managed. The rules forced public figures to make private compromises that often confused the electorate, such as Julia Gillard’s opposition to same-sex marriage (a position said to be taken to win the support of Joe de Bruyn’s “Shoppies” union).

The next generation of aspiring ALP prime ministers are now required to play a more public game. The new rules mean that, to lead their party and have any chance of leading the nation, ALP up-and-comers must now earn the support of both parliamentary colleagues and the party membership. It’s the latter that is making things interesting.

If only party-room insiders get a vote for party leader, the smart strategy for ambitious backbenchers is to play their cards close to the chest, stick close to some powerful friends and never be seen as “off message”.

Now that members hold half of the votes, ambitious ALP backbenchers need to let them know where they stand on big issues. They need to inspire with their long-term vision while reassuring members that they are not simply fairweather friends. While parliamentary colleagues (and rivals) can be bought off with the promise of a cabinet position, party members aren’t so amenable.

Seemingly overnight, a cluster of ALP backbenchers have developed much higher profiles than most government ministers. (Can you name the federal minister for justice?)

In the past 18 months senator Sam Dastyari has leapt from backroom fixer to the public face of progressive tax reform. Where Labor was once nervous about being seen as “anti-business”, Dastyari has used his role on the Senate Economics References Committee to make corporate tax evasion as much an issue of law and order as it is of social justice. It’s hard to imagine that his campaign to make big business pay its fair share of tax has done him any harm among ALP members.

Terri Butler has only been in the parliament since 2014, but has already played a central role in the fight to bind ALP MPs to support same-sex marriage – an issue with a large and motivated constituency among the membership.

Pat Conroy has made a name for himself as the chair of the ALP’s Waste Watch committee, a position from which the MP could claim the scalp of the former speaker, Bronwyn Bishop. No small feat for a backbencher.

Tim Watts and Clare O’Neil, two young MPs from Melbourne, have taken it upon themselves to write a book spelling out a reform agenda, not just for their own party but also for the parliament itself. Among a raft of suggestions, the authors of Two Futures propose the ALP build non-parliamentary “action caucuses” focused on specific issues such as same-sex marriage and climate change. While such a reform is likely to help build party membership and revitalise member engagement with policy development, it would also help an aspiring prime minister build personal support.

Andrew Giles has not attracted as much media as some other high-profile backbenchers, yet his role as chief spear-carrier for those who oppose offshore processing of asylum seekers has won him support among sections of both the membership and the party room. Perhaps just as importantly, the warmth that Shorten and Giles showed each other at the end of an emotional national conference debate in July about “turnbacks” shows that the ALP now understands the significance of playing nice – a key strategy for winning leadership and government.

Rudd’s reform is also changing the traditional factional model of the ALP. Since the last election, groups like Labor for Refugees and Labor Environment Action Network have played an active role in both public campaigning and private lobbying to support the policy positions of ALP members. The development of factions based on policy issues, rather than preselection planning, will inevitably change not just the culture of the Labor Party but also the composition of its party room. Whether or not formal proposals such as those put up by Watts and O’Neil are adopted, the simple fact is that ALP politicians who want to become prime minister will not lightly ignore their members’ priorities, or renege on promises to them.

Political parties get to make up their own rules about how to select their leader, and the rules play a major role in how candidates for high office are shaped. Parties with better rules will choose better candidates, and will have a better chance of forming government.

Now that it is much harder for Labor to change leaders, the party is both less flexible and more resilient than the current government.

Recent public opinion polls had Malcolm Turnbull and Julie Bishop far more popular than Tony Abbott. Scott Morrison, on the other hand, lagged behind. While interesting reading for political tragics, such polls tell us virtually nothing about the Coalition’s true leadership stakes.

The same is not true for Labor. The fact that Tanya Plibersek and Anthony Albanese have strong support within the community reflects their standing among the membership as well. While Shorten is under no immediate pressure, in the long run his future will rest on his ability to reconcile parliamentary politics and popular opinion. Which is how it should be.

The ALP’s new rules may not be perfect. Indeed, it’s not even obvious they will work or be retained. But it is obvious that the Australian public is crying out for a new kind of politics. Democracy requires diversity if it is to offer citizens a genuine choice, and there is no doubt that Labor’s rule changes have already given it a point of difference. Of course, having already changed my mind twice about the best model for selecting a parliamentary leader, there’s no guarantee I won’t change it again.