This weekend marks the 10th anniversary of the swearing in of the Rudd government and the beginning of a lost decade for Australian progressives.

It’s easy to remember the excitement of the times – a wild election night that had seen only the fifth change of government since 1949, the ascension of Australia’s first baby boom leader and a sitting prime minster rejected by his own electorate.

Most Australians want banking royal commission – Guardian Essential poll Read more

With a promise to restore workers rights, a bipartisan commitment to action on climate change and a community sector that had staked out homelessness as a national priority, Australia felt on the verge of a great leap forward.

10 years on, we have a union movement under unprecedented attack by a new government agency politically focussed on its annihilation, a resurgent coal lobby not just blocking climate action but proposing new mines and rising levels of inequality across the society.

What we didn’t know then was that Australia was on the cusp of a period of political instability not experienced since the early days of federation; a decade that has seen five changes of prime Ministers (and counting), all but one of them outside the usual; electoral processes.

Having squandered its time in power in two short and largely miserable terms, Labor now stands like an optimist at the pulpit ready to embark on a second marriage, aware of their limitations but hopeful of learning from their mistakes.

2007 was also the year that Essential started the weekly opinion poll, charting national voting intention – plus a bunch of other questions designed to give us a read on the attitudes of the Australian electorate.

These figures give us the opportunity to look in the rear view mirror and ask ourselves, where did it all go wrong? And more importantly, what can we learn to make sure it doesn’t happen again?

Looking back on the primary vote over the past decade, the most striking thing to note is the increase in voters turning from the major parties.

In 2007 the Labor and Coalition shared around 80% of the primary vote – today that share is closer to 70%. Put another way, the minor parties have increased their share of market by 50% over the past 10 years.

The rise of the outsider creates different challenges for the two parties.

Queensland election offers Turnbull no respite as right-leaning voters show their anger | Katharine Murphy Read more

On the left the Greens have consolidated a national vote in the early teens – taking lower house seats off Labor in the inner city at both federal and state level, secured Senate positions in all states, and were part of what was to all intents and purposes a progressive coalition from 2010-13.

For Labor, where the differences tend to be of degree rather than direction, the Greens influence is to push Labor further to the left, to move harder and faster than if they were simply focussed on the centre ground. The Greens threat is targeted in the high-income inner city areas, where the most progressive Labor candidates tend to hold seats. While threatening these candidates at the ballot box, they also empower them to push their parties to bolder economic and social policies.

With the mature leadership on both sides, the opportunity exists for the Greens to push into high-income Liberal areas and begin to shift the electoral needle in more fundamental ways.

The challenge to the right is more complex and existential. The rise, fall and rise again of One Nation, the brief emergence of Palmer United, the clustering around high-profile independents like Jacquie Lambie, Bob Katter and Nick Xenophon, and now the emergence of the Australian Conservatives, represent a nativist political force.

Rather than a contest of degrees, these candidates represent a fundamental challenge to the Coalition’s pro-business growth model and a practical challenge to the rural National party at the accommodations they make to maintain the Coalition.

On economic policy, the their economic protectionism is at odds with the Liberal market fundamentalism; on immigration their creeping xenophobia is an affront to affluent Liberals. As the Queensland LNP discovered on the weekend, there is no obvious way of reconciling these contradictions.

The real disruption of these new players though is their impact in the way the major parties engage in the political contest: no longer just with each other, but with their energised fringes. And if fighting elections against these new forces is a challenge, finding ways of building alliances in government will prove even more challenging, though inevitable.

Another symptom of the instability has been the wild fortunes of our political leaders. Like a series of slippery slides, political leadership over the past decade has been one of high hopes and rapid disillusionment.

Kevin Rudd set the standard with his extended honeymoon, which abruptly ended when he walked away from an emissions trading scheme. Julia Gillard was dragged down by a mixture of her own political shortcomings and the ruthless negative campaigning of her opponents (both external and within). Rudd came back with a short burst but was gone as soon as people remembered why they had rejected him.

In power the Liberals fared no better. Abbott himself rose more modestly but fell just as sharply over the litany of broken promises that constituted the 2014 budget. Turnbull’s ratings jumped off a cliff the moment it became apparent he had traded in his values for political power.

The lesson of the decade is that with the fragmentation of media, the constant news cycle and a decline of party loyalty, the ship of government has become too unstable for one person to steer. When every ship ends up like the Titanic, you can’t really go blaming the captain.

If there is one good thing about Bill Shorten’s modest approval levels at the moment, it’s that they prevent him putting himself forward as yet another presidential Messiah-like candidate

Instead, Shorten’s Labor dominates the current political cycle as a team effort, a stronger bench and a leader who signals he is prepared to play the team game. He’d hate the comparison, but like John Howard he promises less in the way of excitement. This could be a good thing. And the irony here is the one person he can thank is Kevin Rudd who in his final act – perhaps as an apology to Labor’s lost generation – changed the party rules to deliver structural stability to the Labor leadership.

The final trend of the past decade worth contemplating is the drop in trust of public institutions – the web of organisations that constitute civil society.

With the significant exception of the ABC, all these institutions have experienced a dramatic demise in public standing, with the majority experiencing single digit high trust levels.

The sharp drop was first apparent around 2012, when the hung parliament descended into swamp warfare and the dynamic has not recovered in subsequent years.

Labor to vote against amendments to same-sex marriage bill Read more

The problem of a trust deficit for progressive parties is it starves the electorate of faith in the capacity of government to build something better. Even today, when people say they want the government to intervene in areas like energy and economics, low trust undermines their capacity to believe that things can be better. Cynicism has always been the friend of conservatism.

As we enter the final weeks of the political year, with all conditions appearing conducive to a new era of Labor government, the lessons of the lost decade are worth re-stating.

In a fragmented political world, a pragmatic progressive alliance that focuses on a shared agenda, rather than personality, governing calmly but decisively, ready to rebuild trust in public institutions is the only viable strategy to longevity.

Because living through another decade like the one we’ve just had is just not worth contemplating.