Australia’s union movement is in a period of soul-searching. After three decades of labour market “reforms”, the workforce has fractured and wage inequality has deepened.

Part-time and casual jobs have increased as a proportion of the economy, along with the number of people who say they want to work more hours. Household income is lower in real terms than it was in 2011.

Labor frontbencher Mark Butler has warned that the labour movement in Australia is fast hitting a tipping point where it might cease to exist as a broad-based institution in Australian society.

Butler, a former union secretary before he entered federal politics, observed that union membership collapsed in the 1990s as John Howard brought in waves of labour market deregulation, and because of structural changes in the Australian economy.

Play Video 1:46 The answer is in the graph: Australia's underemployment problem – video

The trend then appeared to stabilise but membership has dived over the past five years and has now reached “a threshold we regarded years ago as existentially threatening” – under 10% of the private-sector workforce. He noted that union density for workers under the age of 25 is running at about 5%.

The public message the Australian union movement gives is that the regulatory system, implemented during the decade of Coalition government from the mid-1990s to 2007, has significantly undermined the capacity of the movement to organise in workplaces, and that is reflected both in anaemic wages outcomes and in the shrinking footprint of membership in workplaces.

But some figures within the movement are prepared to acknowledge that mistakes have been made. Tim Lyons, a former assistant secretary of the Australian Council of Trade Unions, says workers are now paying the price for decisions made when the Hawke-Keating governments ended Australia’s system of centralised wage fixing in the 1980s and 90s.

“I’m increasingly of the view that we need to describe firm-level enterprise bargaining as a failure,” Lyons told Guardian Australia. “If I’d been there at the time, at the ACTU, I probably would have made roughly the same decisions that were made, but in retrospect, we overshot the mark.

ALP president says 'deep crisis' in unions must be addressed Read more

“Going from complete centralised wage fixing all the way back to firm-level bargaining was really a recipe for very poor outcomes.”

Lyons says it was partly those changes, brought forward by a Labor government with the support of the union movement, that have laid the foundations for many of today’s problems.

Wages stagnation: a 30-year story

To understand what is going on with incomes in Australia, we have to look closely at what is going on in the labour market. John Buchanan, a professor and the chair of business analytics at Sydney University, has long warned about Australia’s rising income inequality.

He notes that wages are determined by market and institutional forces, and changes to Australia’s institutional wage-setting arrangements since the 1980s have negatively affected wages.

Low wage growth remains a problem affecting all levels of employment | Greg Jericho Read more

Buchanan says Australia’s architecture of industrial tribunals and awards – with its roots in the 1890s – reduced wage inequality in the 20th century but, after 30 years of “modernisation”, the industrial relations framework is now entrenching it.

“Within the political class there is low-level moral panic about low wages growth,” he wrote. “The irony is that those lamenting this situation are simply witnessing the ultimate outcomes of policies they have long advocated.”

But how did the changes of the 80s and 90s lead to higher wage inequality? And why do some unionists have buyer’s remorse?

Buchanan and Lyons say years of legislative changes have choked the union movement of power to demand bigger pay rises. They say the shift towards firm-level enterprise bargaining in the early 90s was a key moment.

In the middle decades of the 20th century, with the onset of full employment, and in an economy protected from international competition, powerful unions used to set the pace on labour standards and wages.

Facebook Twitter Pinterest Bob Hawke and Paul Keating led Labor governments from 1983 to 1996. Labor was able to moderate union wage demands by increasing the ‘social wage’ under the accord agreements. Photograph: The Sydney Morning Herald/Fairfax Media via Getty Images

When they secured wage gains in informal collective agreements in key sectors of the economy, their gains would be extended through industrial awards to workers with less bargaining power, including non-union members.

The system worked effectively to spread wage gains from the strong to the weak. Workers’ share of GDP was 46.4% in 1959 but it had lifted to 57% by the mid-70s.

But the system wasn’t faultless. It failed to cope with the world economic downturn of the 70s and 80s and multiple outbreaks of inflation.

When growth was stagnating in the mid-70s, and when inflation was rising rapidly, nothing could prevent powerful unions extracting excessive wage increases above award rates, even as inflation hit 18%.

Labor signals plan to beef up union power to help boost wage growth Read more

The Hawke Labor government came to power in 1983 with a radical plan to fix the problem: it would get inflation under control, and achieve macroeconomic balance, by slowing wages growth in awards.

In a series of “accords” with the union movement, principally the Australian Council of Trade Unions, Labor got the unions to moderate their wage demands for years in return for improvements in the “social wage” – improvements to health services through a reformed Medicare, the introduction of superannuation, and a family allowance supplement for poor families, among other things.

Then, in 1993, the Keating Labor government adopted formal “enterprise bargaining” in its Industrial Relations Reform Act, which was also supported by the unions. Enterprise bargaining was to supplant the awards system that had helped unions flow their wage gains across the economy.

“This marked a profound shift in the notion of fairness at work,” Buchanan has written. “Whereas previously awards had worked to spread the gains of the strong to the weak, now they were explicitly prohibited from doing so.”

Butler expressed a more nuanced view of the 30-year trend in his speech on the future of unions in early February. He characterised the shift to enterprise bargaining as “a clever policy response which has served large parts of our economy very well” but he felt it had also stranded some workers, particularly in the growing private-services sector, which was the sector he worked in organisationally as a union official.

Facebook Twitter Pinterest Unionists rally outside Flinders Street station during a train strike and stop work meeting in Melbourne in 2015. Photograph: Mal Fairclough/AAP

“Small, dispersed enterprises – where funding for wage increases is often controlled by third parties, like clients or government funders – have consistently been unable to navigate the enterprise bargaining system in a way that works for them and their workforce,” he said.

Those sectors of the economy had recorded flatter wage growth “than the anaemic economy-wide figures we see commonly reported today” – and that was a very big problem. “This requires substantial thought to ensure that the substantial benefits of the Keating reforms are preserved, while meaningfully addressing the needs and aspirations of workplaces that struggle to work within that system.”

The right to strike

Australia was once a country with a ready resort to industrial action. But over the past 25 years, the capacity of workers to withdraw their labour through strike action has been curtailed. This has had an effect on wages.

The big drop in strikes occurred during the Hawke/Keating years, it fell again under WorkChoices, and really wasn’t changed at all under the Fair Work Act brought in by the Rudd government.

The limit on times and reasons workers can go on strike is much reduced. The old days of wildcat strikes where workers would just walk off the job are long gone.

This year New South Wales train drivers were prevented from going on strike owing to the Fair Work Commission ruling that it was “threatening to endanger the welfare of part of the population” and would “cause significant damage to the Australian economy or an important part of it” – that was from legislation crafted by a Labor government.

The Turnbull government – and other Coalition state governments – have also sought to make life tougher for unions by imposing measures mostly in the name of “transparency” but which unions argue are really about draining their resources.

Facebook Twitter Pinterest Police on the docks in Fremantle during the 1998 waterfront dispute. The ability of workers to organise and strike has been severely restricted since the 1980s. Photograph: Rohan Kelly/AAP

One union official in Queensland told Guardian Australia that “what took a pair of union officials less than two weeks to access a workplace can now take a team of six union officials, plus lawyers, eight weeks to access a workplace. That burns through money to get anything done.”

It’s also the case that employers generally don’t have to do anything – they don’t want to give out pay rises and so they are never in a rush to do so – which forces unions to spend resources just to get the negotiations going.

Sydney train strike stopped: Fair Work Commission blocks union action Read more

The termination of wage agreements: the race to the floor

One of the key underpinnings of modern industrial relations system are enterprise bargaining agreements – which are generally negotiated between employers and unions and usually cover a period of three to four years. They are a key indication of where wages are going.

And right now wages are still going down.

The assistant governor of the Reserve Bank, Luci Ellis, last month noted in a speech that new EBAs “lately have tended to involve smaller wage increases than the ones they replaced”.

She noted that because these agreements are in place for a number of years, “if wage growth is to pick up, wage increases for other workers – including in future enterprise agreements or in other wage-setting streams – will need to pick up”.

But right now they’re doing the opposite.

In the September quarter of last year, the average annual wage increase of approved EBAs was just 2.4% – the lowest recorded, and well below the average growth of 2.9% for all current EBAs – which suggests if it is a just a cyclical event, it still has a way to play out yet:

But there is also a structural issue occurring with EBAs. In the past the previous agreement would be the starting point of negotiations but now more and more employers are seeking to terminate agreements – which effectively puts workers back on the award rate.

Employers are also moving away from EBAs – the number of workers covered by them is now lower than it has been for more than a decade:

The Australian Public Service Commission has announced that future EBAs covering APS staff would be limited to a wage growth of just 2%.

Budget papers predict inflation will grow by 2.25% in 2018-19 and 2.5% for the following two years, that means the government has a policy that will see the real wages of public servants fall – unless they attempt to negotiate individual contracts.

Facebook Twitter Pinterest John Howard, John Anderson, Peter Costello and Peter Reith. The Howard government brought in waves of labour market deregulation in the late 90s and early 2000s. Photograph: Mark Baker/Reuters

A University of Melbourne labour economist, Jeff Borland, told the Financial Review that owing to “the decline of trade unions, changes in bargaining power suggests, if anything, you may be able to push the rate of unemployment lower without causing a big breakout in wage inflation”.

That suggests the change in the relationship between unemployment and wage growth is not just cyclical but ongoing.

Political frontline of the debate

The debate about wages stagnation and voters’ concern about cost of living dominates the sensible political conversation in Canberra.

The prime minister, the treasurer, and the governor of Australia’s central bank insist the rules of supply and demand between wages and unemployment are not broken. Scott Morrison acknowledged in an interview with Guardian Australia there had been a “dislocation” in the economic orthodoxy triggered by the global financial crisis and its aftermath. He says wages will pick up if unemployment keeps falling.

He also argues that low wage growth is just a cyclical thing – and for this he can point to profits.

Company profits may have skyrocketed last year while wages growth was at record lows, but companies don’t start giving pay rises just because of one good quarter or even year.

It might seem like things are out of whack when you compare annual profits growth with annual wages, but when you compare the three-year average annual growth of profits with the annual growth of wages, we see things are pretty much where you’d expect them to be:

After three years of falling profits, 2017 was a boom year, but the carryover of those bad three years remains –the hope is that things will continue to improve and thus so too will wages.

Similarly, perhaps the link between wages growth and unemployment isn’t really broken, it is just that we should be looking at underemployment instead – because wages growth is about where you would expect it to be given the level of underemployment:

If all is well, then as underemployment continues to fall (it fell last year from 8.7% to 8.4%) we should see wages growth begin to increase.

That will probably be the earliest sign to tell us if things have changed.

Because if wages don’t start to go up as underemployment falls, then we will truly know that all the old rules are broken.

Employer groups such as the Business Council of Australia and the Australian Chamber of Commerce and Industry support the government’s rationale and policy response.

Peter Burn, the director of public policy at the Australian Industry Group, agrees that income inequality has increased in Australia since the 80s and 90s. But he says unions are putting a particularly negative spin on the fact.

Facebook Twitter Pinterest Bill Shorten addresses the workers at the Bombardier factory in Dandenong. Labor has discussed pegging the minimum wage to median earnings to address inequality. Photograph: Mike Bowers/The Guardian

“If you compare wage levels in Australia to every other country in the world you find that Australia is still in the top 10% ... we’re still at the rich end of the wages scale,” he said.

“We don’t have relatively high inequality, that’s a fact. And even in terms of wages inequality, we have even less inequality when we look at disposable income.

“But that doesn’t seem to be the unions’ argument. If wage outcomes are correlated to industrial structures and the way they’re planned, then we must have good industrial structures, because we’ve also got high wages.”

Labor is under pressure from the union movement to strengthen the regulatory framework to boost the collective bargaining power of workers, and has been signalling its intention to rewrite the rules for some months.

The shadow employment and workplace relations minister, Brendan O’Connor, says Labor is considering changes to the law which would stop employers “resorting to the nuclear option of terminating agreements when negotiations fail”.

Marking out the terrain for the next federal election, Labor has also left the door open to pegging the minimum wage to median earnings.

“The government is relying on old ‘trickle-down’ economics whereby large company cuts to big business will inevitably end up in the hands of workers,” said the shadow treasurer, Chris Bowen.

“But given a structurally different labour market, this is by no means assured. And this is why Labor is taking a different approach and will strengthen our enterprise bargaining system, opposing cuts to penalty rates and ensuring that workers have the right skills for the jobs of the future.”