In his farewell speech as prime minister last month, Malcolm Turnbull pointed to “an insurgency” in his own party and “outside forces in the media” as the architects of his demise.

If there was any doubt at all who the media forces Turnbull was referring to during those final minutes in the prime minister’s courtyard in Canberra, there is, after the events of the past 24 hours, none now.

Rupert Murdoch is the name firmly in the frame along with his ubiquitous News Corp empire – an organisation which is accused of playing a major role in orchestrating the removal from office of not just Turnbull but also Labor’s Kevin Rudd.

In the case of Turnbull he believed his Liberal colleagues had been gripped by “a form of madness” so the only way they could see to end the unrelenting internal turmoil and negative coverage in the media was to cave into it and replace him as leader.

Rudd, equally, believes the cacophony of negativity from News Corp undermined his first prime ministership, then that of successor Julia Gillard. He has called for a “full-throated inquiry” into News Corp and branded the company “a cancer on democracy”.

But the details that have emerged over the past 48 hours of the role the US-based Murdoch played during last month’s visit to his Australian assets raise serious questions about how Australian politics can be swayed by a concentrated media industry where News Corp dominates.

Turnbull certainly believes he was the target of a News Corp campaign. When he narrowly fended off Peter Dutton in a party room spill on Tuesday 21 August, Turnbull phoned Murdoch to ask him why he was trying to replace him with the home affairs minister.

Rupert Murdoch intends to transform Australia into a conservative nation and he wants to put it on the Trump road Associate Professor David McKnight

Turnbull had watched horrified as shortly after Murdoch’s arrival in Australia, News Corp, the most powerful media organisation in the land, turned on him. The Daily Telegraph warned of “a toxic brawl” over energy policy and that Dutton was preparing to challenge him. On Sky the night-time commentators Peta Credlin and Andrew Bolt ramped up their negative coverage of the national energy guarantee and Turnbull’s performance.

“There was no doubt there was a marked shift in the tone and content of the News Corp publications once Rupert arrived,” one of Turnbull’s former staff told Guardian Australia. “And there was no doubt in our minds that News was backing Dutton.”

The prime minister had another reason to believe the octogenarian media mogul was driving the negative coverage – Turnbull had been warned by another media mogul that Rupert wanted him replaced.

According to both the Australian Financial Review and the ABC, Murdoch had told fellow media billionaire Kerry Stokes, owner of the Seven Network, a few days before that Turnbull should be replaced. Guardian Australia also reported that Turnbull was warned in a phone call from Stokes that Murdoch and his media company News Corp were intent on removing him from power.

Stokes is said to have replied that the likely result of such a campaign would be to deliver government to Labor and Bill Shorten. But Murdoch is reported to have brushed aside such concerns, saying it would only be for three years and he made money under Labor in the past.

If this is accurate, it underlines everything I've written recently about the Murdoch media behaving effectively as a political party in pursuit of its financial interests. It would also undermine everything said in Murdoch's defence by his henchman. https://t.co/oWD1aTZjIH — Kevin Rudd (@MrKRudd) September 18, 2018

By that week’s end the deed was done. Turnbull was out as prime minister, replaced by Scott Morrison after Dutton’s much hyped candidacy failed to get the numbers.

‘A form of madness’

As the dust settled over the carnage in the Liberal party many began to question the role of News Corp in destabilising Turnbull’s leadership and the “form of madness” that had overtaken the party.

Malcolm Turnbull gives his farewell press conference as prime minister. Photograph: Sam Mooy/EPA

“This is familiar territory, and it’s gone on many times before,” says Associate Professor David McKnight from the University of New South Wales, who has written extensively on Murdoch and how he wields political power.

“In my view, Rupert Murdoch intends to transform Australia into a conservative nation and he wants to put it on the Trump road.”

“A cancer on democracy” is the verdict of Rudd, who also fell out of favour with News Corp during his time in office.

He found the media giant lined up against him in 2010, midway through his first term as prime minister, and again in 2013 at that year’s general election. Who could forget the Daily Telegraph’s contribution to political debate on the day the election was announced? “KICK THIS MOB OUT” was its front-page headline.

Turns out there may be more to all this than Murdoch wants us to know. Murdoch ran a vicious campaign in 2013 to elect Abbott. Same in 2018 for Dutton/Morrison. Time for a #MurdochRoyalCommission. https://t.co/bh4nXSCVXn pic.twitter.com/pLfpWpVtE1 — Kevin Rudd (@MrKRudd) September 18, 2018

A big fish in a small pond

Australia has the world’s third-most concentrated media market after Egypt and China, according to a major international study by the US researcher Eli Noam.

News Corp Australia dominates the country’s media sector, with 58% of daily newspaper circulation; a swathe of regional newspapers, the only national broadsheet, the Australian; the only pay TV network, Foxtel, which broadcasts the Murdoch-owned Sky News; and the most-viewed website, news.com.au.

News Corp commentators lined up to warn Liberals to 'stand against Turnbull’s global warming idiocy'

After the media ownership law changes in 2017, the industry is set to become more concentrated. Nine and Fairfax Media are merging their television and newspaper businesses, and more deals are likely. These could see News extending its influence in regional publishing and further into TV and radio. It was Turnbull who pushed for the changes.

The argument is that Australians can get their news from websites around the world and from Facebook and Google. The problem is though that as the world opens up, the gathering of local news – whether it be national or for a capital city or a local community – is becoming more concentrated.

The campaign against Turnbull

The Sydney Morning Herald recently published 260 online headlines from News Corp websites and 2GB (majority-owned by Fairfax Media, also the publisher of the SMH) about politics, inviting readers to “judge for themselves” whether there had been a conservative media conspiracy against Turnbull. A few were neutral but many were laden with the message of incompetence or impending disaster under Turnbull.

After Murdoch arrived in Sydney on 4 August to attend the annual in-house News Corp awards (the top award went to Sharri Markson of Sydney’s Daily Telegraph for exposing the impending birth of the Nationals leader Barnaby Joyce’s love child), the tempo of negative coverage increased.

News Corp commentators lined up to warn Liberals to “stand against Turnbull’s global warming idiocy”, as the columnist Andrew Bolt termed it. They gave large amounts of coverage to the small group of Coalition MPs, including the former prime minister Tony Abbott, who were opposing the national energy guarantee – Turnbull’s attempt to find a way to placate his divided party.

Then News Corp began promoting the views of a minority in the Coalition who saw Dutton as the antidote.

Bolt warned that the Liberal party should look for new leaders who had the guts to stand up to Turnbull’s plan. Dutton went on the Sydney radio station 2GB to talk to Ray Hadley, also part of the anti-Turnbull cabal, warning that the Liberals’ plummeting primary vote in July’s Longman byelection would spell disaster if repeated at a general election.

A News Corp employee walks into the company’s Holt Street headquarters in Sydney. Photograph: Mick Tsikas/AAP

The day after the News Corp awards, Markson wrote: “The energy policy has thrown the prime minister into his own world of woe and there are now live discussions about his leadership.” She was joined by Bolt and his fellow columnist Janet Albrechtsen and the “Sky News after dark” team in talking up Dutton and slamming Turnbull.

Just how close are these relationships? Bolt acknowledges he is friends with Abbott, as is Murdoch’s son Lachlan. Credlin, Abbott’s former chief of staff, is close to both her old boss and a number of the Liberal party’s conservatives. And insiders at News say Credlin and Bolt have direct lines to Lachlan and Rupert Murdoch.

The good, the bad and the downright vicious

Rudd, now out of politics, is one of the few politicians who has dared to speak out about News Corp.

“Time and time again in cabinet I would raise the need to fully take on News Ltd as a political force in our country,” Rudd tells Guardian Australia. “The entire cabinet, including [Julia] Gillard, would say, ‘You can’t do that. They will just destroy us.’ That’s where we have got to – when an entire government thinks it’s so powerless in dealing with a force like that.”

Why don’t politicians speak out? Because they are frightened Kevin Rudd

Since leaving politics Rudd has made it his mission to call out, on Twitter and in public, what he sees as the dangerous and vindictive role that News Corp plays in politics around the world.

Rudd’s campaign comes after the UK inquiries into News Corp and the phone-hacking scandal, and subsequent inquiries into whether News is a fit company to further extend its influence by acquiring the whole of the UK satellite TV company Sky TV.

“Why don’t politicians speak out? Because they are frightened,” Rudd says. “Because the organisation has the capacity for comprehensive retribution.”

In a statement News Corp responded: “Mr Rudd again demonstrates his failure to understand the role of the media in a democracy – it is to challenge, question and hold to account those in public office on behalf of the communities they serve. A free press is critical to a free society. This often means telling uncomfortable truths, but we have a long and proud history of telling the stories that matter to Australians.”

News Corp clearly makes positive contributions to Australian journalism. Back in 1964 Murdoch created a national daily and over the years he has poured resources into creating a newspaper that does its level best to cover national affairs. Together with Telstra he built Foxtel – a pay-TV network with a 24-hour news channel, Sky.

News Corp’s investment in serious, in-depth journalism has been justly recognised by Australia’s Walkley awards throughout the years. The Australian has won nearly 80 Walkleys, while Sky News has been recognised for its election coverage, and the News Corp tabloids often scoop up gongs for sports coverage, photography and their coverage of breaking news. The Australian has devoted extensive resources to Indigenous reporting and to investigations, such as its dogged pursuit of the millionaire politician Clive Palmer or the current investigation into the Lynette Dawson case.

‘Kevin Rudd thought, at least in the early days, that he could tame the beast.’ Photograph: Stefan Postles/Getty Images

But mingled with the exceptional reporting is the accusation of an agenda driven by ideology, vindictiveness and a sense of grievance held against so-called elites.

Last year, in a 13-part series entitled Holy Wars, the independently owned website Crikey documented the Australian’s approach, where perceived enemies are singled out and subjected to an unrelenting campaign of daily stories and personal attacks backed up by a blizzard of criticism from commentators. Often these targets had been critical of News.

Crikey detailed campaigns against the former human rights commissioner, Gillian Triggs, whose job included investigating cases under 18C of the Racial Discrimination Act. This section, which outlaws racial vilification if it causes offence, insult, humiliation or intimidation, is particularly hated by News Corp because it sees it as too broadly drawn and therefore an improper brake on free speech – and because some of its own have fallen foul of it, including Bolt.

Some 30,000 words were written about Triggs in the Australian. Other figures targeted for special treatment included the head of the Australian Press Council, Julian Disney, who supported enhanced powers for the watchdog after the UK hacking scandal, and Paul Barry, the presenter of ABC’s Media Watch, who has criticised News on his program alongside other media outlets.

“Almost all the players in politics, government, academia, science, media and policy know how it works,” wrote Crikey’s publisher, Eric Beecher. “And every month or two they see it unfold, embarrassed, like watching a public flogging where you turn your head away.”

Some of these campaigns are part of News Corp’s culture wars, campaigns that sell newspapers and reinforce the worldview its editors believe Murdoch wants to see reflected in his Australian publications.

As Crikey pointed out, these can be devastating for their targets but often do not cause the government of the day to kowtow, and are frequently ignored by the rest of the media. Yet they do have consequences. They crowd out important issues and, in the case of the campaign for the repeal of 18C, outlawing hate speech, they have created a sense in some sections of the Liberal party that this is an issue of great importance to Australians.

How News Corp influences politics in a declining print market

It has been suggested that the demise of print media means newspapers – and News Corp in particular – are no longer as influential as they once were. But while print media might be losing its direct clout, news is increasingly being pushed out on other platforms.

McKnight says the decline in newspaper circulation does not necessarily mean News’s influence is waning.

News still controls nearly 60% of the daily newspaper sales in Australia and controls the dominant news websites in several capitals as well as the national daily broadsheet.

“Newspapers achieve their agenda-setting role because they have the biggest newsrooms and every day they originate far more stories than any other news medium,” McKnight says.

“These stories provide the raw material for others, which includes radio shock jocks, TV talk shows, other newspapers and the Twitterati. Newspaper-generated stories are still the largest source of online news.”

Newsrooms may be shrinking and the original reporting of issues that matter to Australians is being crimped by the dwindling resources, but it has become exponentially easier for a story to be picked up and rebroadcast, finding its way to the airwaves or social media at lightning speed.

So should we no longer worry that Australia has one of the most concentrated media industries on the planet? Recent events suggest otherwise.

How News turned on Rudd

Some see clear motivations in News Corp’s campaigning.

“I think it’s their clear pattern around the world,” Rudd says. “Rupert was completely locked into Thatcher and then, when he formed a view that Blair was going to win, he locked into Blair.

I was in the throes of handling the GFC in 2008. They unleashed a full fusillade Kevin Rudd

“Rupert was completely locked into Howard and, when he formed a view that Howard was going to lose to me, they didn’t lock into me, but they at least gave me – not even balanced treatment – but I wasn’t completely attacked through the 2007 campaign.

“Once they sense that you can lose, their standard operating procedure was to go back to the conservatives who are going to do deals on industrial relations, which suits Rupert both ideologically and commercially.”

Rudd thought, at least in the early days, that he could tame the beast. He travelled to New York as the 2007 election loomed to seek an audience with Murdoch. By that stage a Labor victory was looking to be a certainty.

Chris Mitchell, a former editor of the Australian, details many of his interactions with Rudd in his autobiography, Making Headlines. Written after his personal relationship with Rudd had irretrievably broken down, his tales raise questions about the relationship between politics and the press in Australia.

Malcolm Turnbull with Chris Mitchell at the launch of Mitchell’s memoir, Making Headlines. Photograph: Mick Tsikas/AAP

Among the more extraordinary revelations is Mitchell’s claim that in the lead-up to the 2007 election, Rudd consulted him about Labor’s industrial relations policy, drafted by Rudd’s deputy, Gillard, after the Australian’s editor-at-large, Paul Kelly, wrote “devastatingly” about it.

Mitchell writes:

Rudd sensed that he needed to massage the policy and asked me to think about whether there were a change, particularly to the proposed individual contract forms – that the paper would find acceptable. I asked Kelly and our industrial editor Brad Norington to come to my office and explained the situation. It was a bizarre request and one none of us had ever received from previous political leaders. But we decided that the best way to preserve the spirit of individual contract reforms … was to exempt all workers earning above $100,000 from Gillard’s plans.

Mitchell claims that a couple of Sundays later Rudd and his wife visited his house at Roseville. In a private chat next to the sandpit in the backyard, Mitchell says Rudd asked whether the paper had given any thought to proposed changes to the legislation.

He wanted to be certain that Kelly and Norington supported whatever change I suggested. When he had the detail clear, he said: ‘Let’s go back in and have another coffee.’

Once he was elected, Rudd’s honeymoon was short-lived. He traces the first warning signs in the relationship to his first weeks. Mitchell told him that the Australian’s political editor, Dennis Shanahan, would be expecting an exclusive every Sunday so that the broadsheet could lead the news agenda.

Rudd was not keen. He wanted to do the right thing by Fairfax and the ABC and rejected the idea.

Mitchell and Rudd had been friends, dating back to their mutual time in Queensland, when Rudd worked for the Queensland premier and Mitchell was the editor of Brisbane’s Courier-Mail. They were so close that Rudd was godfather to one of Mitchell’s children.

What caused the end of the bromance differs, according to whose version you believe. Rudd puts it down to his refusal to play the News Corp game. Mitchell puts it down to his refusal to attend Rudd’s signature policy forum, the 2020 summit, followed by a strange incident at Kirribilli House in 2008, in which Rudd allowed Mitchell to sit in on a call with the US president, George Bush. Mitchell later briefed his reporters who wrote about it; there is a dispute about whether its publication was authorised by Rudd.

Perhaps both are true in the sense that a breakup often has many explanations. What is not in dispute is that the Rudd government’s relationship with News Corp went downhill as the prime minister defied the Australian’s orthodoxy during the global financial crisis and spent up in an attempt to avert a recession.

“That’s when they unleashed John Lyons, how I was a dysfunctional PM – a four-page extravaganza,” Rudd says. “I was in the throes of handling the GFC in 2008. They unleashed a full fusillade.”

The articles, one headlined “Captain Chaos and the workings of the inner circle”, were, Mitchell says, commissioned by him, because he’d had several conversations with cabinet ministers, and eight months into Rudd’s prime ministership he could see the leader repeating his controlling ways that had brought problems in Queensland.

Lyons also refutes Rudd’s claim as “completely untrue”. He says history has shown that many of Rudd’s colleagues soon afterwards came to the same conclusion – that the first Rudd government was one riven by dysfunction and chaos.

“I’d done many stories looking at prime ministers’ offices – beginning with Bob Hawke’s so-called ‘Manchu court’. That’s the reason editor-in-chief Chris Mitchell asked me to look at who was really running the Rudd office. I was not directed to write either a positive or a negative story, and would not have accepted any such direction,” he says.

The breakdown between News and Labor was absolute by the time Rudd returned to the prime ministership in time for the 2013 election, as evidenced by the Daily Telegraph’s opening front page of the election campaign. “Kick this mob out”, the front-page editorial screamed.

While Murdoch stays in touch with his editors on a regular basis, often it is the editors who are driving the agenda. In this case Murdoch had sent his most trusted tabloid editor, Col Allan, to Sydney to oversee the election coverage of News dailies. With all the subtlety of a sledgehammer, Allan didn’t leave anyone guessing who News wanted as the next prime minister.

Abbott won handsomely in 2013, though how much News can claim credit in the face of a Labor party that was at war with itself is hotly debated.

Next in the series: Inside the News Corp tribe – how powerful editors shape the news