The office of the Sydney Morning Herald has been a busy place since Paul Sheehan wrote his infamous piece on “Louise”. The column’s racially charged rape allegations have since unravelled. In the two weeks or so since it was published, it has yielded a video plea from the author to the subject, at least five separate apologies, a correction, a retraction, a press council complaint and an internal investigation by the editor-in-chief of the Herald.

That has now concluded. The editor-in-chief, Darren Goodsir, was said to be distraught at the damage done, and made an impassioned speech to staff on standards. In a public statement he said he had found “unacceptable breaches of fundamental journalistic practice”. And at the end of all this came … a suspension. Sheehan wasn’t even mentioned by name in Goodsir’s statement. As the sun went down last Friday, one of the longest and strangest careers in Australian journalism was diminished, even disgraced. Yet, somehow, it continued.

Even the newspaper’s own staff were incredulous. People inside and outside the paper drew comparisons with the columnist Mike Carlton, who resigned ahead of a reported suspension for swearing at readers in correspondence. Not exactly a cardinal sin. Yet it was the very severity of Sheehan’s repeated transgressions that made him immovable.

“Why do they keep him?” is a question that has been asked for much longer than a fortnight. The usual answer is that he is a useful figleaf. When the suits complain about a lack of conservative writers, “but we’ve got Sheehan” is the response.

The deeper answer is more subtle. Over his decades-long career, Sheehan has made himself an intermediary. He is an essentially tabloid journalist at a broadsheet paper, an insurgent in enemy territory. He is a populist rightwinger in genteel left-liberal surrounds. He is an old-school newspaper columnist who has assiduously learned the language of clicks and shareability, becoming one of the most read and shared commentators in the Fairfax stable.

More than that, he is a conduit between the mainstream media and the fringes, someone who operates somewhere between research and anecdote, who can slide all the way down the line from fact to rumour to falsehood. His articles show a fixation, even an obsession, with race, immigration and crime.

Sheehan’s worldview is shot through with paranoia. Not only has a criminal “other” permeated western society, to his mind, but leftwing do-gooders refuse to face that reality. He believes he is one of the few brave enough to name the crime out loud.

The media generally does a poor job of expressing anxieties about immigration, either reflecting too many or too few. A writer who publishes things that are thought but not said, or said but not read, finds an immediate and grateful audience.

It’s also a valuable one. Before the media grafted for clicks online, newspapers had another debased currency of value: letters. Sheehan was always one for working the mailbag.

The Louise debacle is inexplicable without this context. (For some it’s inexplicable even with it.) Other news organisations are reviewing their own processes in the wake of the scandal, but remain incredulous.

“We just can’t believe it,” says a senior editor at another Australian newspaper. “It’s unthinkable that something like that could end up on page one. It should never – could never happen …

“Sheehan’s reportage has always been a trouble. I remember reading parts of Among the Barbarians [Sheehan’s 1998 book] and thinking, ‘This sounds like bullshit.’ They should have been alert to it.”

And staff at Fairfax were alert to it. Sheehan’s enthusiasm for dubious contacts was well known. Take almost any lunar-right figure in Australia – Cory Bernardi, the Australian Liberty Alliance, Kirralie Smith from Halal Choices – and Sheehan has written a quote-heavy column in their favour, usually rounded out with some kind of oblique endorsement.

Such pieces were widely shared among anti-mosque campaigners and anti-halal groups. They also attracted “hate-reads”, and resonated beyond the fringe, given the anxieties of a white and ageing broadsheet readership. Pieces like this got far greater social media reach than, say, Sheehan’s lighter columns about the dangers of feral cats or the deliciousness of Koko Black chocolate.

But trucking with such characters carried risks. Add in strained editorial resources and Sheehan’s bullying personality and the combination became toxic and unstable. A former senior editor at the Herald describes the fraught culture that developed around the late-career columnist:

Previous opinion editors had been warned that every fifth or sixth piece Sheehan wrote would be unpublishable. They were warned that he would ignore the concerns of anyone but the most senior staff. They were told the pieces would be easy to recognise, and that when they came, the editor should be alerted, who would take care of it. Sometimes pieces were salvaged by some editorial heavy lifting, others were binned at a glance. Former Herald editor Amanda Wilson was particularly unyielding when it came to Sheehan’s sloppiest work. Sheehan was more attuned than any other op-ed writer at the SMH to the power of clicks and comments. He was by far the most-read and often the most commented-upon columnist online, and took great pride in being so. But it was like a drug: the more he obsessed about it, the more pornographic his columns became.

The Louise piece was the most pornographic of all, a racial “outrage” story that recalled the most lurid strains of old-fashioned yellow journalism. Louise claimed she was a former nurse who had been gang-raped and left for dead by a group of Arabic-speaking males. She claimed she had been treated in hospital afterwards but never identified, and that police had failed to take an interest in her case. (No evidence for any of this has yet emerged.)

The column was also a demonstration of how grossly distorted the power imbalances had become between junior and senior staff, and how weak the paper’s fact-checking had grown. Three people in the editorial chain are understood to have queried the story, but were beaten back by Sheehan’s insistence. Both the acting opinion editor, Leigh Tonkin, and the print editor, Heath Gilmore, asked the writer if he had contacted the police. He told them this was unnecessary as it was a comment piece. He was adamant he had all the evidence needed to run the story.

In reality he had almost nothing. He had not even found the Facebook page belonging to Louise when he googled her real name. He had failed to find a video of her speaking at a Reclaim Australia event. The technology was beyond him.

He had not called the hospital where she supposedly worked, or asked how she could have stayed as a patient, unnamed and uninvestigated, for months. He had not inquired after her 79 broken bones, an injury which would put her in a trauma category alongside Evel Knievel. And he had not seen the diaries in which she recorded the incredible attack, for one simple reason: Sheehan had never met Louise.

Louise has a manner of storytelling that quickly escalates into fantasy. That alone should have alerted Sheehan to the fact she was not what she said she was. But he never had a chance to experience it face to face: they only spoke over the phone. They had been introduced by an intermediary, who had met Sheehan and then passed his phone number to Louise. Sheehan was told early on that he would never be able to access the diaries which supposedly contained contemporaneous accounts of her attacks and others. He was not deterred and it wasn’t until the day before the Herald ran his column that he really pressed for more evidence. It didn’t come but he proceeded to publication anyway.

Perhaps even more critical than him not meeting Louise is Sheehan’s failure to speak to police. As I wrote last month, had her story been brought to police, and had the police failed to act, it would have represented a unique failure of the public services. But if he had called the police, not only could they have told him there were no records of any such crime, they may have been able to tell the columnist something else – that Louise was well known to them.

John (not his real name), a former confidant of Louise, was present in 2013 when police raided her flat. Officers had often come looking for her in the past but this time they arrived in more serious numbers. “She told me they were trying to bust her for stealing OxyContin and selling it to homeless people,” John says. Louise told John that the same officers often arrived after calls related to other disturbances, apparently disgruntled they had been unable to make the charges stick. She had a reputation for becoming belligerent when housing authorities were called. Louise really had been a nurse, but was last registered in 2003.

“I had heard the story of this supposed attack many times,” says John. “But it was always nine men, not six. And it wasn’t until after the Martin Place siege [in December 2014] that she changed her story to include Middle Eastern Muslim men. The Man Monis business gave rise to some anti-Muslim sentiment in Australia and I think she was trying to take advantage of that to get more sympathy.”

Sheehan had been obsessed with rapes by ethnic minorities since writing a book – Girls Like You – about the trials of the Skaf and K brothers in the early 2000s. He was convinced these offences were only the tip of the iceberg, that Muslim rape gangs had acted with impunity in the face of impotent, “culturally sensitive” policing. The story of Louise was the corroboration he’d been looking for.

So began the breakdown in journalistic systems at the Sydney Morning Herald. Sheehan was adamant his source was legitimate, and that was enough.

But where did the sense of urgency to go to press come from? There was plenty of time. The story described alleged events 14 years old and there was no danger of anyone else running it. Conducting checks on Louise’s story, perhaps even identifying possible perpetrators, may have even improved the tale, if it was true. And several people weren’t at all sure that it was. But every story in a newspaper becomes urgent if there is nothing else to replace it. Sheehan was not going to let relatively junior staff chip away at his authority. After all, he had written a book on integrity in journalism in the digital age (even though the Herald’s own writers criticised The Electronic Whorehouse).

If that was all, perhaps we could leave the story of Sheehan and Louise here. A credulous columnist with a fixation on racially motivated crimes taken in by a persuasive manipulator, and a news organisation struggling with fact checking and the chain of command in the digital age. But things like this have happened to Sheehan too many times for it to be an accident, and for too long.

It was 30 years ago, in October 1986, that the Herald printed the story that cemented Sheehan’s reputation. Then a news writer, Sheehan made 40 phone calls in a day to get a quote from Malcolm Fraser on the “Memphis trousers affair”. The former prime minister had been found dishevelled, trouser-less and disorientated in the foyer of a hotel. The story, and Fraser’s quote – “I wish I’d never been to bloody Memphis” – entered journalistic folklore.

But just a month earlier, Sheehan had been caught perpetrating a major fabrication. In September 1986, the paper sent him to Punta Del Este in Uruguay, to report on major 92-nation trade talks. He duly began filing reports, describing the negotiations in detail. But Sheehan wasn’t in Punta Del Este at all. He was in fact elsewhere, having a tryst with the woman who would become his wife. Other correspondents were asked why they were missing the colour Sheehan was getting. It was because they were actually present.

The ruse was discovered, and he was packed off to Punta. In the beery, matey world of 1980s journalism, the deception was treated a bit like an outrageous prank.

Sheehan now claims this version of events is wrong. “I reported on the GATT conference at the GATT conference, the only way I could find out what was going on. I recall no problems with the stories.” But this does not match the recollection of those present.

The consequences weren’t serious. They never really became serious. And so a line shifted, and kept on shifting, until no one, perhaps not even Sheehan, knew where it was any more.

After Uruguay came the US, where Sheehan’s ambitions became more serious. He tried to write for the New Yorker and the Atlantic Monthly but found their production protocols glacial and their insistence on thorough fact checking irritating. He also discovered a topic that was to define his future output: race crime. On his return to Australia he described his “own experience with black Americans in the past decade, which includes attempted murder, assault, theft, bottle-throwing, being showered with beer, and being called ‘cracker’, ‘honkey’, ‘motherf…er’, ‘white trash’, and ‘blue-eyed devil’.” That piece, published in 1995 as part of a series called ‘Four stories the US media refuse to tell’, was called ‘The race war of black against white’. It has since become a staple of white supremacist websites. On the largest, stormfront.org, Sheehan’s name appears hundreds of times. The story contains this line: “In the mid-1960s … there was a sharp increase in black crime against whites, an upsurge which, not coincidentally, corresponds exactly with the beginning of the modern civil rights movement.”

Australia in the late ’90s was changing. Paul Keating’s Asian outlook was giving way to John Howard’s white-armband visions, and Sheehan’s views on race found their natural milieu. At least rhetorically, the forces of multiculturalism were in retreat, and Pauline Hanson was voicing the belief that Australia was “in danger of being swamped by Asians”. It was in this environment that Sheehan wrote a series on Asian crime that the then Herald editor, John Lyons, refused to publish.

The claims in the series were so astonishing that Lyons was concerned they might promote racist attacks. Sheehan said a three-month investigation had uncovered Medicare rorting, widespread marriage scams, electricity theft, tax avoidance and Asians calling white Australians “dogs”. But when Lyons pressed for evidence and sources, they seemed to be missing. There was nothing from the Australian Taxation Office, the police or Medicare. Sheehan’s information came from interviews with only a few sources, all unnamed.

He identified an entire Vietnamese family of seven as welfare frauds but hadn’t contacted them for comment. His major piece of evidence for the “rorting” turned out to be a single news story, about the theft of a wallet in Kings Cross, when only a Medicare card was taken. Unconvinced, Lyons killed the series.

“Apart from a few books people had written saying Chinese immigrants were criminals, he was not able to produce a single piece of documentation supporting his claims,” Lyons later told the writer Gideon Haigh. “In my view it was poorly researched. No editor who cared about his paper’s reputation would have published this stuff. Not one of the Herald editors I spoke to thought the piece was of a standard to be published.”

Sheehan published the material anyway, in his book Among the Barbarians. It became a huge bestseller, with more than 80,000 copies in print.

Strangely, Asian crime no longer seems to concern its author. If he still thinks that Asians are culturally primed for totalitarianism and corruption, he now keeps the belief to himself. Colours change as fashions change, and so Sheehan’s concern has moved from black people to Asian people to Middle Eastern Muslims.

The Herald sometimes tried to curb his worst impulses. In 2008 he wrote in the print edition that “thousands of Australians have paid a heavy price for the failed refugee-vetting processes in the 1970s and 1980s, when thousands of parasites who should never have been allowed into the country were approved”. The online copy was quickly amended to read “many people” instead of “thousands of parasites”.

But by this time the Herald’s culture was already running into difficulties. Lyons had moved on and so had many around him. Those with the authority and the knowledge to stand up to Sheehan were thinning; he had outlasted them. Others he ​​manipulated – from the start of his career he had shown a knack for cultivating the affection of older men. Younger women were less impressed.

In 2003 he wrote a bizarre column about the deliciousness of Krispy Kreme doughnuts. It was unusually detailed: “Customers can watch the doughnuts being made behind large glass walls and everyone gets a free Original Glazed, the most popular of the 15 varieties on sale ($1.50 each, or $10.90 for a dozen Original Glazed, or $1.80 and $12.90 a dozen for everything else).”

Sheehan went on to become a significant shareholder in Krispy Kreme, and his longtime friend David Coe became a director. In 2010, when Krispy Kreme Australia went into receivership, the Age’s business columnist Michael Evans wrote: “A maze of companies reveals it is a subsidiary of another company controlled by Kinghorn and John McGuigan. Other shareholders in the structure include David Coe via his private company Monetti, Greg Jones … and journalist Paul Sheehan.”

Sheehan was furious about the disclosure and, at his insistence, senior Fairfax staff scrubbed the reference from one of their websites and removed it from later editions of the paper.

The pattern repeated when Sheehan promoted a magical water with healing properties on the pages of the Good Weekend magazine. The 2002 story made Sheehan a laughing stock among other journalists – he had swallowed wholesale the claims of a cure-all potion, and done almost nothing to check its creators’ credentials. Again, the story was too good to be true:

Miracle water? Can something as simple as this mineral-rich water really combat arthritis, fatigue and osteoporosis … and help you live longer?

The answer was no. It didn’t seem to hurt Sheehan with many readers though, who bought his claims of naivety when Unique Water’s owner fled the country.

But another Sydney Morning Herald contributor was livid. Ben Hills investigated Sheehan’s claims and found they were baseless. He submitted an investigation to the paper in 2005, detailing the many deceptions of Unique Water. But the then editor, Robert Whitehead, sat on the story. It went through an extremely unusual six-week editing process. In the meantime, Sheehan was allowed to file a follow-up piece, in which he insinuated that Unique Water’s founder may have murdered his own wife. Characteristically, Sheehan felt bullied. “I’ve complained to the editor-in-chief about an abusive phone call from Ben Hills,” he told Media Watch.

Hills’ article did run eventually. It noted that “no medical trials had been conducted to test the efficacy of the water” and that it relied on an “endorsement by Sheehan – and the five other people, one dog and one cat (deceased)”.

It described the interplay between Sheehan and the water’s inventor, “a 54-year-old veterinarian named Russell John Beckett, who was quoted copiously, largely uncritically, throughout the article. Sheehan says he spent two months fact-checking the article, producing an 8,000-word commentary containing 42 sources, to satisfy his editors on the credibility of Beckett’s claims. However, much appears to have been overlooked.”

There were huge, glaring falsehoods in Beckett’s story, which Sheehan had put to print intact. As long as the fabrications were coming from someone else, his position was assured. He took leave then returned to work as though nothing had happened.

At times it seemed the paper would back him no matter what. As the federal election campaign ramped up in 2013, so did a notorious rightwing troll on Twitter. His name was Peter Hill, and his repellent postings would not have been worth noting except for one unusual detail. Hill was the estranged, abusive father of a senior Greens staffer and he was trying to ruin her life. He had previously said he would love to see immigrants “rape the shit” out of one of Fairfax’s female writers.

“He was looping in journalists from the get-go,” says the Greens staffer now. “Miranda Devine, Latika Bourke – from the start he wanted to get media attention to the allegations that I had travelled on the public purse.” Sheehan, the writer of last resort, was the only one who took the bait – even as Hill was being banned from Twitter for his abusive behaviour.

The column, published in August 2013, was called ‘An uncivil war: father v daughter, Greens v greens’, and made unsubstantiated allegations that the staffer and Senator Scott Ludlam were using publicly funded trips to facilitate private activism.

Astonishingly, Sheehan’s piece contained examples of Hill’s abuse, followed by Hill’s own commentary. The two were in contact.

Peter Hill has taken the unusual course of attacking his daughter’s political activities in public, via Twitter. In one caustic message, he wrote: ‘I am so sick of people with snouts in the public trough. I am at a loss on your funded trips to the UK (Assange), Japan, USA, Burma. F---. Your carbon footprint must be an embarrassment.’ ‘Shortly after this tweet,’ he told me, ‘I got my account suspended by Twitter.’

An enraged Ludlam wrote to Goodsir. “I find it utterly unacceptable that [Sheehan] has taken personal aim at a staff member of mine, and that he chooses to quote her violent, estranged father Peter Hill as an authoritative source.”

Goodsir responded: “The SMH stands by the story and its main themes. But having said that I generally accept your points about the use of your staff member’s father in the article.”

First the column was rewritten to remove Peter Hill, with a rider added: “This column has been amended subsequent to publication.” The excisions meant it made no sense. On 27 October the paper issued an apology and the piece was finally pulled from the internet. For the staffer, the affair was so traumatic she changed her last name.

Sheehan didn’t respond but on 24 March 2014 he ran a column with the title ‘King of the trolls Scott Ludlam stripped bare’. It began:

There is nothing more popular on the internet than pornography, and that extends to political pornography, the territory of trolls and zealots. Senator Scott Ludlam knows this, and Ludlam is fighting for his survival. He has only just narrowly averted disaster and now he is appealing to the fringe because he cannot appeal to the majority.

The attack was only tangentially related to reality – Ludlam had done not much more than make a rousing speech. The “political pornography” description seemed better suited to Sheehan than his quarry. The columnist had shown no contrition and had suffered no consequences. Now he was attacking one of the complainants as personally as possible. He kept doing what he had always been doing. He had outlasted or overpowered anyone capable of stopping his worst excesses. Any complaint, any whining from ethnic minority councils or social media, was just more evidence of the bravery it took to write his columns.

As a former Herald editor says: “This isn’t just Sheehan’s fault. He and the paper have to share the blame 50:50. His name was on the story so the fundamental errors are his, but the newspaper and the editors are culpable.”

Now their culpability has extended to this. They were given a sensational and unlikely story of race crime. Its creator was a white supremacist-affiliated fantasist known to police, whom their senior columnist had never met, and never vetted in any serious way. And their response was not to balk, or check, or stand their ground. It was to give it top billing.

On this day, at this moment, Sheehan’s aura and history culminated in an awesome failure of institutional memory. A man who had messed up so many times was given one last calamitous benefit of the doubt.

“Hindsight has a ruthless clarity,” Sheehan wrote in his strange non-apology apology for the Louise story. It was at that stage appended to a correction that also wasn’t a correction: it was simply a version of the Louise story with the Arabic-speaking men taken out. And you realised, reading the amended version, that there was no reason for it to be published, that without the sensation of Middle Eastern villains it would never have made the paper. It had no animus.

Louise knew her audience, and so did Sheehan. Hindsight has not yet been ruthless enough.

CORRECTION: Two paragraphs of this story have been altered to reflect that David Coe was not a Krispy Kreme executive at the time Sheehan wrote about the company, and Sheehan only became a shareholder after his column was published.