What the politicians and the journalists have told you is that in his diaries Mark Latham has written a bitter, biased, scurrilous, self-centred, self-aggrandising and ultimately self-destructive book. They are right. What they will not tell you is that, in addition, Latham has produced the most intelligent, perceptive, honest and absorbing book about Australian politics since Don Watson’s Keating masterpiece, Recollections of a Bleeding Heart. The publication of The Latham Diaries raises many troubling ethical questions. Yet in the end these seem less important than the contribution he has made to an understanding not only of the dilemmas of federal Labor during the Howard years but, more deeply, of the sicknesses of the democratic political culture in the age of material plenty. Once Latham has been discredited, and perhaps destroyed, some citizens will notice that the Diaries – for all their grandiosity, occasional cruelty and vituperative madness – are considerably more important in their implication than they have been led to believe.

Latham began to write his diary as a backbencher in 1994. His notes became interesting after he was given the shadow education portfolio by Paul Keating’s successor, Kim Beazley, following Labor’s landslide defeat of March 1996. Latham was at that time one of the party’s only thinkers. At the centre of his political vision was his admiration for Keating’s pro-market revolution and for his audacious, Napoleonic political style.

Yet already he differed from Keating in two main ways. He was sceptical of Keating’s top-down centralism and his uncritical support for the welfare state. And he realised something that Keating could not see, that his government had been destroyed through the force of a new social condition Latham identified as “downward envy”, the “old Australian” resentment at the supposed special favours bestowed by the state on “minorities” – artists, ethnics, refugees, single mothers, Aborigines. As early as July 1995 Latham understood that due to the force of downward envy the voters in his Werriwa electorate were in a “punishing mood”. Even before Pauline Hanson’s rise, Latham saw the political salience of the distinction between cosmopolitan “elites” and the “mainstream”. In the area of cultural politics his grasp was far superior to that of Keating, his hero.

Latham was an enemy of the Beazley leadership from the first. At an early Caucus meeting Beazley apparently described his political strategy as pissing on the government and then pissing off. Latham thought this opportunism of the most disabling kind. At one moment in the Diaries he likens Beazley to a general who orders his troops to march but neglects to tell them in which way. Later he discovered in the line from Alice in Wonderland – “If you don’t know where you’re going, any road will take you there” – an even more precise description of the Beazley leadership style.

Yet there was far more to Latham’s contempt for Beazley than this. As the most faithful Keating economic rationalist on Beazley’s frontbench, who believed that history had a direction and that he knew what it was, Latham was quick to spot and even quicker to condemn the dim flickers of interest shown by Beazley and his shadow treasurer, Simon Crean, in tariff pauses and government measures to support ailing businesses. The “retro-economics” of Beazley’s quarter-hearted flirtations with industry policy and protectionism disgusted Latham. “Make no mistake,” he records on August 1, 1998, “Beazley and Hanson are singing from the same economic song sheet.”

With characteristic grandiosity, Latham believed he could save Labor with his monster pro-market book Civilising Global Capital (“140,000 words, I must be a maniac”), which attempted to marry neo-liberalism with a kind of social democracy. With characteristic naivety, he was dismayed when told by right-wing machine man Leo McLeay that most members of Labor’s backbench were unlikely even to have read it. With a party as hopeless as this, what was to be done? Anti-intellectualism, Latham believed, was rife. When Latham delivered a speech quoting Francis Fukuyama on social capital, Ralph Willis asked him who exactly was this “Fucking-yama”?

Crusading in favour of the market did not, to put it mildly, make Latham a conservative. Throughout his political life two contradictory impulses – neo-liberal reformer and anti-Establishment revolutionary – conducted a never-ending Indian arm wrestle for his soul. During a visit to Bob Hawke’s Sydney Harbour mansion his thoughts turned to the modest cottages of John Curtin and Ben Chifley. As Bob and Blanche’s masseur arrived he was only too happy to be able to escape. Latham could never overcome his hatred for privilege or even his impulse to destroy. These were strange feelings for a self-styled neo-liberal civiliser of global capital. On the day some left-wing trade unionists staged a riotous demonstration at the doors of the Commonwealth parliament, Latham confided to his diary: “Part of me wanted to join in, to grab a sledge-hammer and rip and tear against the Tories and their tin-pot Parliament. But I walked away, all neat and respectable.”

In July 1998 Latham’s separation from his colleagues deepened in a different way. On behalf of ALP secretary Gary Gray, his mentor Gough Whitlam warned Latham of rumours circulating about his sexual harassment of a female parliamentary staffer. Latham believed that although the false rumours originated in John Howard’s office, they were being fanned by Gray and by the Victorian Right’s factional chief Robert Ray. Glenn Milne – a political journalist he particularly despised, known in the diary simply as the Poisoned Dwarf – said the rumours had reached him from “people close to Beazley”.

Until that time Latham had believed at least in Beazley’s moral credentials. He reckoned Beazley would have made a fine priest. Even that belief was now gone. “People think Beazley is a big angel, but behind the scenes he’s in the gutter.” Beazley’s supposed ignominy and hypocrisy became for Latham now an article of faith. He could not abide the political culture of perpetual, anonymous, behind-the-scenes backgrounding. He came to believe that inside the ALP the working-class code of honour was dead.

Campaigning in the 1998 election was a tale of concentrated dreariness – “tedious event after tedious event with candidates who can barely string two words together” – leavened only by the occasional injection of farce. In Burnie, on behalf of local MP Sid Sidebottom, he was asked to judge a lasagne competition. Having to have an opinion on purple lasagne was even more demeaning than Keating’s characterisation of popular politics as tripping over TV cables in shopping malls. By this time Latham was ready “to pull the pin on the whole stinking wreck of a Party”.

Although he did not pull the pin, not yet, Latham did now return to the backbench. He gloried in his splendid isolation. He saw himself as the Caucus Cassandra. He called himself a “one man think tank”. He believed he was the only Labor politician who had kept his passion and the only innovative thinker on the Labor side. “We are a party of centralists. And then there’s me.” Occasionally he likened himself to another politician who had gone to the backbench 60 years before, Winston Churchill.

To help pass the time he needled journalists. Believing Gerard Henderson to have made a career out of pedantry, exposing trivial errors in other people’s work, Latham pointed to the errors in a “femocrazy” book by his wife. With Henderson, he felt, if you write 50 words of criticism you could expect 50,000 in return. He had made an enemy for life. Latham accepted a dare from a parliamentary staffer to call Janet Albrechtsen – one of the strident right-wing columnists in the Murdoch press, who Latham referred to as the Dancing Bears – a “skanky ho”. Even more rashly, after a 60 Minutes exposé on Keating’s piggery, Latham launched a counter-attack on Kerry Packer during an adjournment debate. He knew the risk. But he was nothing if not reckless. Should he now bite his tongue? “Stuff it,” he wrote on August 12, 1999, “it’s too much fun baiting the tossers.”

As a consequence of his Packer attack, Latham struck up a friendship with Keating, who was searching for a Labor politician who could destroy Howard. Keating encouraged Latham to believe he was suitable. Their political friendship was founded on the conviction that Beazley was hopeless, a “boredom machine”, altogether lacking the capacity for indignation and the instinct for the kill. Latham was fascinated by the Keating version of campaign politics as a blood sport. “I see it like the National Geographic ad,” Keating told him, “where in slow motion the wildebeest grabs the lion on the arse, with blood and fur and shit flying everywhere. That’s what the mob wants in the last fortnight of the campaign.” Keating convinced Latham that those who made a difference in politics were the maddies. Keating described himself as “mad as a cut snake”. He flattered Latham with the thought that he was the same. For a long time Latham was entranced. “Paul’s a genius, an instinctive feel for how the show works … it’s all policy analysis and advocacy, the flow of ideas and personalities.” Eventually he became somewhat disillusioned. There was something unhealthy about Keating’s anti-Howard obsession, what Latham called “his 100% bile”.

While smouldering on the backbench, Latham’s most serious endeavour was to begin to invent a new Labor philosophy and strategy suitable to the global age. In a lecture delivered at the University of New South Wales he argued for a new anti-statist form of collectivism, based on mutualism and the empowerment of communities. From the American political sub-theorist Dick Morris, he embraced the idea that the traditional positions of Left and Right could be transcended by a micro-politics of small-scale initiatives which, with an extraordinary condescension, Morris called “stooping to succeed”.

Latham had noticed that the most crucial contemporary political battles now turned on social values. In a lecture to the Menzies Centre in London, he outlined a new way of thinking about Australian society: a division between “insiders” and “outsiders”. Insiders were the two groups who, at this time, the boy from Green Valley most despised – the Tory Establishment and the Progressive Establishment, Hugh Morgan and Phillip Adams, as it were. Outsiders were the two groups with whom he most identified, the winners and losers in the age of globalisation – on the one hand the “aspirational” middle class, and on the other the real strugglers, the single mothers, the unemployed. From a psychological point of view, the insider–outsider division enabled Latham to reconcile the neo-liberalism of his intellect and the passionate egalitarianism of his heart. By combining his new collectivism and his insider–outsider distinction, Latham genuinely believed he had created the framework for Labor’s revitalisation and for its victory in the culture wars.

The election defeat of November 2001 strengthened Latham’s anti-Beazley faith. With Tampa, Beazley’s small-target strategy had failed. Beazley had erred in opposing Howard’s first Tampa bill. Armed coercion was the legitimate instrument of the sovereign state. That so many Labor insiders now supported an “open door” refugee policy was for him a sign (God help us) of the party’s ethical bankruptcy. Nor was it a good sign, in a booming economy, that the more affluent the electorate the greater had the swing to the Coalition been. Labor’s task was “to appeal to the fastest growing group that still carries Labor values – the new aspirational class … Our people now have a taste of success and they want more.”

Having witnessed the predictable collapse of Beazley’s small-target strategy and having invented his own alternative strategy, Latham returned to the Opposition frontbench. He was determined to prove his critics wrong by showing total loyalty to new leader Simon Crean. For two years this was precisely what he showed. At this time, Latham tells us in his diary, he had no clear political plan mapped out for himself. In part of his mind he thought the idea of eventually leading Labor preposterous. In another part he thought it might just be a possibility in the long term. And in yet another he wondered whether being prime minister or having a family was the more valuable human goal. Latham loved his second wife dearly. He was utterly besotted by his two young sons, in part because he saw them, after his battle with testicular cancer, as an unanticipated gift, and in part because of the pain his remoteness from his own father still caused in him. He was deeply affected by the words of former Howard minister Warwick Smith. “A day you miss with your kids is a day you never get back.”

Despite his free-market enthusiasm and his crude, old working-class anti-refugee prejudice, Latham identified with the historical heroes of the Labor Left, Lionel Murphy and Jim Cairns. Of all the Labor premiers the one he most admired was also once a man of the Left, Jim Bacon. At Bacon’s funeral service Latham “wept and wept”. “As I get older, these things seem to affect me more severely. Is it that or my empathy with the man, a rebellious Labor larrikin?” The two years of Crean’s leadership were the years of the Howard government’s embrace of every twist and turn of George W. Bush’s policy for fighting the war on terror. Eventually, Latham discovered in his opposition to the Howard government’s foreign policy the perfect political outlet for his radicalism.

Inevitably this brought him serious enemies. As early as June 2002 Latham called Howard an “arselicker”. At much the same time he launched an attack on the “new political correctness” of the right-wing columnists in both the Murdoch and Fairfax press. Paul Kelly, of The Australian, had spoken privately to Latham about the second rate-ness of the Murdoch management in Australia. When he met with John Hartigan, News Limited’s CEO, he learned what Kelly meant. Despite Hartigan’s best efforts Latham was unwilling to retract. Having already crossed Packer over Keating, he had now displeased Murdoch on the question of foreign policy and his Dancing Bears. As Latham understood it: “The Evil Empire will exaggerate and manipulate anything to get at me now.” In this prediction he was, of course, right.

On January 23, 2003, Latham took part in what he called a “hate session” with Laurie Brereton and Joel Fitzgibbon on “the folly of invading Iraq”. Even Brereton was “surprised when I said we should rethink the American Alliance”. Latham bitterly opposed those he called the party’s “Big Macs” – Beazley, Rudd, Swan, Conroy, Ray – with their circumlocutions and endless stream of incomprehensible caveats. “We might support an American invasion not sanctioned by the United Nations if the majority of Security Council members vote to invade but one of the permanent members exercises a veto. Try selling that jumble of words in Western Sydney,” he exclaimed in his diary. Latham favoured for Labor the simplicity of the Bob Brown line. “Bush is an imbecile and Australia should not be part of this war.”

In April 2003, following a Bulletin interview with Beazley which the political nation interpreted as the first shot in a leadership challenge, Latham exploded. “The avuncular Beazley” was, as he had long known, “a dirty dog”. Latham wholeheartedly supported Crean in the first Beazley challenge. He was offered and took the shadow treasury portfolio. Latham’s thinking about his political future was now clearer. If (or really when) Crean failed at the next election, the choice would be between the two most likely members of the next generation, Wayne Swan and himself. He thought he most likely had the numbers.

In the spring of 2003 Crean’s leadership began to collapse under the weight of impossibly bad internal polls. Beazley and Latham were possible replacements. When the factional chiefs moved against Crean, Latham won the narrowest of victories, and confided to his diary: “Unbelievable … When they grow up, I’ll be able to tell Oliver and Isaac how I started out in a public housing estate and became Leader of the Opposition. Maybe Prime Minister.”

Latham developed a promising plan for taking power within a year. He would win the culture war with a values rhetoric appealing to outsiders: “Opportunity for all, aspiration and social mobility, mutual responsibility (hard work), community building (mutualism), and public participation (reinventing democracy).” He would transcend the old politics of Left and Right with a Dick Morris-inspired “stooping to succeed” program – reading books to children, mentoring boys, strengthening communities. He would blunt Howard’s wedge by re-defining the symbolic issues of the Left in a language the outsiders understood: “We want the children out of detention centres, not because of UN Conventions, but because we care for children … We want to say sorry to the Stolen Generations, not because of indigenous rights, but because of family values.” He would wrong-foot Howard with an attack on a symbol of the insiders’ club – the profligate superannuation scheme for federal politicians. He would prove Labor’s environmental credentials and consolidate Greens preferences by recruiting Peter Garrett and visiting Tasmania’s old-growth forests. He would attack the fawning Howard foreign policy by promising to bring Australian troops in Iraq home by Christmas. Whether or not Australians wanted their country to become an American colony was what the next election would, most fundamentally, be about.

At first the Latham leadership brought cheer to the Labor Party and fear to the Coalition. The polls were for a time excellent. As late as May 17, 2004, Latham wrote of “the smell of victory still in our nostrils”.

He had underestimated both the immensity of the task and the ruthlessness of his enemies. Throughout 2004 one anti-Latham story followed another. Because of similarities in a speech he was accused of plagiarising Bill Clinton. It rattled him more than it ought. Even though his opposition was vindicated by the course of events, the Murdoch press pursued him mercilessly over Iraq. Often inaccurate gossip, inspired by old enemies from his years as Liverpool mayor, kept surfacing. Most dismayingly, he had to battle scurrilous reports concerning his private life put about by Liberal Party members and by his estranged former wife, known simply in the diary as Gwyther. Has any political leader in the history of Australia had to endure a prolonged salacious campaign like this? Latham came to think of Australia as a nation of voyeurs. In late-August 2004 he suffered an acute bout of pancreatitis. “Politics,” he wrote, “is killing me.”

During the 2004 election Latham was too busy to make entries in the diary. When he returned to it his mood was one of complete despair. Latham by now had lost the election and his capacity for self-criticism. He lashed out against his chief of staff, Mike Richards, and his campaign director, Tim Gartrell, who had pointlessly and against advice directed advertisements against Treasurer Peter Costello while failing to notice the damage being done by the central Howard government theme – the interest-rate scare. “One thing I underestimated,” he added, “was the capacity of the Liberals to use large parts of the commercial media as an extension of their campaign.” Latham now believed even more vehemently that the Labor Party had become completely dysfunctional. By a process of cell division, the original three factions of the 1980s had sub-divided into 30 micro-blocks. “It’s a funny thing to admit,” he wrote a fortnight after the election, “there are so many sub-factions, personal intrigues and pay backs, that even I don’t know how they all work.”

On December 20 he penned an incantatory entry of overwhelming pain and psychological strangeness.

Prying eyes everywhere we go, people staring for what feels like an eternity … It is not natural to have two men in officewear marching towards your family on the beach, barking questions and taking photos … It is not natural for your wife and two children to watch some clown on national TV reaching under his chair for a non-existent copy of a buck’s night video supposedly featuring their husband and father … It is not natural to have people you have never met stop you on the street, talk to you as if they have known you all your life, and then try to cuddle your children … It is not natural to be on the Truman Show.

His vision was dark and yet still penetrating. He had lost faith in the Australian people, who had abandoned participatory politics. On the relevant night more people watched Australian Idol than the election debate. His great white hope, the aspirational voters, had been hopelessly corrupted by individualism and greed: “People live in their highly geared McMansions, on $60 or 70,000 a year, couple of kids at a non-government school, and they say to the politicians, ‘I’m a real battler, help me, make me secure before helping anyone else …’ It is the rise of the material/me society – no mention of others, the poor, community concerns.” He was even willing to look more kindly on the formerly despised cosmopolitan elites. “I’ve bagged these inner city types over the years, but at least they have a sense of common good.” His insider (bad)–outsider (good) portrait of society had collapsed.

In the depths of his despair Latham was a man waiting for a final straw. In early December, when he was told about a completely false Labor rumour that he had had to be pulled off new MP Kate Ellis, the straw arrived. Formally his leadership was destroyed by his Asian tsunami silence. In reality he had already given up. Latham severed his last emotional ties with Labor and his only extant political heroes after he learned that Keating had shown indifference to his illness and that, even before his resignation, Whitlam was busily lobbying for his replacement in the seat of Werriwa. Latham retreated to the world of the family. He had in his locker only one last shot – the publication of his diaries. The newspaper rights were sold to his most remorseless enemy, the Murdoch media. Even before publication, The Australian closed in for the kill. Its anti-Latham editorial of Saturday, September 17, is among most vicious I have ever read. As Latham must have anticipated, his weapon had turned out to be a suicide bomb.

One element of Latham’s insider–outsider vision still hit the mark. Very many Australians outside the political class are instinctively aware of some essential phoniness in the way the insiders – politicians and journalists – play the game of party politics in the age of spin. Whatever his faults, one thing that differentiated Latham from other political insiders was his unwillingness to play by the rules. On May 27, 2003, he produced a brilliant short analysis of what was going wrong:

The more I see the press gallery, the more I struggle with them. There is a huge disconnection between the electorate and the gallery. The public is deeply cynical about the system – the whitebreads, the polly-waffle etc. Yet the media/gallery is the funnel by which these artificial messages and processes are conveyed. They see politics as a game and not real life, and encourage the MPs to play it. Some rules of the game: punish politicians who are “off-message”, even if they are telling the truth. Politicians should always focus on their opponents, thereby “applying pressure”. Politicians should only create narrow points of difference with each other, and milk issues at the margin. And finally, unorthodox politicians should be stigmatised as “mad”. These are the rules of the parliamentary game. But it is not the game the Australian people want their parliamentary representatives to play. It is, therefore, thoroughly predictable that the public would despise politicians and journalists alike. And they do with a passion.

After his election loss Latham felt he had nothing left to lose. He did more than leave the political club. He decided to broadcast its secrets to the world. For the club members this represented an unspeakable betrayal. Rather than ask each other about the serious and specific allegations Latham had raised – about politicians’ dirt files, about journalists’ duplicity, about their collusion in the trivialisation of democratic politics – the members of the club instantly closed ranks. Latham had tried to tell the people that the Emperor had no clothes. Clearly, he was unhinged. He had to be destroyed.