Cheviot Beach near Portsea where Harold Holt went missing. Credit:Eddie Jim Waves rear up from Bass Strait and bash themselves against rock outcrops. You can discern rips sucking the spent water back out to the open sea, clear as fast-flowing rivers. A swell boils over underwater shelves. There is an old haunting here, one from 80 years before Harold Holt transformed into a ghost. Cheviot Beach takes its name from a ship wrecked there on the night of October 19, 1887. The SS Cheviot, bound for Sydney, lost its propeller as it drove through the heads out of Port Phillip Bay, and though the captain ordered sails to be set and anchors dropped, a howling south westerly wheeled the ship around Point Nepean and forced it on to rocks off the shore. Though rescuers brought 24 passengers and crew to safety, the ship broke up before dawn, trapping many and drowning 35.

Beneath the turbulent waters of Cheviot Beach lies a memorial to Harold Holt. If you were drawn by Cheviot's hideous history and tempted to ignore the official ban on scrambling down the hillside, a sign on a gate locked across the old track to the beach offers a more compelling deterrent. Warning. Unexploded Bombs. Indeed, what happened down there on the beach and within its restless waters on December 17, 1967 – a Sunday, just as it is these 50 years later – hit Australia with the impact of a bomb exploding. A plaque at Cheviot Beach. Credit:Eddie Jim An Australian prime minister went swimming ... and was never seen again?

How was such a thing possible? Nothing remotely like it had happened to a nation's leader anywhere in the world, before or since. Cheviot Beach near Portsea. Credit:Eddie Jim Cheviot Beach is only six kilometres beyond the last of the multi-million-dollar holiday homes of Portsea, but it's about as isolated as lonely beaches go. It's known to older locals by another name: Quarantine Back Beach. Back beaches along the Mornington Peninsula are those facing the wild open waters of Bass Strait, rather than those on the other side of the peninsula, on sheltered Port Phillip Bay. The peninsula is such a narrow isthmus here you can stand above Cheviot Beach and look back over your shoulder and see ships plying their way across the calm bay. Divers search rock pools in Cheviot Bay. Credit:Staff photographer

But even today you can't drive the last kilometre or so to Cheviot. It's within the Point Nepean National Park, once a bulwark against invaders and disease. It was a defence fortification – hence the unexploded bombs – and a quarantine station. It was the sort of place a prime minister who wanted privacy could leave the world behind. In 1967, it was closed altogether to the public. Searchers look for the body of Harold Holt amongst rocks at Cheviot Bay. Credit:Staff photographer But Holt was a prime minister. He had a beach house in Portsea. All he needed to do was climb into his big 1962 maroon Pontiac Laurentian, head down Defence Road – a ribbon of tarmac between forests of mahogany and moonah and coastal teatree – offer a nod and a wink to the military guard on the gate, and he was gone. He was famously fond of pitting himself against the sea, spear fishing and getting himself photographed in a wetsuit surrounded by pretty young women.

Navy skin divers prepare to enter the ocean at Cheviot Beach to search for Harold Holt in December, 1967. Credit:The Age He didn't like protective police cramping his style. When he motored down Defence Road on December 17, bound for Cheviot Beach, he had no bodyguards accompanying him. His guest for the ride was a woman named Marjorie Gillespie. Holt had spent a couple of days attending dinners, playing tennis, dropping in to parties –- the sort of thing one does at Portsea in the summer – and Mrs Gillespie, whose family had a beach house within a few yards of Holt's place, featured in a number of these events. Holt and Gillespie were secret lovers, though Australia wouldn't know for a very long time.

Holt's wife, Zara, 20 years afterwards, said she knew all about it and wondered whether she should have told "Marj" that she was just one of a list: Holt was "was having affairs everywhere". With Zara back in Canberra at The Lodge, Holt and Marjorie headed first to the very tip of Point Nepean, the old fort, to watch British round-the-world yachtsman Alec Rose sail his Lively Lady in to Port Phillip Bay. Trailing Holt and Gillespie in a separate car were Gillespie's daughter Vyner and two friends, Martin Simpson and Alan Stewart. The party soon moved to Cheviot Beach, the wild "Quarantine Back Beach". Marjorie Gillespie, interviewed later by police, said "The surf was higher than I had even seen it..." And Alan Stewart: "I noticed that the tide was very high and the surf was very turbulent, in fact the biggest I have seen on that beach."

Martin Simpson: "I went into the water almost knee-deep and there was a fairly strong undercurrent, so I just splashed around without going in too far." And Holt himself, according to the Gillespie party's statements to police? "Mr Holt at this stage did mention in conversation 'I know this beach like the back of my hand' and he then went on to say how the tide was unusually high." The heaving cove was awash with driftwood of all sizes, each of the party stated. Regardless, clad in a pair of tight blue swimming shorts he'd donned behind a rock, the prime minister of Australia plunged right in and swam out, his progress accelerated by one of those underwater rivers, a rip.

Gillespie was the last person to see Holt alive, according to the police report. "She had watched Mr Holt continuously from the time he had entered the surf, and she saw the water become very turbulent around him very suddenly and appeared to boil and these conditions seemed to 'swamp' on him. He was not seen again." After decades of conspiracy theories and supposition – was it suicide, or murder, or did a Chinese submarine spirit him away? – a coroner finally found in 2005 that Harold Holt had drowned accidentally while swimming. Well, of course. A powerful man in his late middle-age, showing off to his girlfriend, wanted to prove he could handle the toughest surf at his own secret beach, one he knew "like the back of his hand". And he over-estimated his ability.

It was the most Australian vanishing act. Ever.