It's been 50 years since sitting prime minister Harold Holt went for a swim near Portsea, in Victoria, and never returned.

Friends and family members of the late Mr Holt are gathering for a private commemoration service to mark his disappearance and will remember his achievements while in Parliament.

Australia's 17th prime minister served three decades in Parliament before becoming prime minister, only to disappear two years into his first term.

Federal Member for Flinders Greg Hunt said Mr Holt had a strong social and economic legacy including the introduction of a national child endowment scheme as part of the Menzies government.

Sorry, this video has expired Harold Holt was a keen swimmer

Decades of conspiracy theories

But for most Australians he will always be known for his abrupt disappearance on December 17, 1967.

Under the law at the time of Mr Holt's disappearance, an inquest could not be held because there was no body found, but the ruling changed and in 2005 the Victorian coroner found Mr Holt had drowned and his death was accidental.

But the most powerful man in the country does not go missing in the middle of the Cold War without a mountain of conspiracy theories, ranging from suicide to abduction by a Chinese submarine.

Ten years ago, the National Archives released some of the best letters written to authorities in the aftermath of Mr Holt's disappearance.

Harold Holt's disappearance sparked a range of conspiracy theories. ( National Archives )

There was a letter from an American lawyer, apparently written the day after his disappearance, speculating there was a better than 50 per cent chance his death "resulted from expert sabotage, probably foreign".

They might have used "some delayed-effect drug, which he might have got in refreshments on his way to the beach," the letter writer said.

"This would be revealed by expert autopsy, unless it's one of the new disappearing ones."

Others suggested the prime minister was still alive, believing he faked his own death to run off with an alleged mistress.

Or, in true Cold War fashion, had been abducted by Chinese communists.

"I think that Mr Holt was kidnapped and taken away by submarines, and is being brainwashed for political information," one letter read.

"Our enemies know that Mr Holt and president [Lyndon B] Johnson were close friends. A word to the wise is enough — and I shall leave the rest to you and our government."

Sorry, this video has expired A 2005 coronial inquest found Holt died in an accidental drowning ( ABC News: Freya Michie )

The Indian mystic's 'vision'

One of the more bizarre offers of assistance the government received came via the Australian High Commission in new Delhi India, from an Indian "mystic" who claimed to know where Mr Holt's body could be found.

An Indian mystic's claim to knowledge of whereabouts of Mr Holt's body made its way to the Government. ( Supplied: National Archives of Australia )

Apparently it was not the first such offer the High Commissioner had received, but the mystic came with the recommendation of an Indian member of parliament, A.D Mani.

"Mr Mani recommends a Mr Dadi Balsara, a mystic, who believes that he can find Mr Holt's body," the letter addressed to the Department of External Affairs from a representative of the commissioner wrote.

"Mr Balsara telephoned the High Commission shortly after Mr Holt's disappearance and told us of his vision and sought our assistance in pursuing his conviction that he could find Mr Holt's body.

"We have had a number of letters and calls on the same subject which we have ignored.

"But the High Commissioner felt that in view of Mr Mani's position as a prominent Independent Member of Parliament … as well as the claims which he makes about the people who patronise Mr Balsara, he could not completely ignore the request to refer the matter to the Australian Government."

Which is how the Australian Government came to receive the strange offer.

"Please do not smile," the Indian MP implored the government in his letter referring to the mystic's "vision".

"The vision is constantly urging him to go to Australia to make the discovery and he is prepared to go to Australia at his own cost, if no other means can be found for financing his trip," the letter reads.

All he asked from the government of the day were "facilities" to help him undertake the mission.

The government for its part, recommended the High Commissioner write back explaining there had already been a thorough search, and attaching police reports into the disappearance.

But they did say he was welcome to pass on the "details of his visions" to be referred to authorities for further investigation.

Harold Holt and Bob Dyer collect orchids along a north Queensland beach in 1967. ( National Archives )

Australia's 'first 20th century PM'

But some commentators say it is time to remember Mr Holt for his achievements and not the conspiracies surrounding his disappearance.

The executive director of the Menzies Research Centre, Nick Cater, has recently co-authored a monograph on Mr Holt's life and legacy.

"Inevitably, if you end your prime ministerial career so tragically and so dramatically as that, that is the one thing people remember," he said.

But Mr Cater said while Mr Holt was only prime minister for 680 days, the reforms he introduced were "timely, innovative and politically courageous".

"It's an incredible list of stuff that … a lot of people thought were the work of later prime ministers, you know, particularly Gough Whitlam," Mr Cater said.

"Holt pushed further into Asia, deepening ties with nations outside the Commonwealth."

"Holt ended the White Australia policy, or began the process of dismantling that, which was at the time a very difficult political thing to do because they weren't certain how people would react to it … they didn't make a big noise about at the time, they just quietly did it.

"The referendum, giving rights to Aboriginal people happened under his first year and that was 90 per cent in favour so that was an incredible result.

"They introduced decimal currency almost without a hitch.

"A lot of other things including the Australia Council for the Arts … and the expansion of higher education, all these things sound like they came much later but he was doing these in '67 and it was a big change after Robert Menzies," Mr Cater said.

Mr Cater said Mr Holt was Australia's first 20th century prime minister.

"He modernised the office."