The Australian music promotor and celebrity agent Harry M Miller has died aged 84.

The “visionary” entrepreneur died peacefully in Sydney on Wednesday, HMMG, the agency he founded, said in a statement.

“By his side were his long-term partner Simmone Logue, daughters Justine, Brook and Lauren and their mother Wendy,” it said.

Harry Maurice Miller was born in Auckland, New Zealand, on 6 January 1934, the only child of Jewish parents, Sadie and Jim Miller. When Miller was two years old his father, who worked as an indent agent, broke his spine in a fall. He died six years later.

Harry M Miller in 2000. Photograph: Dean Lewins/AAP

“The Jewish community bundled me off to a Jewish orphanage in Wellington ... called Dextons,” Miller later recalled.

His first taste of show business was running a “peepshow” for fellow students – a shoebox with cellophane windows through which he would wind a comic strip: “I used to charge kids a marble.”

After school Miller worked as a dairy hand, on a trans-Tasman passenger ship, as a salesman for knitwear and frypans, and in a restaurant. He began organising entertainment – a sideline that led him to found a record company.

His first signing, four Māori singers known as the Howard Morrison Quartet, enjoyed local success.

His first big-name act was the US jazz musician Louis Armstrong. Miller travelled to the US to approach Armstrong’s manager, Joe Glaser. When Glaser asked, “You got any bucks, kid?,” Miller said he did not.

Miller recalled: “He thought it was so funny, that this kid was trying to buy Louis Armstrong, that he thought I should get it. And he did.”

Armstrong toured in 1963, the same year Miller moved to Australia, founding Pan Pacific Productions with Keith and Dennis Wong, who owned Sydney’s Chequers nightclub.

Miller brought out Judy Garland to do three concerts in 1964. Garland, a drug addict, held up for the Sydney Stadium shows, but the concert at Melbourne’s Festival Hall was a shambles. Miller stood near the back, “copping anger and abuse as the public filed out”.

Miller at the opening night for Alegria by Cirque Du Soleil at Moore Park in Sydney. Photograph: Dean Lewins/AAP

But he rarely put a foot wrong, coupling chutzpah with charm, good looks and a keen business sense.

Other acts he brought out in the 60s included the Rolling Stones, the conductor Arthur Rubinstein, Herman’s Hermits, the Beach Boys and Sonny and Cher.

Miller consolidated his reputation in 1969 by staging the US musical Hair in Australia.

He went to Boston for auditions, hiring the 16-year-old Marcia Hines but not realising she was pregnant. Miller became Hines’ guardian until she turned 21.

Miller followed Hair with the Andrew Lloyd Webber-Tim Rice musical Jesus Christ Superstar, and then Richard O’Brien’s Rocky Horror Show.

In 1970 Miller joined the ruling council of the Art Gallery Society of New South Wales. In 1972, determined to make the society less stuffy, he ran to be its chairman. The campaign was bitter. As Miller recalled in a 2003 interview with the Sydney Morning Herald, there was a feeling against “the vulgar New Zealander who had brought the Rolling Stones to Australia”.

But the campaign of Miller’s rival, Mervyn Horton, collapsed after an ABC television crew recorded his aside that some people were “concerned about Mr Miller’s Jewish background”.

As chairman, Miller reinvigorated the society. Over five years its membership climbed from a few hundred to about 6,000.

In the 70s Miller became a director of Qantas, was on the board of the Meat and Livestock Corporation (he bred Simmental cattle), and organised the Queen’s silver jubilee celebrations in Australia.

“I think I was in the situation where I was asked to, and accepted to be, the bride at every wedding and the corpse at every funeral,” Miller once said.

In 1978 he overreached by establishing a ticketing company, Computicket. Sydney’s Sun newspaper reported: “Harry M can see the day when, with the aid of cable TV, customers will be able to book shows from their living rooms.”

The idea was visionary but within six months Computicket was in receivership.

In 1982 Miller was convicted on five charges of fraudulent misappropriation of $728,000 in connection with Computicket; he spent 10 months in Long Bay and Cessnock jails.

When he resumed his career, he was reluctant to discuss any aspect of the Computicket affair. In a 2010 interview on the ABC’s Talking Heads he recalled he had “cried every day” in jail – but continued to run his clients’ careers from his cell: “You just don’t fall and crumble just because something goes wrong.”

What the media quickly learned, thank God, was that if they didn’t play the game, they weren’t even in the game Harry M Miller

The people represented by the Harry M Miller Group included Lindy Chamberlain, the racing trainer Gai Waterhouse, the fashion entrepreneur Maggie Tabberer, Big Brother contestants, and Judy Moran, the wife of Melbourne underworld figure Lewis Moran. Miller would take up to 25% of earnings.

Miller also negotiated the “cash for comment” deals for the broadcaster Alan Jones.

He attracted criticism for making money from tragedy and sensation. He represented Stuart Diver, who survived the 1997 Thredbo disaster. Miller also handled the funeral of the INXS singer Michael Hutchence.

Miller told Talking Heads that his greatest success was handling Chamberlain’s story when she was freed after being wrongfully jailed over the death of her daughter Azaria.

He described his role as “broker/salesman”, keeping unwanted media at bay until he had sold Chamberlain’s story: “What the media quickly learned, thank God, was that if they didn’t play the game, they weren’t even in the game.”

Miller’s private life was complicated. “My wandering eye is something I have struggled to control all my adult life,” he wrote in his autobiography.

His first marriage, in 1957 to Zoe von Uht, produced a son, Simon, but ended in 1962.

He married an American, Patricia Mitchell, in 1963, but that ended unhappily four years later when she took their two children back to the US.

“She took everything,” Miller recalled in his 2009 autobiography, Confessions of a Not-So-Secret Agent. “I remember coming back from a trip and standing in our empty house. I think the kitchen sink was still there but very little else.”

Harry M Miller and Simmone Logue attend Derby Day at at Royal Randwick, Sydney, in 2005. Photograph: Getty Images

In 1972, after the death of his mother, Sadie, Miller married a 23-year-old vet, Wendy Paul. They had two daughters, Brook and Lauren. Wendy stood by him during the Computicket scandal, running both the Harry M Miller Group and their large Simmental cattle property, Dunmore, at Manilla, in the NSW New England area.

This was followed by an 11-year relationship with the model Deborah Hutton, whom the Harry M Miller Group had steered into the corporate world. (Hutton was initially the public face of the department store Grace Bros.)

In the late 90s Miller met the society caterer and businesswoman Simmone Logue, whom he described in 2010 as “the love of my life”.

He retired in 2009 and handed his business to his daughter, Lauren Miller Cilento.

In 2011 Miller was diagnosed with vascular dementia. He moved into an aged-care facility but spent weekends with Logue. In August 2015, when he and Logue were photographed in Sydney’s east, she was pushing his wheelchair.

Miller is survived by Logue and his five children – sons Simon and Miles and daughters Brook, Lauren and Justine.

In his autobiography, Miller detailed his funeral plans: “It’s the producer in me, I guess.”

He wanted his ashes to be scattered down the cliff in front of his house at Wombarra, north of Wollongong.

“There will be no funeral service as such, but a memorial, preferably at the Capitol Theatre in Sydney. People will get up and speak and sing and they should all have a drink without having to leave the premises.”

The funeral arrangements are still to be confirmed.