The man who sparked nationwide condemnation by flying a Nazi flag in his backyard has lashed out at A Current Affair.

The resident, only known as Bill, charged towards a TV crew when confronted at his home in Beulah, four hours' drive from Melbourne.

"Put your camera f------ down you c---," an irate Bill yelled.

Beulah resident Bill, who has a Nazi flag flying above his house, confronts A Current Affair's crew. (A Current Affair)

It follows revelations the neo-Nazi's neighbour is an 83-year-old Holocaust survivor, who has been left too terrified to leave his home after the flag went up.

"He's still got the tattoo on his arm from being in a concentration camp," his friend and Beulah local Terry Halmshaw told A Current Affair.

"His parents were gassed, he was next."

The flag is covered in Nazi symbols. (A Current Affair)

Dr Dvir Abramovich, chairman of the Anti-Defamation Commission, said for a Holocaust survivor, seeing a Nazi flag was "as scary and frightening as being threatened with a gun".

"We are living in very frightening and dangerous time," he said of the emerging threat of white supremacy.

Mr Halmshaw said the flag, containing a swastika and other Nazi-related symbols, was "absolutely disgraceful".

Local man Terry Halmshaw said Bill's resident was a Holocaust survivor. (A Current Affair)

"Everybody's furious about it," he said.

"I was going to jump the fence, pull the thing down and burn it in his driveway.

"They're flaunting it in the face of people who were in the concentration camps, it's just disgraceful."

While ACA was filming at the property on Tuesday afternoon, police arrived and pleaded with Bill to remove the sign. After around an hour, he complied, asking officers to take it down on his behalf.

Police finally got Bill to agree to take the flag down. (A Current Affair)

Bill's partner Cheryl Lawdorn previously defended her right to fly the flag because of her German ancestry, The Age reported.

When confronted by a resident about the international media exposure the matter had received, Ms Lawdorn said it was "stiff s--- for the town", according to a video recording obtained by ACA.

"I've put Beulah on the f------ map then haven't I?" she said.

Dr Dvir Abramovich said anti-Semitism was infecting many Australians. (A Current Affair)

Ms Lawdorn and her partner Bill are understood to be relative newcomers to Beulah, only moving in within the past year.

There are currently no laws prohibiting the public display of the offensive insignia, meaning neither the police nor council can compel people to remove the material.

"It's terrifying to think that in Australia 2020, we have individuals walking our streets who are proud worshippers of Hitler," Dr Abramovich said.

"We are facing the mother of all threats.

"In the very month that we are commemorating the 75th anniversary of the liberation of Auschwitz (concentration camp), the virus of anti-Semitism is infecting so many people in our country – it's reached unprecedented levels."

Dr Abramovich is campaigning for Nazi symbols to be outlawed and is hoping a Victorian parliamentary review of anti-vilification laws will see the offensive material banned from being sold or displayed.