A former resident of a children's home in Ballarat says she has been left devastated by a decision to drop sex abuse charges against her overseer.

Cheryl Bihari was seven when she was packed up with her four siblings and taken to the Ballarat Orphanage in 1963.

Her mother was considered "unfit" for the role and her father was a criminal.

"Mum was in bed drunk and they decided that we had to be taken off her," Ms Bihari said.

"My father was there and he asked if he could take us children. The answer was 'no'."

The superintendent — the man in charge of the home — was Hylton Sedgman.

"They'd have a trolley that you'd put your plate on and there was a tin, called a swill tin, for the pigs," she said.

"And he would inspect those all the time to see that there was nothing in there for those pigs— if there was, whoever put it in there would get punished."

Soon after Ms Bihari arrived, the superintendent sought to reprimand her for talking in bed.

Cheryl Bihari was seven years old when she was taken to the Ballarat Orphanage. ( Supplied: Cheryl Bihari )

"He came in the back door and he made me get out of bed, and go into the common room, where he lifted up my nightie, sexually assaulted me, and yeah, I had to hold his penis," she said.

"I was petrified of him.

"Then he put me over his knee and slapped me so hard and he said 'don't you ever, ever tell anybody', and he said 'now get back to bed'."

Ms Bihari cried herself to sleep, and for more than 50 years did what she was told.

"Nobody knew. I would not tell because I was embarrassed. I thought it was my fault because I was talking [in bed]. I'm thinking 'this is what's going to happen every time I talk at night'," she said.

In 2014, a detective from Taskforce SANO knocked on her front door asking about Mr Sedgman.

"And I broke down and said: 'He's dead'," she said.

"He said: 'No, he's alive'."

She agreed to give a statement and became one of seven alleged victims — six women and one man — to form part of a criminal prosecution for historical child sex abuse.

Dropped case 'a kick in the guts'

Mr Sedgman was facing more than 30 charges.

"'I want to look this man in the face', I said. And I want to be able to tell him, what he did was wrong," Ms Bihari said.

About 60 former residents of Ballarat Orphanage received settlements for being abused by staff. ( Supplied )

The Royal Commission into Institutional Responses to Child Sexual Abuse excluded the Ballarat Orphanage from its public hearings because Mr Sedgman was due to face court.

But after two years of delays, prosecutors were forced to drop the case, due to the 87-year-old's ill health.

"We were going to court, and I got a call from the DPP; they said 'Cheryl, we have a problem, it may not go through'," she said.

"I just said 'there's another kick in the guts isn't it? You have got all of us to come out and bring all of this forward, to go to court, and now it's not going to happen'."

She said she had been robbed not only of a public forum to air the horrors at the home, but of any prospect of justice.

"He gets to go into a nursing home or a retirement village," Ms Bihari said.

"That's the worst thing. I said 'he gets away with it, we all suffer'. No, I'm not happy."

Lawyer Penny Savidis has negotiated settlements for around 60 former residents of the Ballarat Orphanage, involving multiple perpetrators.

"Our complaints span over about a 40-year period, so there are a number of individuals who have been identified by our clients as alleged offenders," she said.

"It's fair to say that abuse at the Ballarat Orphanage was systemic in nature over a lengthy period of time."

Marking out abuser's name 'the right thing to do'

Ms Savidis said she's not aware of a single criminal conviction against the home's former staff.

"Having a conviction against someone is quite a strong recording of guilt, and it's unfortunate for many survivors that's not an option, either because the alleged offender is deceased or because the prosecution decide not to proceed for whatever reason," she said.

Child and Family Services is the legal successor to the Ballarat Orphanage.

Mr Sedgman's name is printed in gold-plated lettering on an honours board in the organisation's current-day conference room.

Developers have bought the derelict orphanage site but it has been left untouched for years. ( ABC News: Charlotte King )

Chief executive Allan Joy decided earlier this year it was time to draw a line in the sand, and a red mark has now been placed through the name.

"We thought it was inappropriate to have a name on the honour board with somebody who's allegedly responsible for the serious abuse of many children in care," Mr Joy said.

"If we came across any others, we'd do the same thing — it's just the right thing to do."

Weathered from the lifelong impacts of abuse, it's little comfort for the victims.

"I don't trust people. I really do not trust people," Ms Bihari said.

"[I've] just put up a big brick wall, [so] that nobody would hurt me."

Ms Bihari has three photos from her childhood. The images were taken when the seven-year-old and her siblings went on short excursions to the Eureka Stockade park.

"I like that because it's something I haven't got many of," she said.

"And I'm smiling, and I'm with my siblings, you know and they were really important to me back then, that I have them happy.

"And no one's hurting anybody."