Police are digging up the grounds of a former Ballarat orphanage, in a search for the remains of children.

Police began the search on Monday at 200 Victoria Street, which was once home to the Ballarat Orphanage, the Ballarat Children's Homes and finally Damascus College.

The site was sold to developers in 2011, who applied to have it rezoned for commercial and mixed use.

Ballarat mayor John Phillips confirmed that the allegations had been raised at a council meeting in 2013, when the council were considering plans to redevelop the site.

He did not confirm that the council then referred the claims to the coroner and the Government.

The site is under police guard.

"The investigation relates to alleged activities that may have occurred whilst the orphanage was operational, prior to its closure in 1968," Victoria Police said in a statement.

Detectives are working closely with the coroner and forensic scientists and the Government, police said.

Care Leavers Australasia Network (CLAN) vice-president Frank Golding lived at the orphanage for 11 years, from the age of four, until 1953.

'Children just disappeared overnight' at orphanage

Mr Golding said that the allegations had not come as a surprise to him.

"I think that part of it is that children just disappeared overnight, sometimes they would be ill and they simply wouldn't be there at roll call the following morning," he said.

"We had no information on what happened to those children, there were never any formal farewells ... so I think in an era where children were also quite viciously punished at times and had illnesses which were neglected, that leads to speculation, and it's not just confined to the Ballarat Orphanage.

"In that sense I think it's very good that the police are doing an investigation in this instance."

Mr Golding said he was at the council meeting in 2013 where two women, Phylis Read and Edith Orr, raised the issue of children being buried on the site.

"It was in the context of whether or not the council would agree to the demolition of the historic buildings that remain on the site to allow for private and commercial development," he said.

"The women then raised the issue of the need to just slow down and conduct and investigation into those claims that children were buried on this site.

"I don't recall the details, but it was simply that there were children who had died on the site and they were buried on the land and that the land should not be re-developed ... with mass digging there ... without police and other experts having an opportunity to test out these claims."

Mr Golding understood that Ms Orr and Ms Read, who are Aboriginal women, were particularly concerned about Aboriginal children's remains.

He said that in his time, 10-15 per cent of the children at the orphanage were members of the stolen generation, whose parents did not know where they were.