Yet another probe into Victoria's youth justice system is under way, just as human rights lawyers start a new legal challenge against the use of adult jails to house teenagers.

Victorian Ombudsman Deborah Glass will investigate what needs to be done to ensure the state's prisons and youth justice centres conform to an United Nations protocol.

The UN's Optional Protocol to the Convention Against Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment will be ratified by the federal government in December.

"I am undertaking this investigation in order to help all parties involved in OPCAT gain an understanding of where the challenges to Victorian implementation of the protocol lie," Ms Glass said in a statement on Monday.

Ms Glass is not actively seeking submissions for the investigation, but people wanting to make contact are welcome to.

The youth justice system has been under heavy scrutiny after months of riots rendered parts of the Parkville centre uninhabitable, a mass break-out from the Malmsbury centre in January, and a legal fight over whether the youths can be housed in an adult prison.

Some detainees are now being held in the Grevillea unit of the Barwon maximum security prison despite court challenges.

The latest legal challenge began in the Supreme Court on Monday.

The Human Rights Law Centre is fighting the state government over the prolonged solitary confinement of detainees in Grevillea, the use of capsicum spray by guards, and assaults.

"We're asking for a court ruling that an adult maximum security jail is unlawful as a youth justice facility," Mr Hugh de Kretser said outside the Victorian Supreme Court on Monday.

A parliamentary committee and the Commission for Children and Young People are investigating the system, while a Department of Health and Human Services review is also under way.

In early February, Ms Glass released a report on facilities in the Grevillea unit, Malmsbury and Parkville, raising concerns over the conditions detainees were subjected to.

The government has also announced a new jail will be built west of Melbourne and responsibility for youth justice will move from DHHS to the Department of Justice in April.