Young black men now account for nearly 40% of the population of youth jails in England and Wales, according to a report by the chief inspector of prisons.

The report, published jointly with the youth justice board, shows that the proportion of black and other minority ethnic young men in young offender institutions (YOIs) has risen from 23% in 2006 and 33% in 2009/10 to 39% last year.

The changing demographic profile of the population inside youth jails in England and Wales also shows an increasing proportion of young Muslims, up from 13% last year to 16% this year. Foreign national young men account for a record 6% of the population.

The chief inspector of prisons, Nick Hardwick, says young people aged 15 to 18 are being held in deteroriating conditions in the YOI network, with fewer feeling safe while they are locked up.

The inspection showed that fewer young inmates felt they could tell someone they were being victimised or believed a member of staff would take them seriously. Only half said they had done something while they were inside that would make them less likely to reoffend in the future.

The report also reveals that more that one-third of the young men had been physically restrained as part of the disciplinary process at their YOI. The highest restraint rate – 66% – was at the Keppel unit at Wetherby, which deals with male teenagers who have not responded to a "normal" YOI regime. The lowest – 8% – was at the Carlford unit near Woodbridge, Suffolk, which holds 30 teenage boys serving long sentences.

The over-representation of young black men in youth jails comes despite a sharp fall in the number of children and young people in custody that has already led to the closure of five YOIs, including a specialist unit for young women.

The total population of the youth justice "secure estate", which includes eight male YOIs and three specialist units for girls and young women, continued to fall from 1,977 in March 2010 to 1,822 this March, before this summer's riots.

Hardwick says, however, that the number of black and minority ethnic children in custody has not fallen at the same rate as the number of white children being locked up.

"Between 2007 and 2011 there was a 37% reduction in white children in custody, compared with a 16% reduction in black and ethnic minority children," says the report.

The report does not discuss the reasons why young black people make up an ever greater proportion of the shrinking youth jail population. But Hardwick does note that an increasing number – 53% now, compared with 39% last year – of young men are being sent to prison for the first time.

Hardwick said: "This report has highlighted some deterioration in children and young people's experience of custody. Despite the falling numbers, this population has well-defined vulnerability and increasing numbers within minority groups. The need, therefore, to provide these people with support during their time in custody and in preparation for their release is as great as ever."

Frances Done, the chair of the youth justice board, which commissions places in youth prisons, said it would be working with all secure establishments to make sure that young people's time in custody has positive results.

The inspection was based on the experience of 1,115 young men and 47 young women in YOIs and specialist units.