The proportion of children convicted of a crime who are from black, Asian and minority ethnic backgrounds has nearly doubled in eight years, an increase experts have described as an urgent problem.

The steep rise comes despite a drastic drop in the overall number of youth cautions or sentences handed down, which one academic described as an indictment on the whole justice system.

In 2018, 27% of the 10 to 18-year-olds who received a youth caution or sentence were from a minority ethnic background, compared with 14% in 2010, according to official figures from the Ministry of Justice released earlier this year.

Just 26,681 children were cautioned or sentenced in 2018 compared with 106,969 in 2010, thanks to a shift toward dealing with less serious offenders out of court. The increased use of “diversion”, however, in which minor offenders and those deemed at low risk of reoffending are steered away from the courts, seems to have disproportionately benefited white children.

When the Guardian monitored all cases passing through Greater Manchester youth court in August, of those defendants whose ethnicity we could establish, 36% were from a black, Asian or minority ethnic (BAME) community. All were male, and many were also in the care system. Only 22% of Greater Manchester’s youth population are not white.

Q&A What is the Children in the dock project? Show Hide Children in the dock is an investigation into the youth justice system in England and Wales, which in 2018 saw 26,881 children aged 10-17 cautioned or convicted of crimes ranging from shoplifting and joyriding to rape and robbery. No other EU country puts such young children on trial. Most other nations do not use an adversarial system which can see children separated from their parents and lawyers via bulletproof glass in what is essentially an adult courtroom with a different sign on the door. Officially, the focus in youth courts is on rehabilitation, tackling the underlying causes of youth offending. But although there has been an 88% drop in the number of children cautioned or convicted since 2007, reoffending rates have increased. Just over 40% of children who go to court reoffend within a year; for those sent to youth custody, that figure rises to 71.6%. Children who end up in trouble now are disproportionately likely to be of black, Asian or minority ethnicity. They are vastly more likely to be in care and they are extremely likely to have ADHD, autism spectrum disorders and communication difficulties. This series examines how the court system deals with these vulnerable children and explores whether there is a better way to break the cycle of offending and help them build productive and fulfilling lives.

The Labour MP for Tottenham, David Lammy, published a review of the overrepresentation of BAME individuals in the criminal justice system in 2017. He said the figures had got worse since and that “the fact that half of our youth prison population comes from a black or ethnic minority background would [attract] far more comment and scrutiny” had the political debate not been so focused on Brexit.

Becky Clarke, a senior criminology lecturer at Manchester Metropolitan University, said the “failure to act, given these issues of racial injustice have been known and worsening for many years, is an indictment of the whole justice system”.

She said the increase in levels of disproportionality was a direct consequence of BAME children being dealt “different treatment by practitioners across the criminal justice system”, and called for wholesale changes to policing, transparency in the charging and prosecution of cases, and an end to the unequal application of laws such as joint enterprise.

Clarke welcomed the overall reduction in criminalisation of children but said negative racial profiling meant it “hadn’t benefited all groups equally, meaning BAME children cannot be diverted out of the system in the same ways as their white counterparts”. She put this down to “perceptions of risk and criminality” in the police, Crown Prosecution Service (CPS), courts and youth justice service.

The Guardian spoke to senior representatives of all four groups and few could point to positive changes in outcomes since the Lammy review, which cited youth justice as the “biggest concern”.

Olivia Pinkney, the chief constable for Hampshire and national policing lead for children and young people, conceded there was still “so much more to be done” and “none of us should rest and be comfortable” with the rise in the proportion of young BAME people criminalised. “It’s certainly something that will be unwitting, but we need to understand why.”

She said the police were taking steps to tackle the problem, pointing to the increased focus on the use of stop and search and mandatory equality and inclusion training.

Gerry Wareham, a senior prosecutor and CPS lead for young people, said the CPS could only choose to prosecute from the cases that the police brought to it. “We’re in a demand-led position. We get what the police bring to us,” he said.

Lammy said his review not only showed the disproportionate criminalisation of black youth, but also similar trends for young Muslims and a “historic problem” of overrepresentation of young people from Gypsy and Traveller backgrounds.

The youth justice system is in need of further reform, he said. “There’s a real need to get a grip on how we’re supporting families, how we’re supporting young people, what second chances they are getting.

“Those are big questions that obviously affect all young people who fall into the criminal justice system, but they are generally young people from working-class and poor socio-economic backgrounds and most definitely young people from ethnic

He said the government had accepted the “lion’s share” of his recommendations, but had chosen not to impose targets for improving diversity in the judiciary, where he believed there were “still real issues”. Only 12% of magistrates and 7% of district judges are from BAME backgrounds, according to figures published in July. Youth cases are either heard by three magistrates or a district judge. The Office for National Statistics (ONS) puts the UK’s minority ethnic population at 13.8%.

John Bache, the chair of the Magistrates Association, said some magistrates had found it “very difficult to accept the results” of the Lammy review. He said the reason for that was “absolutely obvious, because every magistrate believes that they are acting with scrupulous fairness … so to be given a report that suggests that that might not be the case is first of all surprising, but also very upsetting because none of us want to be unfair to anybody who appears in front of us.” Unconscious bias training for all magistrates and legal advisers launched in June this year.

The Magistrates Association, which represents the 15,000 justices of the peace in England and Wales, has recently convened a group to address disproportionality, including representatives from the Ministry of Justice, the CPS, the Youth Justice Board and charities. The group agreed it was an “urgent problem, especially in relation to the alarming disproportionality illustrated by the proportion of BAME children and young people in custody”. It proposed recruiting more diverse magistrates and promoting awareness among magistrates of the different factors that BAME individuals face.

The Ministry of Justice said: “The issues that lead to black, Asian and ethnic minority children being overrepresented in youth custody begin before they enter the criminal justice system, which is why work is under way across the system to change race disparity.”