A Guardian analysis of newly released government figures has revealed a shocking increase in the proportion of ethnic minority children and young people being held in the youth justice system.

The figures, issued by the Youth Justice Board (YJB) and which cover young offender institutions, secure detention centres and secure training centres in England and Wales, show that during 2014-15, 40% of prisoners aged under 18 were from black, Asian, mixed race or “other” ethnicity backgrounds (BME).

In 2005-6, when the ethnic mix of British prisoners started to be measured by prison authorities, there were far more children and young people in state custody overall – over 2,800 in 2005, compared to 1,048 in the last financial year. Although the total number of teenagers and children held by the state has more than halved since 2005, the numbers of BME young prisoners has not fallen as much. The share of Asian prisoners as a proportion of the total youth custody population has risen by 75% and the share of black prisoners, who now account for one in five young people locked up, has risen by 67%. The proportion of mixed race young prisoners is also up by 42%, over a 10-year period, accounting for one in 10 of young detainees. In contrast, the proportion of white young people in state custody has fallen from nearly three-quarters of detainees to 60%.

Campaigners say several factors could explain this worrying phenomenon. Kashan Amar, a former regional prison manager, believes four in five young Muslims reoffend after release for a number of reasons that the authorities have failed to address. “There is a significant barrier on resettlement for Muslims, lack of community support and taboo amongst their families,” says Amar. “We see that Muslims are under-represented in sex offender and substance rehabilitation programmes, for example.”

Stop and search, an area of policing plagued by accusations of racist targeting by patrolling police officers, is also a major area of concern. In 2013, the Equality and Human Rights Commission found that young black and Asian citizens were six times as likely to be stopped as white citizens, noting the figure had remained “stubbornly high”.

Earlier this year, the Sunday Times reported that more than 600 children under 10 years old, of varying ethnicity, had been stopped and searched in the last 10 years, normally under suspicion of drug trafficking.

According to Amar, stop and search policies are key to understanding why the representation of BME young people and children in state custody has rocketed. “The level of discretion for individual police officers on the street needs to be decreased, they have far more now than their commanding officers,” he says.

People from BME communities are over-represented at almost all stages of the criminal justice process

Former home office official Mike Hough, now associate director at the Institute for Criminal Policy Research, also believes that the most likely explanation is “differential treatment” of young ethnic minority suspects by both police and prosecutors. “Someone more likely to be stopped and searched would be more at risk of arrest or warning, which would put them further down the road to formal proceedings if they were re-arrested, and so on,” Hough explains.

A recent independent investigation into the over-representation of young Muslim and black prisoners in Britain’s justice system, led by Baroness Young, found that less than 1% of offences by Muslims were terrorist offences, yet the community was still viewed “with suspicion” by authorities. According to the report, black offenders “are stereotyped as drug dealers, and Muslim prisoners as terrorists”.

Campaigners also claim that BME offenders are more likely to receive custodial sentences than their white counterparts, even for the same crime – suggesting a degree of institutional racism in the courts.

Figures provided by the Institute of Race Relations show that people from BME communities are over-represented at almost all stages of the criminal justice process, disproportionately targeted by the police, more likely to be imprisoned and more likely to be imprisoned for longer than white British people.

Black offenders presented before a sentencing judge, for example, are 44% more likely to be given custodial sentences for driving offences than white people, 37% more likely for public order offences or possession of a weapon, and 27% more likely for drug possession. Meanwhile, Asian offenders are 19% more likely to be given custodial sentences for shoplifting, and 41% more likely for drug offences.

Imtiaz Mubarek, who campaigns for better treatment of prisoners after his 19-year-old nephew, Zahid Mubarek, was murdered in 2000 in a race-motivated attack by his cellmate in Feltham young offender institution, says: “The government needs a completely new action plan with community organisations and regulatory bodies to deal with this issue, and they need to work more closely.”

The previous government, in particular, refused to listen to black and Asian community leaders, which just means they’re being unrealistic in their responses to the problem, he adds.

Baroness Young says that successive governments “had not taken the issue seriously,” but she argues that the solutions are about more than just challenging institutional racism.

“We need to be a bit more sophisticated – firstly because if you say it’s racism, that’s a really huge challenge and you don’t know where to begin. There are smaller practical steps that can be taken. Also there are individuals straining really hard within the justice system to create positive change, and we need to support them.”

Juliet Lyon, the director of the Prison Reform Trust, also believes there is a range of possible solutions. “These could include reducing school exclusions, improving police response and addressing discrimination in the justice system,” she says.

According to the Young review, the disproportionate representation of young black and Muslim prisoners in the UK is now greater than that in the US.

Penelope Gibbs, director of Transform Justice, a charity working for an open and effective justice system says: “In the US they refer to disproportionate ethnic minority imprisonment as ‘the new Jim Crow’. We have a different history but the result is as bad. Somewhere the system is going badly wrong.”

The YJB admits that the “over-representation of BME young people in custody is an issue”. A YJB spokeswoman says: “This group does not show as much benefit as others from the overall reductions made … To help establish why this is the case, the YJB has developed a ‘disproportionality toolkit’ for youth offending teams to use. This toolkit, which is currently being trialled, aims to identify the key factors in local areas which influence the admission of BME young people into the youth justice system.”