The justice secretary, writing in the Guardian , says there is an urgent need to stop reoffending among a 'feral underclass'

The justice secretary, Kenneth Clarke, has blamed the riots that swept across England last month on a "broken penal system" that has failed to rehabilitate a group of hardcore offenders he describes as the "criminal classes".

Revealing for the first time that almost 75% of those aged over 18 charged with offences committed during the riots had prior convictions, Clarke said the civil unrest had laid bare an urgent need for penal reform to stop reoffending among "a feral underclass, cut off from the mainstream in everything but its materialism".

Writing in the Guardian, Clarke dismisses criticism of the severity of sentences handed down to rioters and said judges had been "getting it about right". However, he adds that punishment alone was "not enough".

"It's not yet been widely recognised, but the hardcore of the rioters were in fact known criminals. Close to three quarters of those aged 18 or over charged with riot offences already had a prior conviction. That is the legacy of a broken penal system – one whose record in preventing reoffending has been straightforwardly dreadful."

He says: "In my view, the riots can be seen in part as an outburst of outrageous behaviour by the criminal classes – individuals and families familiar with the justice system, who haven't been changed by their past punishments."

Clarke uses his intervention to call for the coalition government to adopt a "renewed mission" in response to the riots that addressed an "appalling social deficit".

His comments will reignite the debate on the causes of the disturbances, which the prime minister, David Cameron, has said "were not about poverty".

The first attempt at an empirical study of the causes and consequences of the riots was announced by the Guardian and the London School of Economics on Monday.

The study – Reading the Riots – will involve researchers interviewing hundreds of people involved in the disturbances. The research, supported by the Joseph Rowntree Foundation and the Open Society Foundations, will also include interviews with residents, police and the judiciary, and an advanced analysis of more than 2.5m riot-related Twitter messages.

The project is based on a groundbreaking survey conducted in the aftermath of the Detroit riots in 1967 by the Detroit Free Press newspaper and Michigan's Institute for Social Research.

The professor who led the Detroit study, Phil Meyer, is advising on the research into the disturbances in England. The LSE's involvement will be led by Professor Tim Newburn, head of the university's social policy department.

There is little agreement in Westminster about the causes of the worst civil unrest in England in a generation. The government has resisted calls for a public inquiry. A "victims' panel" announced by the deputy prime minister, Nick Clegg, will take evidence from residents in areas where there was rioting and report preliminary findings in November. The four-person panel will be chaired by Darra Singh, chief executive of JobCentre Plus.

A parliamentary hearing into the riots will on Tuesday hear evidence from the mayor of London, Boris Johnson, and the acting commissioner of the Metropolitan police, Tim Godwin. The home secretary, Theresa May, will speak to the home affairs select committee about the unrest on Thursday.

"There is an urgent need for some rigorous social research which will look, without prejudice, at the causes and the consequences of the recent riots," Newburn said. "Crucially, it is vital that we speak with those involved in the disturbances and those affected by them to try to understand any lessons for public policy."

Executives at Twitter's headquarters in California authorised the collation of 2.5m tweets, pooled from hashtags relating to the riots and their aftermath, so they could form part of the study. A spokesman for the company said: "Twitter provided publicly available information that is accessible to researchers and others."

The project will also interrogate a second database, compiled by the Guardian, containing 1,100 defendants who have appeared in court charged with riot-related offences. The data, covering more than 70% of the defendants processed through English courts for offences linked to the disorder, indicates that sentencing by crown court judges has replicated the punitive response of magistrates in response to the riots.

An initial analysis of the crown court cases suggests the three most severe sentences relating to the riots were handed to individuals who did not directly participate in the disorder, but were convicted of inciting riots via Facebook.

They include Perry Sutcliffe-Keenan and Jordan Blackshaw, who received four years for inciting riots in their home towns of Warrington and Northwich. None of the messages posted by either individual led to a riot and both men are appealing against their sentences, which were condemned by some quarters.

However, Clarke warns against criticism of the judiciary. "The judiciary in this country is independent and we should trust judges and magistrates to base decisions on individual circumstances," he writes. "Injustices can occur in any system: but that's precisely why we enjoy the services of the court of appeal."

The tone of the justice secretary's article – his first public response to the disorder – contrasts with that of the prime minister, who has taken a more hardline approach and denied that the riots were connected to poverty.

David Cameron has described the riots as "criminality pure and simple" and blamed the "the slow-motion moral collapse that has taken place in parts of our country these past few generations".

Clarke writes: "The general recipe for a productive member of society is no secret. It has not changed since I was inner-cities minister 25 years ago. It's about having a job, a strong family, a decent education and beneath it all, an attitude that shares in the values of mainstream society. What is different now is that a growing minority of people in our nation lack all of those things and indeed, have substituted an inflated sense of expectations for a commitment to hard graft."