The motley gaggle of miscreants shuffles into the court, lining up silently in three rows on the benches. There are some 20 of them, men and women of all ages, most with long records of theft, violence and weapons misuse, and all with hardcore drug problems. I am passed their court biographies; the top one describes a man who has spent 12 years in jail, has 26 convictions over two decades and lists his "drugs of choice" as cocaine and heroin.

While Felin Bell's fellow convicts look like they have walked straight off the set of a Hollywood crime caper, he is a portly man in a smart check shirt who could pass as a middle manager. Almost before he has sat down, he is picked out by the judge as the week's shining star, commended for his positive attitude and sent home as a reward. "You look surprised – you shouldn't be," he is told.

Already it is clear this Dallas court for drug offenders is no normal court. As proceedings unfold, it seems like therapy crossed with a reality television show. The judge doesn't wear a robe, seldom sits on the bench and swears a lot. His name is Robert Francis, a fast-talking 52-year-old Republican, a rock fan and a keen hunter who proudly showed me, in his office before proceedings started, the heads of huge hogs he has shot.

As the offenders troop in, he warns if anyone lies or bullshits he will go "fucking ballistic". Then he discusses the difficulties of staying straight as he dissects their jobs, their families, their desires for the future. He responds to their comments with bawdy jokes, short homilies or sharp threats. "Stay positive, brother," one man is told, while another is warned: "You might think I'm crazy but I'm the crazy bastard who can put you back in jail."

There are outbursts of applause, then cheers for a young man who looks embarrassed as he reveals he got married two days earlier. The judge tells a woman who has started helping her mother around the home that she makes him proud. "You gotta be proud, too," he says.

Until recently, these people would have been discarded in overcrowded prisons. After all they were caught in Texas – the toughest state of a nation that locks up more offenders than any other in the world, with more than one in every 100 adults behind bars. Instead they receive counselling and assistance with housing and employment, although they can be sent back to jail if they fail drug tests, abscond or reoffend. One woman, a crystal meth addict, tells me the sessions in court are like walking on eggshells. But there are small incentives for those doing well, such as $10 gift vouchers or – on the day I visited – barbecue lunch out with Francis. "These people have to believe we care and want them to succeed," he tells me later. "Once they believe in me they can start to change."

They are beneficiaries of a revolution in justice sweeping the United States, one with illuminating lessons for Britain. It is a revolt led by hardline conservatives who have declared prison a sign of state failure. They say it is an inefficient use of taxpayers' money when the same people, often damaged by drink, drugs, mental health problems or chaotic backgrounds, return there again and again.

Remarkably, this revolution was unleashed in "hang 'em high" Texas, which prides itself on its toughness and still holds more executions than other states. But instead of building more prisons and jailing ever more people, Texas is now diverting funds to sophisticated rehabilitation programmes to reduce recidivism. Money has been poured into probation, parole and specialist services for addicts, the mentally ill, women and veterans. And it has worked: figures show even violent crime dropping at more than twice the national average, while cutting costs and reducing prison populations.

In the process, right-wingers have allied with liberals who long advocated such an approach, detoxifying one of the most poisonous political debates at a time when US party divisions have never been sharper. "This used to be one of the most emotive and ideologically divisive issues in the country," says Adam Gelb of the Pew Center on the States, a social-policy research charity which is backing the initiative. "We are starting to see the triumph of sound science over soundbites.

"There is not agreement on the causes of crime or even the purpose of punishment," Gelb continues, "but there is agreement on the solutions. Liberals and conservatives are getting to the same destination from very different routes." Now the Texan tactics are being adopted in other "deep red" republican states such as Alabama, Georgia, Kentucky, Oklahoma and South Carolina, while well-known conservatives flock to promote the cause, including Newt Gingrich, Jeb Bush, Bobby Jindal and Grover Norquist. It is a Nixon in China moment. "The fact that it began in Texas has resonated around the country," says Gelb. "We hear again and again that if Texas can do it, then it cannot be soft on crime."

Just as in BRITAIN, it has been an iron rule in US politics that candidates win elections by talking tough on crime. The result has been a wave of stiff sentencing laws which, combined with the backfiring "war on drugs", mean that the prison population is currently growing 13 times more quickly than the general population. As a result, a nation with 5% of the global population accounts for 25% of prisoners worldwide – and is spending £43bn a year keeping them there. The criminal justice system also stands accused of worsening racial inequality, with Hispanic men three times as likely to be locked up as white men and black men nearly seven times more likely. According to a landmark Pew report, one in nine black men between the ages of 20 and 34 is behind bars.

Texas typified the trend. Just eight years ago, it had the highest incarceration rate in the world, with one in 20 adults in prison, on parole or on probation. The biggest deficit in state history had led to cuts in probation, giving judges little alternative but to increase prison admissions.

In 2006, however, a political earthquake started to shake conventional wisdoms. It began when the state budget highlighted the need for $2bn for seven new prisons to accommodate a predicted 17,700 extra inmates by 2012. When Republican stalwart Jerry Madden was appointed chairman of the house corrections committee, the part of the state government responsible for criminal justice, jail and parole, he was asked to avoid building more prisons since they were too expensive.

The white-haired Madden is an unlikely hero of prison reform. A fan of George W Bush and sympathetic to the Tea Party, he happily describes himself as "a typical Texan Republican – which makes me very conservative when viewed nationally". He admits that when he took over the committee, he knew nothing about the subject and had no interest in what has since become his life's mission. But he was trained as an engineer and, with an open mind, set about working out solutions. Perhaps it helps that he is at the end of his political career – when we meet, he is packing up his office in Austin, having announced his retirement. "I'm not interested in the feel-good stuff, I'm interested in what works," he says. "And since my job was not to build more prisons, I had to investigate the alternatives."

Madden turned to Tony Fabelo, a former adviser to both Democrat and Republic governors, for evidence to defeat the "lock 'em up and throw away the key" lobby. Fabelo's figures were frightening: in just two decades incarceration rates in Texas had risen from 226 people to 691 people per 100,000 residents, yet other states with slower growth rates saw faster falls in crime. Meanwhile, one in three Texan inmates was back behind bars within three years. "They had created a totally broken system," says Madden.

Fabelo told Madden to focus on short-term facilities to tackle the underlying problems of repeat offenders, such as substance abuse and mental health problems. He suggested spending $240m on these services – under half the amount earmarked for new prisons that year – and pointed out that every dollar spent in this way saved at least $2 in the long run. Each prisoner, after all, costs $50 a day.

Madden looked at the numbers and took a leap of faith. He went on the attack, using traditional right-wing arguments to subvert those seeking hardline penal policies. "We moved the issue from one of being soft on criminals to one of being smart over the use of money. If you are keeping people in prison who do not need to be there, then that is a waste of taxpayers' money.

"We call it the department of corrections, so we should try our best to correct people, not just incarcerate them," Madden continues. "Some people deserve to be in prison until they die, but you don't fill up prisons with people we are mad at who have done dumb things in their lives. You try to change their behaviour."

Few typify this better than Jose Barajas. As this bulky 31-year-old man told me his life story a shy smile flashed occasionally across his face; when it did so he looked almost like the sweet kid whose father was taken into the desert and shot dead by rival gangsters. Jose was just 11 then.

In subsequent years Jose drank hard, took lots of drugs and was determined no one would get the chance to take advantage of him like the men who murdered his father or abused his mother. "Me and my pistol was enough," he tells me. "I felt like a cowboy in the Wild West. I did a lot of robberies, a lot of carjackings, all sort of other things."

Barajas was furious at a world that seemed unforgiving. One night, off his head at a party, he decided to die in a blaze of glory. "I wanted to go out with a suicide by cop, get them to shoot me down," he said. "So when the police came I started shouting at them, taunting them, but they refused to shoot me. I was so angry I did not die. Now I realise how selfish that would have been to my family and kids."

Given his record, Barajas was looking at a long stretch in prison when he was arrested. Instead he is working as a roofer, trying to be a good dad and nursing ambitions to open his own tyre shop. "I always thought I was on my own and never understood all my anger," he says. "I'm not saying I don't have bad days and bad experiences, because of course I do, but I am confident I am going to make it."

There are only two ways to cut prison populations: slow down the flow of people entering the system or speed up the numbers being released. Texas did both. Figures reveal that crime rates fell 8.3% in 2011, far outpacing national falls, with murder and robbery rates down about 15%. Property crime fell 10 times faster than in the rest of the country. Meanwhile, despite the soaring state population, prisoner numbers dropped by 2,500 last year.

Probation services used to rely on little more than gut feelings to determine if inmates might get into trouble again. Now they use sophisticated risk-analysis tools that have cut the number of low-risk offenders who reoffend within a year from 26% to under 1%. "We kept asking politicians to listen to us," says Geraldine Nagy, head of probation in Travis County. "But I thought the idea that prison was the only option was so ingrained in the minds of so many people we would never get this shift."

At the root of the reforms is an idea alien to many on the right: to understand more and condemn less. "The people we are dealing with are not like you and me," Judge Francis told me. "I found this a shock. I grew up in a house with married parents, both of whom had college degrees. I thought this was normal, but now I know it isn't."

The vast majority of people parading through his court come from broken homes, failed to graduate from school, began using drugs in their teens and had children before they were 20. "These people are preconceived to have a harder path through life than the likes of us," he said. "I don't even say we offer rehabilitation. We are trying to provide 18 years of parenting in one year of drug court."

Francis's 15-strong team deals every year with about 320 offenders, who typically spend six months in special treatment units in prison, then a year attending court several times a week with regular drug testing, followed by a decade on probation. Recent graduates include Lavoris Neal, a smartly dressed 33-year-old who grins as he tells me his check shirt, chinos and shiny black shoes cost $12 from a thrift store. He never knew his father, had an illiterate mother, was drinking by the age of eight, delivering drugs by 12 and using them the next year. "I thought this was normal," he said. "No one told me I was raised wrong, and when they did I was in tears. But I was a kid and did what I was told."

Soon he was armed and dangerous and using PCP – a hallucinogen that leads to paranoia. He stole, shot at people, went on the run from police for months. When I ask what was the worst thing he did, he smiles. "I've not been caught for some things." Then he pulls out some documents from a folder. "Look at this," he says. "I've never had insurance for a car before."

Now this energetic character works as a business consultant, has twins on the way with a new partner and spends his leisure time devouring political biographies at the local library. "I never want to know that person I was again," says Neal. "This was the best thing that could have happened to me. I feel like I am starting life completely afresh."

The revolution that began in Texas and is sweeping conservative America remains in its infancy; the hope is it will also reach this country, which locks up more prisoners than anywhere else in Europe and where debate is stagnant. Whatever populist politicians say, it is far tougher to force people to change errant behaviour than it is to simply slam them behind bars. Yet all the evidence shows this approach cuts both crime and the costs of incarceration. As Judge Francis says: "What I am doing here is the most conservative thing possible. I am getting the biggest bang possible for the bucks of taxpayers and achieving something positive for society at the same time."

The author travelled to Texas on a grant from Make Justice Work (makejusticework.org.uk)