Something terrible lies at the heart of New Orleans - a rampant, widespread and apparently uncontrollable brutality on the part of its police force and its prison service. The horrors of its criminal justice system from decades before Hurricane Katrina and up to now lie somewhere between, with little exaggeration, Candide and Stalin's Gulags.

Spit on the sidewalk here, and you may be arrested - New Orleans has the highest incarceration rate of any city in the United States - and if you're poor and black and can't pay bail, you will enter a place where any protection under the American constitution and the Bill of Rights is stripped away. You will wait weeks or months to be charged, whether innocent or not, and in the meantime you will be subjected to foul, overcrowded jail conditions, prisoner-to-prisoner violence and the brutality of the deputies who guard you. God help you if you have a medical condition, or a mental-health problem, or if you're pregnant (you may deliver in leg chains - it has happened). "A minor offence in New Orleans," one civil rights attorney told me, "can get you into a hellish place."

On 17 March this year, the federal department of justice (DoJ) decided that enough was enough and it has made moves to have the New Orleans police department (NOPD) placed under the supervision of a federal judge. The New Orleans jail system will likely follow.

The department released a report covering only the past two years and ignoring several current federal investigations of police officers for murder. It says, more or less, that the NOPD is incapable on any level; that it is racist; that it systemically violates civil rights, routinely using "unnecessary and unreasonable force"; that it is "largely indifferent to widespread violations of law and policy by its police officers" and appears to have gone to great lengths to cover up its shootings of civilians. "NOPD's mishandling of officer-involved shooting investigations," the report says, "was so blatant and egregious that it appeared intentional in some respects."

The department can't even handle its sniffer dogs: "We found that NOPD's canines were uncontrollable to the point where they repeatedly attacked their own handlers."

There has been a New Orleans season on British television. There is Treme (pronounced "tre-may"), named after a district in the city. This is the journalist-turned-TV-writer David Simon's successor to The Wire, and is currently showing on Sky Atlantic. Its subplots deal with these themes - the disappearance of people without trace into the criminal justice system, the bullying police. But, because of their nature, neither Treme nor Spike Lee's four-hour documentary When the Levees Broke, which aired here in February, can match print for the chilling forensic details of the New Orleans horror story. For prisons, you must read the deeply shocking reportage and oral history published in 2006 by the American Civil Liberties Union (ACLU) called Abandoned and Abused. And it is a print journalist, A C Thompson, who, remarkably, has pieced together what many policemen and white vigilantes thought they'd got away with, during and after Katrina.

The gallant South

This month, two policemen were up in court, one accused of the killing and both of its cover-up in July 2005, a month before the flood, of Raymond Robair, a 48-year-old handyman from Treme. He was, it is alleged, viciously beaten and dropped off by both of them in a wheelchair in front of Charity Hospital. He died there of a ruptured spleen from the beating he took.

Both men pleaded not guilty. It was Thompson who doggedly pursued this and other stories over 18 months, with the help of many local activist groups, and reported it in the Nation magazine. Then there's Mary Howell, a civil rights lawyer on whom the Treme character Toni Bernette, played by Melissa Leo, is based. Howell has been litigating against the NOPD and the city's prisons for 30 years. In Treme, policemen leave the restaurant when Bernette walks in.

Howell told me: "I tried at the time to get the justice department to investigate the Robair case and to no avail. Essentially, after 11 September 2001, certainly here in New Orleans, virtually all federal government civil rights enforcement stopped and everybody was diverted into anti-terrorism. Only in the summer of 2008 - when Thompson's pieces started appearing - did they start investigating.

“Without him and without Obama being elected, none of this would be happening. The last time I looked there were 11 different investigations that the feds were conducting here and at least 20 different police officers who were either indicted or found guilty of a variety of federal offences coming out of Katrina and the immediate pre-Katrina period."

Henry Glover, a 31-year-old African American, was shot by a police sniper as he picked up goods behind a shopping mall during Katrina. He was taken by his brother, a friend and a passer-by to a nearby school that police were using as a special operations centre. There a Swat team let Glover bleed to death and beat his rescuers. Another policeman took the body in the rescuer's car to the levee and torched it, putting two shots into the body (he later called that "a very bad decision"). The incinerated car with Glover's remains inside it lay a block from the police station for weeks.

Last December, three policemen were convicted for the crime: one of manslaughter, one of burning the body and one of falsifying evidence. Eleven other officers who admitted they had lied in testimony or withheld knowledge were reassigned to desk duty or suspended.

That the police force in New Orleans is "a significant threat to the safety of the public", as the DoJ says, is obvious. But the same problems can be seen all over the South, from Miami to Mississippi to Alabama; and the same nationwide, according to Paul Craig Roberts, a former editor of the Wall Street Journal and former assistant secretary to the treasury under Ronald Reagan, who wrote recently: "Police in the US now rival criminals, and exceed terrorists as the greatest threat to the American public."

In New Orleans the culture of systemic brutality is old and deep. In 1970 a producer friend went to sign the great pianist James Booker, then in Orleans Parish Prison. He came into the warden's office shackled, walking on his knees. In the mid-1990s what Howell calls "a series of horrific events" culminated in roughly 20 police officers being prosecuted for major felonies, state and federal: rape, arson, kidnapping, bank robbery. "We had a cop who was doing bank robberies in his lunch hour," she says. "We have two now on death row, one of whom is there - a first for the US - for having a citizen murdered for filing a complaint against him for misconduct."

Howell adds: "Going into Katrina, our police department was a train wreck - in terms of the police, in terms of the jail, in terms of what was going on in the courts. It was just a deeply dysfunctional system. Katrina didn't cause the dysfunction in the system, it just exposed it."

A young civil rights lawyer, Chloe Cockburn, who spent time working for criminal justice reform in New Orleans, recently wrote a term paper on the subject of the return of corporal punishment to American prisons.

The movement towards rational punishment - from a time when segregation from society was considered punishment enough - has been abandoned in favour of retribution, Cockburn argues. "There's evidence across the culture of people accepting the brutal treatment of prisoners, an idea that because you committed a crime you deserve everything you get," she says. "I think it's impossible for Europeans to truly comprehend how horrible it is here."

You could take the "squirrel cages". These are used in the prison in St Tammany Parish, one of the richest districts in the New Orleans conurbation, and an area to which many white people fled from the city in the 1980s. The metal cages measure 3ft by 3ft and are 7ft high, meaning the prisoner can stand but can't lie down.

New Orleans, according to the ACLU, is a city "without mental health care". The cages are therefore used for prisoners who report being suicidal, have some mental disturbance or are simply being punished for a misdemeanour.

Until August last year at least, there were six of them in the booking area of the jail. Katie Schwartzmann, an attorney with the ACLU, established that prisoners were kept in them for a minimum of 72 hours and often for "days, weeks and even over a month". She added: "I spoke to one prisoner a few days ago who went completely crazy when they put him in there. He started banging his head against the wall as hard as he could and had to have eight staples."

Boatloads of goons

The ACLU sent a letter to the parish, noting that "the cages have frequently been used to hold more than one prisoner at a time and that staff often ignore prisoners' requests to use the bathroom, forcing people to urinate in discarded milk cartons". It also pointed out that the St Tammany Parish code states that dogs must be kept in cages at least 6ft by 6ft, with "sufficient space to lie down". Sick prisoners in the parish were being afforded a quarter of the space afforded to animals. Following the ACLU report, the parish said it would use the cages only in an "emergency".

Then there was "Camp Greyhound", a detention facility known for organised brutality - a little-known, near-exact facsimile of Guantanamo Bay, set up in the bus station in downtown New Orleans. There are few photographs of it - it came and went in a few weeks - but there is a detailed description of it in Dave Eggers's non-fiction bestseller Zeitoun. The book gets its name from an American Muslim, a Syrian-born building contractor who had lived in New Orleans for 11 years. Abdulrahman Zeitoun had sent his wife and their children to Baton Rouge and stayed back to check on his properties.

A boatload of goons from the various militias - government and not - that had started patrolling the city in boats after the hurricane arrived at Zeitoun's flooded place. They arrested him and three companions, one a fellow Muslim Syrian by birth called Nasser Dayoob. The charge sheet he saw many weeks later read: "Looting." Roughed up - face in the mud, knee in the back - handcuffed and shouted at, they were taken to the Union Passenger Terminal bus station in the centre of New Orleans. A wooden sign outside said: "We're taking our city back." One of Zeitoun's companions asked a passing soldier: "Why are we here?" "You guys are al-Qaeda," was the reply.

In the car park they saw a vast construction of chain-link fences, 16ft high, topped with razor wire stretching 100 yards. It was divided into smaller cages, all brand new. Sixteen of them. "It looked precisely like the pictures he'd seen of Guantanamo Bay," Eggers wrote of Zeitoun, noting that many of the prisoners were wearing orange jumpsuits. "Like Guantanamo it was outdoors, all the cages were visible and there was nowhere to sit or sleep."

Each cage had a portable toilet in the open. Electricity was provided by a stationary Amtrak train engine, roaring 24 hours a day. Bright floodlights lit it at night.

The detention unit was purpose-built for the maximum discomfort of its inmates. As Eggers writes: "In the cage, the men had few options: they could stand in the centre, they could sit on the cement, or they could lean against a steel rack." It was run along Gitmo rules. No one brought here had been charged with an offence; none had or would see a lawyer.

This is where Zeitoun and his companions spent three agonising days, being subjected to humiliating strip searches, the guards pushing ham sandwiches through the wire even though they had been seen praying. They watched as one mentally handicapped inmate was tied up and pepper-sprayed in the face until "he was cowering in a foetal position wailing like an animal, trying to reach his eyes with his hands".

Anyone who complained or touched the wire was dragged out, tied up and pepper-sprayed, or shot with beanbag guns. Eventually the guards just shot at men and women through the wire indiscriminately. The worst torture for Zeitoun and the other prisoners was not being allowed to make a phone call - to reassure their distressed relatives, to check on their families in this disaster. It drove Zeitoun's wife nearly mad with worry: even after he was moved to a "normal" prison, she heard nothing of him for almost two weeks.

The orders were undoubtedly punitive - the prison's rules served no other purpose, and even taking a message from a prisoner was an offence. It was also a breach of the prisoners' rights. A jury ordered the city to pay out $650,000 to two white tourists who had their cellphones confiscated and who, as a result, got lost in the gulag for several weeks. They could afford bail and would have obtained it - they claimed - if they had been permitted to use their phones.

Zeitoun and the three others were moved to the Elayn Hunt Correctional Centre. Then, mysteriously, his wife got a call from homeland security saying he was free to go. It still took an astute lawyer several more days to get him out on $75,000 bail. Nasser, his Syrian-born companion, spent five months in jail; of the other two, one was locked up for six months and the other eight. All charges were dropped.

How and why had Camp Greyhound been built with such speed and efficiency, with its food and portable toilets, when the rest of the stranded population had been abandoned for days by the government and was fighting for food and water? It was constructed by the inmates of Angola, the 18,000-acre Louisiana state penitentiary, a former slave plantation and the toughest of all American jails, where the average sentence is 89.9 years. Burl Cain, the warden of Angola, had brought his labour force of convicted murderers and rapists to the New Orleans bus station, where they slept overnight, and used his own equipment and supplies to construct it. He had it done in two days. "A real start to rebuilding New Orleans," Eggers quotes him as saying. Angola has some of the lowest-paid prison guards in the United States, and few of them have graduated from high school. Cain kept them at Camp Greyhound as part of the package.

Who had picked up Zeitoun and his friends? It was hard to tell. Every gun club in America had responded to the NOPD's call for help. It was the chief of police who had said that babies were being raped in the Superdome sheltering thousands of the homeless after the hurricane; his assistant who had, within earshot of many police officers, said they should "shoot looters". The mayor of the city called, farcically, for martial law to be declared where no such ordinance existed in Louisiana. It was a call to arms, and anarchy. It established a free-fire zone - one white vigilante, since indicted for murder, incautiously described it as being "like the pheasant season in South Dakota".

March of the militiamen

One of Katie Schwartzmann's clients was arrested by a gang called the Iowa Guard. There were also at least five mercenary outfits, all licensed by homeland security, including a firm named, unbelievably, Instinctive Shooting International. It described itself as being staffed by "veterans of the Israeli special task forces". The investigative reporter Jeremy Scahill, author of Blackwater: the Rise of the World's Most Powerful Mercenary Army, told me how he met two of these vets outside the affluent gated community of Audubon Place, and saw some of them, with ISI logos on their backs, posted as snipers on adjoining rooftops.

He wrote in his notebook at the time: "Both say they served as professional soldiers in the Israeli military and one boasts of having par­ticipated in the invasion of Lebanon. 'We have been fighting the Palestinians all day, every day, our whole lives,' one of them tells me. 'Here in New Orleans, we are not guarding from terrorists.' Then, tapping on his machine-gun, he says, 'Most Americans, when they see these things, that's enough to scare them.' They were helicoptered in by powerful businessman James Reiss, who serves in Mayor Ray Nagin's administration as chairman of the city's regional transit authority." As Scahill told me: "Reiss was talking openly of the need to change the 'demographics' of NoLa [New Orleans, Louisiana] after the hurricane."

“There are policy decisions that are made because of the fact that we are a largely African-American city," says Howell. "And that's something that was so shocking, that not only did the local authorities not care, [but] every level of government failed. Every level."

Sick system

Camp Greyhound, when it was exposed, was the focus of much retrospective anger in the black population: it made it clear where the priorities lay - a holding jail was more important than food, water and medical help. Meanwhile, the 7,000 inmates of the main buildings of Orleans Parish Prison had been left, more or less, to drown. They very nearly did: the buildings are in the lowest part of New Orleans.

Marlin Gusman, who is still the city's sheriff, refused to evacuate the jail when the floodwaters came. "This is a very, very lucky sheriff, is all I can say - that there were not significant deaths as a result of that," said one activist I spoke to. As the jails started filling up with water, many deputies left their posts, abandoning the prisoners in their cells, in the dark, with no way of knowing if they would get out.

Almost every prisoner reported going without food and water for days after the storm. They lived in terrible heat - with broken air-conditioners and no windows, in stinking floodwater. One said: "I witnessed several inmates with various medical conditions suffer from dehydration - we were forced to live off toilet water and lie in our own waste and body fluids. We were drinking out of toilets because that is all we had . . . When the rescuers arrived, I was still locked in my cell and they had to pry the bars open. I walked out in chest-deep sewer water."

A deputy who came back to try to release the prisoners recalled: "Before the water got to my waist, we put them all on lockdown, and the scary thing about that was the cells wouldn't open back up. We had to go under the water and try to open them manually." The rescuers only just succeeded.

Hundreds of prisoners were moved from other buildings to the prison's central lock-up area, where they remained standing in deep water for as long as 12 or 13 hours - mostly because the sheriff didn't have enough boats to transport them to higher ground. For those who went to the Jena Correctional Facility, a former juvenile prison, it was "the beginning of a new nightmare", according to the American Civil Liberties Union. "They were subjected to egregious physical and verbal abuse almost immediately after they arrived . . . At one point in their stay several prisoners were told to line up, place their hands behind their heads and press their groins against the buttocks of the prisoners in front of them. An officer taunted them saying, 'Hard dicks to soft ass! I know y'all are getting hard, because I am.'"

This makes it appear that the Abu Ghraib prisoner scandal was not an aberration - it was a sample export of everyday abuse across the criminal and penal system in the US. But Sheriff Gusman dismissed the entire ACLU report. "Don't rely on crackheads, cowards and criminals to say what the story is," he said.

People in the jails disappeared in their hundreds. Almost every person arrested in New Orleans is sent to jail on a "money bond", and public defenders or state lawyers are required to secure your release or reduce your bond. The system broke down because the money to pay the public defenders - which came mostly from traffic fines - dried up. "There were almost no records of these people, almost no files," says Howell. "So a volunteer group of public defenders and criminal defence lawyers, a sort of Dunkirk rescue mission, got together and started creating their own database, locating people. But even if you found them, you wouldn't be able to get them out necessarily. You had to go and prove their identity, then file court proceedings to get them out."

The big struggle

In 2007, two years after Katrina, when the murder rate rose again - to five times that of comparable-sized cities - there was an explosion of anger at the failure of the New Orleans criminal justice system. "Enough! Officials reviled in public show of mass outrage", declared the Times-Picayune newspaper. The people began to speak with a collective and powerful voice. Hundreds had been locked up for trivial offences and murder kept on rising. Zero tolerance wasn't working; besides, it was very expensive.

But as the New York Times wrote: "There are serious risks in taking on sheriffs in Louisiana, given their political heft." The reason why there are so many arrests and so many people in jail is that the city gives the sheriff a daily payment of $22.39 to house them each night (it's more for taking in state prisoners). And in New Orleans you can be held for 45 days, or 60 days for a felony, before you are even charged.

The jail is every sheriff's power base; it gives him one of the most influential positions in government. It gives him jobs to dispense - non-civil service, non-union jobs - and a large, pliable workforce that can be called on for any task, such as getting the vote out.

Sheriff Gusman (who is black) strongly criticised the size of Orleans Parish Prison when a councilman - before Katrina, it had the largest number of inmates per capita of any city in the US. Later he changed his tune, campaigning to build a new jail with a similar capacity, of 5,500 beds. Building such a jail would cost a quarter of a billion dollars, and involve big contracts. But then Gusman hit a snag.

Some new members had been elected to the city council in 2006, among them James Carter, a prominent young African-American attorney who wanted reform. The council had sought help from the Vera Institute of Justice, a non-profit organisation that advises governments. The leading figure behind the recommendations Vera made was Jon Wool, its local director. He had galvanised progressive-minded government officials as well as community and activist organisations that wanted change.

“It would have been a terrible disservice to rebuild what all would agree is a chronically poor system," he says. "It had to be reinvented. And the whole story turned on the size of this new jail. There was resistance. The prospect of a jail more in line with good practice was seen as a threat to the status quo in most corners of the criminal justice system. But these leaders and community groups stood together in a very effective way and captured the debate."

The counterproposal was to build a jail with 1,438 beds. To make this work, Wool proposed reforms that would reduce the jail population. He set up a pre-trial release system for non-violent crimes which has sped up processing for minor offences from 60 days to five days; summonses have replaced custodial arrests in more than half of minor cases. By December last year there was a drop of 500 inmates from the previous June. The public defender's office - crucial for poorer defendants - was reinvigorated with grants of $4m; lawyers were required to give up private practice and do public work full-time.

Further reforms are afoot which, Wool is sure, will bring the jail numbers in line with his new prison figure. "He was like water crashing over a stone, over and over. He's persistent," said a city functionary who has worked with Wool over the years. The result was that, on 3 February, the city council voted unanimously to pass an ordinance mandating the sheriff to build a new facility limited to 1,438 beds. It was an important turning point - and a victory for community action.

But it's not over. The sheriff is not happy, and Wool sees the window for change staying open for only a short time. Funding could be cut off. Political whimsy could put an end to all reforms. "The real question and the hard part," says Mary Howell, "is making real changes that have a prayer of lasting. We can't wait another 30 years for solutions."

James Fox is the author of "White Mischief" and collaborated with Keith Richards on the writing of his memoir, "Life" (Weidenfeld & Nicolson, £20). You can follow him on Twitter here at @jamesfox01.