With his casual dress sense, ready laugh and broad vowels, Bruce Western immediately strikes you as the expatriate Queenslander he is. The Harvard-based sociologist is also one of the leading scholars of crime in the United States, and a few years ago he presented a seminar about his research at the United States Studies Centre in Sydney.

Now, I’ve attended hundreds of conferences and academic seminars, and I don’t recall ever gasping out loud. But I did when Western’s PowerPoint presentation began reeling through the following facts.

US jails currently hold over two million people, more than 1 per cent of the adult population. Among men aged twenty to thirty-four who didn’t complete high school, the imprisonment rate is a jaw-dropping 12 per cent for whites and 37 per cent for blacks. That’s right — 37 per cent of young, black high school dropouts are currently behind bars.

But it gets worse, because that’s just the proportion behind bars on any given day. By the time black high school dropouts reach their mid thirties, Western estimates that 69 per cent of them will have been imprisoned. In other words, if you’re a black man who doesn’t finish high school, the chances are two-in-three that you’ll see the inside of a prison cell.

As Western points out, one of the things that a high incarceration rate does is to make other statistics look good. For example, official employment surveys exclude the prison population. If the official statistics counted prisoners, then the employment rate among young black dropouts would fall from 40 per cent to 25 per cent. A similar effect is likely to show up in other measures, such as income inequality and ill health. Because numbers often drive policy, this kind of invisible disadvantage can readily be missed in public debates.

Another feature of a persistently high incarceration rate is its intergenerational impact. In the United States today, 2 per cent of white children have a parent in jail. Among African-American children, the figure is 11 per cent. One-in-ten black children — 1.2 million kids — have a parent behind bars.

The extraordinary thing about this pattern is that while more people are being sent to jail, fewer people are committing crimes. From 1960 to 1990, the US homicide rate doubled. From 1990 to 2010, it halved. Most other violent and property crimes have followed a similar trajectory. So with the US murder rate lower than at any other time in the past generation, the incarceration rate is at an all-time high. In fact, while the US murder rate has halved over the past twenty years, the incarceration rate has doubled.

Over recent years, a number of good books have analysed the interplay of factors that have brought the United States to this point. Economists like Paul Krugman and Jeffrey Sachs have pointed to the stagnation of middle-class wages as proof that the American dream is in danger of slipping away. Sociologists and criminologists like Bruce Western and Mark Kleiman have argued that US criminal justice policies need to be fundamentally rethought. And political scientists like Larry Bartels and Jacob Hacker have laid the blame at the feet of ideological polarisation, as the Republicans moved sharply to the right.

Still, none of those authors has actually gone quite as far as James Gilligan’s Why Some Politicians Are More Dangerous Than Others. Gilligan has written a book about how Republicans kill people. Specifically, Gilligan shows that when a Republican president occupies the White House, the homicide and suicide rates tend to rise. Conversely, when the president is a Democrat, the homicide and suicide rates tend to fall. He also shows that the same pattern holds across the United States: states that have traditionally voted Republican tend to have higher homicide and suicide rates than those that have traditionally voted Democratic.

At this point, you might expect a Labor politician like me to use these findings to beat up on my ideological opponents. But the more time I spent with Gilligan’s book, the less convinced I became that the associations he describes are truly causal. First, the empirical evidence turns out to be fairly fragile. There aren’t that many presidents, particularly if you omit those who served at times of national crisis, such as world wars and depressions. And taking a snapshot across states means that you’re capturing plenty of things that have nothing to do with policy, like a state’s racial composition. (The same problem plagues Richard Wilkinson and Kate Pickett’s book, The Spirit Level.) If you ask more nuanced questions like “Does homicide rise significantly when a Republican governor replaces a Democratic governor?” the answer is no.

Then there’s the causal channel. On the face of it, you’d think that both Republicans and Democrats have pretty strong incentives to reduce the rate of violent death. To the best of my knowledge, no one has ever run for election on the slogan “Let the killing spree begin.” So it’s not obvious why we should expect to see a relationship between election outcomes and violent crime. Gilligan argues that Republicans cause bad economic outcomes (higher unemployment and more inequality) and these outcomes cause homicide and suicide. But careful economic studies (such as those by Bruce Weinberg and Naci Mocan) suggest that the effect of joblessness and inequality on crime is weak or non-existent, so this can’t be the answer. At the end of Gilligan’s book, I was left feeling that there still might be a relationship between politics and violent death rates; but that if it exists, it’s a good deal more complex than he proposes.

So if simplistic partisan models don’t explain crime rates in the United States, what does? In Freakonomics, Steven Levitt and Stephen Dubner famously showed that one factor that explains the drop in crime rates in the 1990s was the legalisation of abortion in the early 1970s. The decision in Roe v Wade allowed women who didn’t feel they were ready to raise a child to have a legal abortion. Because criminal careers typically peak in the early twenties, the timing fits the data, but the implications shocked many. In his academic work, Levitt has also attributed the crime drop to more police and the waning of the crack epidemic. And he argues that higher incarceration rates cut crime — an issue to which I’ll return.

But the drop in crime wasn’t uniform across the country. In New York City, crime rates fell about twice as much as they did in the rest of the country, and continued falling through the noughties. The crime-ridden New York that features in Crocodile Dundee bears little resemblance to what’s there today. The chances of a New Yorker being killed are a fifth of what they were in 1990. The odds of a New York woman being sexually assaulted are a quarter of what they were two decades ago.

In The City that Became Safe, criminologist Franklin Zimring asks what can be learned from the dramatic fall in crime in the Big Apple. Having previously written a book on the nationwide crime decline, Zimring is keen to see what made New York different. How could it be that the chance of being robbed in the Times Square precinct is now one-twentieth of what it was two decades ago?

The conventional view on New York’s crime decline is that it was due to “broken windows policing.” The theory goes that cracking down on minor offences — street prostitution, riding the subway without a ticket, public gambling — changed the social norms about offending. By signalling that the city had a “zero tolerance” for criminal behaviour, this policing strategy changed the norms around misbehaviour.

Economists like me tend to be dubious of theories based on social norms, and Zimring’s book shows that scepticism to be well founded. Despite the public rhetoric, he finds that arrests for street prostitution and public gambling actually fell during this period. Broken windows may have been a good public story, but it’s a flawed account of what New York’s police officers were doing.

Yet other policing strategies appear to have been deployed successfully. Zimring credits part of the crime decline on “hot spots” policing, which targets locations (such as open-air drug markets) that have a history of criminal activity. Another approach that seems to have worked is the Compstat policing database, which allows top cops to see in real time the crimes that are being reported around the city. More controversially, Zimring concludes that “stop and frisk” policing helped to reduce crime. Although he acknowledges that the burden fell disproportionately on minority communities, he points out that they also enjoyed the benefits of the crime decline. For example, a strategy of stopping and searching cars with darkened windows was disliked by the young minority men who tended to drive such vehicles but was effective in reducing handgun deaths.

Despite this, there is still much that we don’t know about successful policing. As Zimring points out, many policing tactics are based on intuition rather than evidence. A few US criminologists — such as Lawrence Sherman — have collaborated with police departments to run randomised experiments. Very often, their findings challenge conventional wisdom. And yet most police departments still rely more on anecdotes than data, much as medicine did in the nineteenth century.

The most interesting aspect of New York’s crime decline was the extent to which it occurred without a commensurate increase in incarceration. If New York had followed the national trend, there would be five times as many black and Hispanic men in its jails. Somehow, New York has managed to stop the school-to-prison pipeline that plagues so many other parts of the United States. Understanding this is perhaps the most important lesson of the New York crime success, since it suggests that the whole country could potentially have both low crime rates and low incarceration rates.

America’s mass imprisonment is one of the most troubling features of an extraordinary country. In a recent article in the New Yorker, Adam Gopnik pointed out that more than 70,000 prisoners are raped annually — and yet the subject remains standard fodder for comedy. On our television screens, lovable cops often use the threat of prison rape to force a suspect to confess. Gopnik surmises that future generations will look back on our society’s sadistic attitude to prison rape much as we look back at “eighteenth-century japery about watching men struggle as they die on the gallows.”

Mass imprisonment is also a symptom of a broader illness in the US polity. If you’re a high school dropout there, your inflation-adjusted wages have barely budged since the 1970s. Adjusted for inflation, the typical American family was $4000 poorer in 2010 than in 2000. Over the past two decades, more than half the income gains in the United States have gone to the top 1 per cent.

As with so many things, Australia’s social trends look like a smoother version of America’s. Our crime rate never rose as high, but our rise and fall followed a similar pattern. Your odds of being a homicide victim are half what they were in the late 1980s. And yet our incarceration rate has risen to record highs.

Last year, my colleague Shayne Neumann and I moved a motion in the House of Representatives aimed at reducing crime and incarceration. The motion pointed out that over recent decades, Australia has invested in prison-building at an astonishing rate. In 1991, the national imprisonment rate was 117 prisoners per 100,000 adults. By 2011, it had risen to 167 prisoners per 100,000 adults. Each prisoner costs the taxpayer nearly $300 per day — about the price of a nice hotel room in the CBD.

The story is even worse for Indigenous Australians. When the royal commission into black deaths in custody reported in 1991, there was widespread shock about the level of Indigenous incarceration in Australia, at 1739 prisoners per 100,000 Indigenous adults. Yet over the next two decades the Indigenous incarceration rate increased even further. In 2011, 2248 out of every 100,000 Indigenous adults were behind bars; in Western Australia, 4 per cent of all Indigenous adults are currently in jail. Even adjusting for the fact that Indigenous people tend to be younger, they are still fourteen times more likely to be in jail than non-Indigenous people. By their mid twenties, 40 per cent of Indigenous men have been charged by police with a crime.

As in the United States, Australia’s prison population has increased while crime has fallen. The drivers are changes in the law, such as tougher bail conditions and mandatory non-parole periods. Longer sentences have an incapacitation effect, but probably not much impact on deterrence. Increasing sentence lengths from five to ten years may sound tough, but if you’re dealing with someone who lives from day to day it could have no impact on crime rates.

With few exceptions, the Liberal and National Party speakers on the motion seemed unconcerned by the rise in imprisonment. If this is the pattern in the federal parliament, it bodes ill for state jurisdictions, where it’s harder still to escape the shock jocks. Progressive legislators like Western Australian MLA Paul Papalia, NSW attorney-general Greg Smith and ACT attorney-general Simon Corbell are thinking about approaches such as justice reinvestment, but we need more politicians who are willing to adopt evidence-based criminal justice policies.

One solution may be for Australia to invest in a blue ribbon commission on “what works in criminal justice.” Accompanied by a commitment to conduct more randomised policing experiments, such an approach would strengthen the hand of politicians accused of being soft on crime.

The lesson of New York is that we don’t need to rebuild the gulags to keep the streets safe. Mass incarceration may cut crime temporarily, but at a huge cost to the budget and to the prospects of those locked up. And that’s before we consider the impact on children. There’s nothing family-friendly about the fact that around 40,000 Australian children currently have a parent behind bars. If we care about breaking the intergenerational poverty cycle, we need to reduce incarceration rates.

Unlike Gilligan, I believe that the solution must be a bipartisan one. In the heat of an election campaign, slogans often trump facts. This is particularly true of issues that strike an emotional chord. It’s not for nothing that the most famous negative advertisement in political history is the Willie Horton attack ad deployed by George H.W. Bush against Democratic presidential candidate Michael Dukakis. Featuring Horton’s bearded mugshot, the ad described in chilling terms how Horton used weekend prison leave (a policy permitted in Massachusetts while Dukakis was governor) to kidnap a young couple, stabbing the boy and raping the girl. Faced with such a horrific set of facts, our minds shift away from facts and reason towards thoughts of anger and revenge.

As books by psychologists like Daniel Kahneman and Drew Westen have pointed out, the political landscape is replete with the hulks of logical arguments that were defeated by emotional appeals. And yet it’s not beyond the realm of possibility to imagine that we can achieve a better set of justice policies than we have today. As the United States shows us, a strategy of “lock them up and throw away the key” has a massive social cost. There are smarter ways to keep our streets safe. •