Mass incarceration is a bad thing that we’ve become exceptionally good at in the United States. This dismal statistic may already be familiar: we are 5 percent of the world’s population, yet we lock up 25 percent of the world’s prisoners. Our reliance on imprisonment as a tool of social policy is horrifying more and more people across the political spectrum.

Last September, the Brennan Center for Justice put on a blockbuster conference at NYU Law School to examine this everyday crisis. How can law enforcement priorities, from policing to sentencing, be shifted to reverse the 40-year trend of skyrocketing incarceration? What can Washington do—only 6 percent of the nation’s prisoners are federal—and what can the states do? The conference, which included everyone from Baltimore’s police commissioner to Attorney General Eric Holder, revealed a broad consensus against mass incarceration.

Many of the conferees also revealed, often inadvertently, just how distant the goal really is. There was Jeff Tsai, special assistant attorney general for California, who preened about the Golden State’s successful reduction of the state prison population, while failing to mention that 142,000 of such prisoners were merely crammed into county jails, or that the state is now spending $2 billion more on prisons than when Gov. Jerry Brown’s reform began in 2011. Paul Fishman, U.S. attorney for the district of New Jersey, informed the attendees that “nobody really wants us to stop enforcing any laws,” despite a growing trans-partisan chorus against wanton overcriminalization. And Neera Tanden, head of the Center for American Progress, a Democratic-leaning think tank, repeatedly intoned the phrase “public-private partnerships,” that New Democrat mantra void of all policy meaning.

Yet despite the countless failings of the Democratic Party on criminal justice matters, the political bidding for harsher policing and sentencing has been led for the past half-century by the right. From Nixon’s “War on Crime” to Reagan’s “War on Drugs”—campaigns that long ago ceased to be metaphors—it has been Republicans who have first leapt on more policing and longer sentences as an effective wedge issue, while Democrats anxiously wimp-proofed their right flanks by supporting the same measures with a little extra funding for midnight basketball.

And although mass incarceration is a nationwide problem, it is concentrated in red states, particularly in the South. Gov. Rick Perry loves to tout Texas’s very modest reforms in incarceration as a national model, but his state continues to lock up its people at a higher rate than that of the gulag-era Soviet Union—think about that—and about half again as much as California, twice as much as New York, and four times as much as Vermont.

Given the leading role of the American right in fueling hyper-incarceration, the participation of a few prominent conservatives at the Brennan Center’s conference was especially welcome and a tribute to the liberal host outfit’s political vision.

One of the conservatives was Mark Earley, who represented the coastal district including Virginia Beach for 10 years as a Republican member of the Virginia state senate before serving as the Old Dominion’s attorney general from 1998 to 2001. After losing the 2001 gubernatorial race to Mark Warner, Earley served for 10 years as president of the Prison Fellowship, the Christian ministry founded by reformed Watergate convict Chuck Colson. Now Earley practices criminal defense law.

On a beautiful October morning a few weeks after the conference, Earley and I talked about incarceration and politics and religion at a café on Central Park South.

Chase Madar: What were some of the tough-on-crime policies you pushed for as state senator and then as Virginia’s attorney general?

Mark Earley: There was the three-strikes repeat offender law. We lowered the age at which a juvenile could be tried as an adult from 16 to 14. We abolished parole—that was a big one—and we established mandatory minimum sentences. We went on a huge prison-building boom in the early 90s, developing so much excess capacity that we now take prisoners from other states.

CM: Did they seem right at the time?

ME: Oh yes. My mindset was that anything tough on crime is good for society. I didn’t really look at criminals as people—I sort of viewed them as “other.” It was a whole other world at that time. Huge increases in drugs and drug violence helped these policies get traction.

CM: Which of those harsh policies still seem right to you?

ME: Well, [there are] very few of them I’d go back and do again. I definitely wouldn’t lower the age you can try youths as adults from 16 to 14. I wouldn’t have abolished parole—now people have to do a minimum of 85 percent of their sentence in Virginia. And I wouldn’t have supported the three-strikes laws either. Or the prison-building boom. The vast majority of the inmates seem to have drug or alcohol addiction problems, and the cost to incarcerate them is about $20,000 a year. I would have found another way.

CM: You said at the conference that you’re dedicating the second half of your life to undoing the things you did in the first half. Did you have a Damascene moment that changed your mind and heart about our harsh criminal-justice policies?

ME: I think I had a couple of them. When I was attorney general of Virginia, I initiated a task force on gangs and juvenile violence. I asked to speak with inmates convicted of violent crimes. I talked to 40 or so kids, and I found out that only one of them had grown up with his father present. The young guy who made the biggest impression on me, he was African-American and 16, was convicted of a double murder. He never knew his dad, although once, the kid said, he and his father recognized each other when the dad bought drugs from the kid. This doesn’t excuse violent behavior, but it does go a long way in explaining it.

Another thing that made prisoners cease being “other,” that made me realize these people could be me or my child, was visiting prisons. I visited literally hundreds of prisons. To see it and to—well, smell it for myself. There usually wasn’t any rehabilitation or correction going on, we’re just warehousing a large number of people, and the number of them benefiting was so small.

When I visited as a Prison Fellowship guy it was a very different experience from visiting as a government official, where a warden will give you the official line. But to talk to prisoners one-on-one and to the corrections officers, you get a different kind of story. You begin to see the collateral damage of imprisoning people. The greatest sociological predictor of whether someone’s going to wind up in prison isn’t their color or their income, it’s whether their father or mother is in prison. The consequences are just devastating.

CM: Conservatives tend to value order and authority but also limited government—how does this tension play out in the politics of criminal justice?

ME: There was never a coherent, intellectually thought-out position on law enforcement compared to the more robust, well thought-out conservative approach to taxation.

But, of course, one of the most potent arguments against mass incarceration, for conservatives, is that if you believe in limited government and are against dependence on the state, and you look at our criminal-justice system, you’re just not going to be very impressed by it. We have about one out of every hundred adults in this country under total state control. Think about that.

I saw that in the ’80s and ’90s, criminal-justice policies were driven more by what constituents wanted, what worked in the short term. But if you do that long enough, then all your constituents wind up having family members in jail.

CM: Our country has a long history of slavery and white supremacy—and not just below the Mason-Dixon line. About 40 percent of those incarcerated in our country are black Americans. What role does racism play in our penal state?

ME: I think it’s pretty significant. You can’t have centuries of slavery and abject racism that don’t have consequences across generations. Without a fair amount of consideration, minorities are gong to fare poorly in any criminal-justice system, especially with a history of marginalization.

CM: What do you think about the death penalty?

ME: I can make a conceptual argument for why the death penalty is necessary. But I find it harder and harder to make a practical argument because our system has such a hard time getting it right.

CM: We as a country are much more religious than peer nations, and Christianity is a real force here. Christianity is supposed to value mercy and forgiveness—how does this reconcile with our intense punitive zeal as the world’s biggest incarcerator?

ME: Well, you can never accuse Christians of getting it right all the time! I’m not sure religion is behind it. Politics had a lot to do with it. From the ’60s through the ’80s, there was an increase in drug use and crime, the Willie Horton ad, crime being used as a wedge issue.

CM: You were president and CEO of the Prison Fellowship for 10 years—could you tell me a little about that organization’s policy reform arm, the Justice Fellowship, and what they’re doing to reduce incarceration, whether through sentencing reform, or policing reform, or other efforts?

ME: “Restorative justice” is what we’re after. When community norms are transgressed—a “crime”—the goal is to bring restoration to the victim. And the United States government does not have this perspective. I saw it firsthand in Rwanda, where so many people killed, and were killed, and so many people [were] imprisoned later. Restorative justice was the model for letting perpetrators go back to the communities where they slaughtered people and confess at public meetings and ask forgiveness.

CM: Prison Fellowship ministers to the religious needs of prisoners, but what does it do to change government policy on sentencing, on mass incarceration?

ME: Prison Fellowship has been heavily involved through its advocacy wing, the Justice Fellowship, in undoing mandatory minimums, in shrinking the crack/powder-cocaine sentencing disparity. State by state, they are very involved in getting local governments to look at alternatives to incarceration, especially regarding technical violations of parole. And another goal is to make punishments swift and certain but also small. Hawaii and Kansas are leaders there.

Pat Nolan, a former California state legislator who was convicted for accepting illegal campaign contributions, is the head of it all.

CM: There’s some push right now towards federal sentencing reform and federal clemency—which is great, but only 6 percent of the nation’s prisoners are in the federal system. What then can Washington do to alter state and local policy, for instance with federal grants and the incentives they carry?

ME: States get a lot of federal grant money, and I support the idea of tying federal grant money to reducing incarceration, reducing recidivism, doing what works. National standards can be a good thing. I’m a firm believer in working things both ways, both in the states and federally.

CM: You don’t see this as an unwanted federal intervention? Or is this then the federal government working to undo its intervention from the past 40 years, ever since Nixon declared a “War on Crime.”

ME: Blame here is a blanket that goes across the whole of the country, at every statehouse, and Washington too. National campaigns—when one candidate tries to outbid the other on who’s tougher on crime—have had a huge impact on the criminal-justice debate everywhere.

CM: What’s politically feasible in the short term for undoing mass incarceration, and what’s in the long-term horizon?

ME: When I was in Washington recently I sat in on a meeting organized by Right on Crime, and it was telling who was there, the variety of high-level people. There was Gary Bauer. Newt Gingrich. David Keene, the former NRA head and Washington Times opinion editor. But the only congressional staffer present was a Rand Paul aide, even though Republican legislators had all been notified and invited.

It’s true that there’s a more libertarian, more hands-off view growing, particularly with young people. What impact will Rand Paul’s likely presidential candidacy have? Will the issue of mass incarceration resonate with voters? It was a game-changer when George W. Bush talked about the need for better prisoner re-entry programs in one of his State of the Union addresses.

CM: You work as a defense lawyer now. Is this new? I just assume that since you were a state attorney general you were a former DA, like most state AGs and a depressingly large proportion of our political elite.

ME: Oh no. I’ve always worked as a defense lawyer. And that’s what my legal practice is now.

Mark Earley is not the only conservative who now sees mass incarceration as a social evil rather than a policy solution. But like other mass traumas, the prison explosion has generated thick clouds of mythological fakelore. For instance, many conservatives ardently believe that mass incarceration is somehow a byproduct of the welfare state’s postwar growth. But like it or not, nations with more lavish welfare states, as in Northern Europe, have radically lower incarceration rates than those of the United States. On the left, it’s widely believed that ending the war on drugs would speedily end mass incarceration. But the fact is, even if every single nonviolent drug offender were released tomorrow, the U.S. would still have the highest imprisonment rate in the world, barring a few tiny outlier island nations.

(According to a paper by Yale legal scholar James Forman Jr.—“Racial Critiques of Mass Incarceration: Beyond the New Jim Crow”—nonviolent drug offenders make up 20-25 percent of our incarcerated total. Significant, to be sure, but far from everything.)

Undoing our carceral state will require a generational effort. The modest dips of recent years—in 2013 the absolute number of prisoners increased slightly, while the incarceration rate dropped a little due to population growth—are only marginal improvements. Depenalizing American society is going to take changes in policing, in sentencing, in parole and reentry, and in leaching out the lust for state punishment that has come to saturate so much social policy.

No political tribe has the power to do this by itself. Even in states with conservative supermajorities, rolling back Soviet-like incarceration levels is not going to happen without broad coalitions—not blandly “centrist” but quirkily transpartisan. The real challenge is going to be shifting the enormous depoliticized mass in the ideological middle, which has become passively radical in its acquiescence to hyperincarceration and overpolicing.

How will we get there? Movement building isn’t easy; it takes a lot more than the Internet’s latest social media app. It means a lot of meetings, and an unhappy ratio of listening to talking. But on criminal-justice matters, now is a time for comity—a time for bridge-building rather than ideological border-patrolling. It is also a time for intelligent aggression against the bipartisan backers of the American prison state.

Chase Madar is an attorney in New York and the author ofThe Passion of Bradley Manning: The Story Behind the Wikileaks Whistleblower.