The United States has the largest incarcerated population in the world, climbing from 600,000 to over 2 million in just a few decades. We also have the highest percentage of our population behind bars of any country. The people most likely to languish behind bars are Black, Latino, Native American, and poor. It’s a legacy rooted in Jim Crow-era policies that continues in the thinly veiled racism of the war on drugs, as lawyer Michelle Alexander points out in her book The New Jim Crow.

But there’s another number that dwarfs the prison population: Nearly 5 million people are under some sort of parole or probation supervision in the United States. That’s a fourfold increase since 1980, according to a new report released this week by the Columbia University’s Justice Lab. During that time, the requirements for people under judicial supervision have become more stringent. The number of conditions people must adhere to has increased, for example, as has the length of supervision required. As a result, many people wind up in jail—not for committing another crime, but for a technical violation of probation or parole conditions.

In fact, from 1990 to 2004, the rate of people on probation who were sent back to jail for non-compliance grew by 50 percent, from 220,000 to 330,000, according to the report.

This suggests a way to vastly reduce the rate of imprisonment in the United States, just by reducing the number of people who wind up in jail because of these technical violations.

In some cases, the infractions that send someone back to jail are as simple as coming late to a meeting with a parole officer or failing to make payments on a fine. Being a person of color increases the odds of winding up in prison for a parole violation, according to a study by the Urban Institute.

One of the most chilling reasons for being sent back to prison is failure to pay a fine or court or supervision fees, payments that can be out of reach for the low-income people most likely to be caught up in the criminal justice system.

In some jurisdictions, about 20 percent of those serving time were incarcerated because they didn’t pay their criminal justice debts, according to a Council of Economic Advisors issue paper.

Consider the impossible situation faced by newly released inmates. Many are poor when they enter the system; along with their conviction comes fines and fees. As one example, in Washington state, these average $1,300 for a felony conviction, according to research by University of Washington sociology professor Alexes Harris.

Interest is charged on the original debt, and by the time an inmate is released, their debt may have grown quite large. Finding a job that pays enough to make payments on these debts is difficult after incarceration, especially since public housing and other services are denied to those convicted felons. This challenge is even greater for released inmates who are mentally ill, physically disabled, have a history of substance abuse, or have few social support systems.

“High fines and fee payments may force … difficult trade-offs between paying court debt and other necessary purchases,” says the Council issue paper. “Unsustainable debt coupled with the threat of incarceration may even encourage some formerly incarcerated individuals to return to criminal activity to pay off their debts.”

These fines and fees result in what is tantamount to debtors’ prisons. They are expensive and counter-productive.

Which suggests a way out. The Justice Lab report proposes policies that would increase the chances that someone who has served their time will successfully transition to life after prison.

These policies include reducing the number of years of supervision on probation or after release. This could reduce the chances that a routine infraction would result in additional fines and jail time, and also reduce caseloads of overworked parole officers, who could then focus on providing support to parolees who are at risk of reoffending. Or we could shorten time on probation or parole based on “good behavior,” so people can work their way out of the system. And we could reduce fines and fees, so there is less financial pressure on people just as they are trying to regain their footing.

Van Jones, host of CNN’s new segment The Van Jones Show and founder of #cut50, a national effort to cut in half the people behind bars, put it this way, according to a news release for the Justice Lab report: Now is the time to “reimagine how probation and parole can truly help people reintegrate back into society—rather than simply being a trap that leads to needless reincarceration.”

The Justice Lab report shows that jurisdictions that have taken this route have seen reductions in crime, reductions in spending, and fewer people locked up. A win-win-win, except for those who profit from the prison-industrial complex.