The differential treatment of crack cocaine and powder cocaine by America's criminal "justice" system has been exposed as discriminatory and admitted to be unfair. Yet, the secret nightmare continues for thousands of African Americans still in prison for crack cocaine offenses, while people convicted of powder cocaine offenses – the majority of whom are white or Hispanic – have served far shorter sentences. Even as the US government has reformed the injustice of punitive sentencing for crack, it has doubled down on the injustice for those imprisoned before the reforms.

I'll never forget the first time I had to explain federal crack cocaine laws to a client. I was 25 years old, fresh out of law school, and working as a public defender in Alabama. I had come to the local jail, where my client was being held in a 6ft x 8ft cell with three other men and no access to fresh air or to a single window.

My client thought he would be released from jail and home with his family after our first appearance in court. Instead, I told him, the tiny bag of crack cocaine that police had found in his car – less than half the size of a ping-pong ball – meant that he would likely spend the next five to 40 years in prison.

Last month, President Obama quietly did something that should shake every American to the core. Seeking to enforce federal crack cocaine laws that have since been repealed, the Obama administration asked a federal appeals court to ensure that thousands of human beings, mostly poor and mostly black, remain locked in prison – even though everyone agrees that there is no justification for them to be there.

To understand Obama's decision and to understand why you have probably not heard about this, it is important to realize just how normalized rampant incarceration has become in our society. America now puts people in prisons at rates unparalleled in the modern world and unprecedented in American history. The US has 5% of the world's population and 25% of its prisoners.

Half of all human beings in federal prison are there because of a nonviolent drug offense. The number of nonviolent drug offenders in prisons and jails around America has increased 1100% since 1980. The vast majority of those in prison are very poor. African Americans constitute only 14-15% of the nation's drug users (about the same as their percentage of the general population); yet, they constitute 37% of those arrested for drug offenses, 59% of those convicted for drug offenses, and 74% of those sentenced to prison for a drug offense.

The result: twenty years or so into the "war on drugs", America incarcerated blacks at a rate six times higher than South Africa did during apartheid.

Until President Obama signed the Fair Sentencing Act, on 3 August 2010, there had been, for over two decades, a 100:1 disparity in the quantities of drugs seized that would trigger the mandatory minimum penalties faced by tens of thousands of people like my client. In other words, while my client's five to 40 years was triggered by only 5g of crack cocaine, it would take 500g (half a kilo) of powder cocaine to trigger the same penalty.

A further disparity: 82% of federal crack cocaine defendants are African-American; only 27% of powder cocaine defendants are African-American. In the early years of the 100:1 ratio, from 1988 to 1995, no whites at all were brought to trial by federal prosecutors under the crack provisions in 17 states, which included major cities such as Boston, Denver, Chicago, Miami, Dallas, and Los Angeles.

The president, Congress, and every serious expert agreed that this disparity was unjustifiable. Indeed, the two substances are pharmacologically identical. Curiously, though, without giving any reasons, Congress did not reduce the disparity to 1:1, as even federal law enforcement officials had conceded was appropriate. Instead, it picked the nice round number of 18:1. Even in making that arbitrarily decision, however, Congress did not indicate what was to happen to all those already sitting in prison under a law that Congress and the White House had now found to be unjust.

This was a moment almost unique in history: when a supposedly free society admitted that many of its citizens remain in prison for no good reason.

Strangely, the Obama administration initially urged in federal courts across the country that the old discriminatory penalties should still be applied to those arrested but not yet sentenced at the time the law was passed. However, the administration reversed course after significant criticism, and the US supreme court held last year that the new, more "fair" sentences must be applied to those not yet sentenced.

But that case did not decide the fate of any of the thousands of people already sitting in prison because of what all agree is an unfair law. For those people – sentenced, in some cases, just days or weeks before the Fair Sentencing Act was signed – our society's acknowledgment that they remain in prison for no good reason may not help them at all – because the government did not care to reduce their penalties retroactively when it declared them unjust.

For several years, federal judges have done nothing to remedy this injustice; one famously concluded that the prisoners sentenced under the old law had simply "lost on a temporal roll of the cosmic dice". So, there are American citizens serving tens of thousands of years in prison because, according to all three branches of government, it's just their tough luck?

Apparently so, until two months ago. On 17 May 2013, the US court of appeals for the sixth circuit held that the new, "fair" sentences must be applied to all those previously sentenced under laws that everyone acknowledges were discriminatory. The two-judge majority opinion wrote forcefully (pdf) and with unusual candor about the history of unequal treatment under the old laws. The judges ordered that those sentenced under those laws were entitled to ask federal judges to reduce their sentences.

The Justice Department is now seeking to overturn that decision – which will be devastating news to many thousands like my original crack cocaine client. The Obama administration would surely condemn an oppressive foreign dictator's regime for the singular cruelty of declaring to its population that thousands of its citizens must continue to sit in prison for no good reason. The fact that few have even heard of the stunning position taken by President Obama is a sad reflection on how incurious mainstream US public opinion is about what underpins our mass incarceration society.

• Editor's note: though not technically incorrect, this article was amended to include the phrase "or Hispanic" in the first paragraph, at 1.30pm ET on 23 July