(CNN) Looking to capitalize on an unusual political moment that unites him with some of his fiercest conservative rivals, President Barack Obama on Tuesday made a case for overhauling aspects of a criminal justice system that has locked away millions of Americans -- many of them of young black men -- for non-violent crimes.

During a fiery speech at the annual convention of the NAACP in Philadelphia, Obama declared the current U.S. justice system "remains particularly skewed by race and by wealth," and called for legislation that would minimize, or eliminate altogether, minimum sentences for non-violent drug crimes.

And he spoke bluntly about the conditions inside U.S. prisons -- including prison rape and solitary confinement -- which he said "have no place in any civilized country."

"In too many places, black boys and black men, and Latino boys and Latino men, experience being treated different under the law," Obama said, claiming his assertion wasn't "anecdote" or "barber shop talk," but instead backed by data.

"Mass incarceration makes our country worse off and we need to do something about it," Obama said.

The president's speech came as a confederation of support for new sentencing laws spreads across the political spectrum. Obama and his aides hope the bipartisan endorsements will help spur lawmakers into taking action to help resolve deep-seeded problems that have led to the largest prison population in the world.

Among the advocates for easing mandatory minimum sentencing laws: Hillary Clinton, the 2016 Democratic frontrunner, whose husband oversaw large increases in the prison population as president in the 1990s; Sen. Rand Paul , running for the GOP nomination, who has introduced legislation reforming sentencing laws; and Charles and David Koch, the industrialist billionaires who have funded efforts backing reform.

"The eyes of more Americans have been opened to this truth," Obama said. "The good news -- truly good news -- is that good people of all political persuasions are starting to think we need to do something about this."

Making the case that poor prison conditions lead to higher reincarceration rates, Obama called for new examination of how America's criminals are held.

He said that included refusing to tolerate prison rape by using it as a punchline, and reviewing the practice of solitary confinement, which some studies show has adverse psychological effects on prisoners. He said he'd asked his Attorney General Loretta Lynch to reassess solitary confinement practices at the federal level.

"DOJ (Department of Justice) will conduct a review of restrictive housing (solitary confinement) in federal, state and local prisons to both identify the extent of its use, as well as potential reforms or best practices. Upon completion of the review," said "Justice Department spokeswoman Melanie Newman said. The results will be reported back to the President.

When prisoners complete their sentences, they shouldn't necessarily be forces to identify as criminals when they apply for jobs, Obama said. And he proclaimed they should be able to vote.

Those topics are likely to arise again on Thursday, when Obama becomes the first sitting president to visit a federal prison when he tours the El Reno medium-security facility in Oklahoma.

His speech to the NAACP came a day after Obama commuted the federal prison sentences of 46 non-violent drug offenders, a move that brought the total number of commutations he's delivered to nearly 90 — the most since President Lyndon B. Johnson. He's now commuted more sentences than his four predecessors combined.

But even those statistics hardly compare to the tens of thousands of similar cases that are still winding their way through a long and complex system. A massive backlog of federal clemency applications has swelled since the administration began soliciting requests from prisoners serving long sentences for drug related crimes — more than 15% of the total federal prison population has now petitioned Obama for a sentence reduction.

"This is an all-hands-on-deck situation," said Cynthia Roseberry, the director of Clemency Project 2014, a group created to help funnel clemency applications to the Department of Justice.

Many of those applying were sentenced during a drug-and-crime wave in the 1980s and 90s, when fears over urban violence led to harsh new sentencing laws that led to mass incarcerations. Many of the prisoners whose sentences Obama has commuted were punished for crack cocaine violations, which came with drastically higher mandatory sentences than those for powdered cocaine. The discrepancy in the laws disproportionately affected minorities and the poor.

Reviewing and granting clemency to all of those prisoners is a task that both reform advocates and White House admit could be impossible to surmount.

"This is such a tip of the iceberg. These 46 people are lucky," said Julie Stewart, the founder and president of Families Against Mandatory Minimums. "But there are literally over 2,000 people in federal prison serving life without parole for non-violent offenses. I doubt the president is going to get to each one of them before he leaves office."

Like other domestic items Obama hopes could form his domestic legacy — including immigration reform and climate change — the most powerful action on criminal justice reform can only come through Congress.

Republican Sen. John Cornyn of Texas and Democratic Sen. Sheldon Whitehouse of Rhode Island proposed a plan that would allow for prisoners to reduce their sentences by participating in jobs and other programs. The measure hasn't yet come up for a vote in the Senate.

Another bipartisan pair of senators — Sen. Dick Durbin, D-Illinois and Sen. Mike Lee, R-Utah —introduced a measure that would retroactively apply changes in crack cocaine sentencing laws to prisoners who were incarcerated years ago. It also hasn't been taken up by the full Senate.

Obama's push to reduce minimum sentences for non-violent drug offenses, along with other items on his agenda, including raising the minimum wage or expanding overtime pay, is partly an attempt to address racial inequality in the United States. The NAACP has long advocated bringing sentencing guidelines in line with the crimes they're meant to punish.

When he spoke to the group on Tuesday, Obama made the point that the resources used now to incarcerate millions of Americans could be better diverted to crime prevention or public safety programs.

And he reiterated the point he underscored in a White House video announcing the sentence commutations handed down Monday -- that "at its heart, America is a nation of second chances."

"While people in our prisons made mistakes -- sometimes big mistakes -- they are also Americans," he said. "We have to ensure as they do their time and pay back their debt to society we are increasing the possibility they can turn their lives around."