The US government took the first tentative steps toward tackling its 1.5m-strong prison population on Monday by announcing that minor drug dealers would be spared the mandatory minimum sentences that have previously locked up many for a decade or more.

Reversing years of toughening political rhetoric in Washington, attorney general Eric Holder declared that levels of incarceration at federal, state and local levels had become both "ineffective and unsustainable."

The Department of Justice will now instruct prosecutors to side-step federal sentencing rules by not recording the amount of drugs found on non-violent dealers not associated with larger gangs or cartels.

"Our system is in many ways broken," Holder told the American Bar Association in San Francisco. "As the so-called war on drugs enters its fifth decade we need to ask whether it has been fully effective and usher in a new approach."

"Too many Americans go to too many prisons for far too long and for no truly good law enforcement reason," he said, adding later: "We cannot simply prosecute or incarcerate our way to becoming a safer country."

Holder also announced a review into sentencing disparities, pointing to a recent study showing black male offenders received sentences nearly 20% longer than whites convicted of similar crimes. "This isn't just unacceptable, it is shameful," said Holder.

Since Richard Nixon declared the "war on drugs" in 1971, US prison numbers have soared to account for 25% of all the world's prisoners even though it has only 5% of the world's population. Drug-related offences drive the vast majority of this, and people convicted of conspiring to sell 5kg of cocaine will currently receive a 10-year mandatory minimum sentence.

But the US total of 1,571,013 prisoners has begun to edge down in the last three years amid falling crime rates, and several recent legislative attempts to reform sentencing policy have led to hopes that the era of mass incarceration may be coming to an end.

Mark Mauer, executive director of The Sentencing Project, an advocacy group for reform, described Holders proposals as a "significant development" which he hoped would stimulate debate and effect real change.

"Since the War on Drugs, there have been huge developments in drug courts and drug treatment but mandatory sentencing has acted against those. This represents one way to open that up a bit and increase the potential scope of other options."

Around half of the 200,000 people in federal prisons are locked up for drug offences and about 60% are sentenced under mandatory sentencing provisions, according to Mauer. Around 45% of the 25,000 people incarcerated every year for drug offences are lower level offenders such as street level dealers and couriers.

Ethan Nadelmann, executive director of the Drug Policy Alliance, also welcomed Holder's proposals but said that they could have been put in place earlier to avoid the unjust suffering of thousands of Americans and their families as the prison population continued to grow.

Nadelmann said: "There's no good reason why the Obama administration couldn't have done something like this during his first term – and tens, perhaps hundreds of thousands of Americans have suffered unjustly as a result of their delay."

"But that said, President Obama and Attorney General Holder deserve credit for stepping out now, and for doing so in a fairly decisive way."

Nadelmann said that the national politics of the issue has shifted significantly recently, enabling support for Holder's "bold" proposals.

Republican governors and Senators, such as Rand Paul of Kentucky and Mike Lee of Utah, have worked to allow judges to depart from mandatory minimum sentencing when circumstances merit and bipartisan bills are being introduced on the issue.

Nadelmman said: "Holder gives credit to Red states like Texas and Arkansas, while California and New York have decreased incarceration to a greater extend. But it was politically wise to do so."

Holder also announced other proposals to curb America's vast prison population, including compassionate early release for elderly inmates who are no longer viewed as dangerous and will promote drug-treatment programs as prison alternatives.

"We must never stop being tough on crime. But we must also be smarter on crime," he said. "Although incarceration has a role to play in our justice system, widespread incarceration at the federal, state and local levels is both ineffective and unsustainable."

Aggressive enforcement of federal criminal laws is necessary, but "we cannot simply prosecute or incarcerate our way to becoming a safer nation", Holder said. "Today, a vicious cycle of poverty, criminality and incarceration traps too many Americans and weakens too many communities. However, many aspects of our criminal justice system may actually exacerbate this problem, rather than alleviate it."

"We need to ensure that incarceration is used to punish, deter and rehabilitate – not merely to convict, warehouse and forget," said the attorney general.

Holder said mandatory minimum sentences "breed disrespect for the system. When applied indiscriminately, they do not serve public safety. They have had a disabling effect on communities. And they are ultimately counterproductive."

Holder said new approaches – which he is calling the "Smart On Crime" initiative – are the result of a Justice Department review he launched early this year.

The attorney general said some issues are best handled at the state or local level and said he has directed federal prosecutors across the country to develop locally tailored guidelines for determining when federal charges should be filed, and when they should not.

"By targeting the most serious offenses, prosecuting the most dangerous criminals, directing assistance to crime 'hot spots,' and pursuing new ways to promote public safety, deterrence, efficiency and fairness – we can become both smarter and tougher on crime," Holder said.

The attorney general said 17 states have directed money away from prison construction and toward programs and services such as treatment and supervision that are designed to reduce the problem of repeat offenders.

In Kentucky, legislation has reserved prison beds for the most serious offenders and refocused resources on community supervision. The state, Holder said, is projected to reduce its prison population by more than 3,000 over the next 10 years, saving more than $400m.

He also cited investments in drug treatment in Texas for non-violent offenders and changes to parole policies which he said brought about a reduction in the prison population of more than 5,000 inmates last year. He said similar efforts helped Arkansas reduce its prison population by more than 1,400. He also pointed to Georgia, North Carolina, Ohio, Pennsylvania and Hawaii as states that have improved public safety while preserving limited resources.

Holder also said the department is expanding a policy for considering compassionate release for inmates facing extraordinary or compelling circumstances, and who pose no threat to the public. He said the expansion will include elderly inmates who did not commit violent crimes and who have served significant portions of their sentences.