Non-violent drug offenders will avoid mandatory minimum sentences under proposals to cut America's soaring prison numbers due to be unveiled on Monday.

Attorney general Eric Holder is expected to issue new guidance to prosecutors that could have a dramatic effect on the lengthy jail terms that judges are required to give anyone convicted of possessing more than a proscribed amount of certain drugs.

In the future, many low-level charges against dealers not judged to be part of a large gang or cartel will no longer detail how much they were caught with – side-stepping the federal minimum sentencing laws.

Holder is also expected to announce other proposals to curb America's vast prison population, including compassionate early release for elderly inmates who are no longer viewed as dangerous and will promote drug-treatment programs as prison alternatives.

"We must never stop being tough on crime. But we must also be smarter on crime," he is expected to tell the American Bar Association on Monday afternoon. "Too many Americans go to too many prisons for far too long, and for no good law-enforcement reason … Although incarceration has a role to play in our justice system, widespread incarceration at the federal, state and local levels is both ineffective and unsustainable."

The US accounts for 25% of the world's prison numbers, even though it has only 5% of the world's population. Drug-related offences drive the vast majority of this, and people convicted of conspiring to sell 5kg of cocaine will currently receive a 10-year mandatory minimum sentence.

Aggressive enforcement of federal criminal laws is necessary, but "we cannot simply prosecute or incarcerate our way to becoming a safer nation", Holder said. "Today, a vicious cycle of poverty, criminality and incarceration traps too many Americans and weakens too many communities. However, many aspects of our criminal justice system may actually exacerbate this problem, rather than alleviate it."

"We need to ensure that incarceration is used to punish, deter and rehabilitate – not merely to convict, warehouse and forget," said the attorney general.

Holder said mandatory minimum sentences "breed disrespect for the system. When applied indiscriminately, they do not serve public safety. They have had a disabling effect on communities. And they are ultimately counterproductive."

Senators Dick Durbin, Patrick Leahy, Mike Lee and Rand Paul have introduced legislation aimed at giving federal judges more discretion in applying mandatory minimums to certain drug offenders.

Holder said new approaches – which he is calling the "Smart On Crime" initiative – are the result of a Justice Department review he launched early this year.

The attorney general said some issues are best handled at the state or local level and said he has directed federal prosecutors across the country to develop locally tailored guidelines for determining when federal charges should be filed, and when they should not.

"By targeting the most serious offenses, prosecuting the most dangerous criminals, directing assistance to crime 'hot spots,' and pursuing new ways to promote public safety, deterrence, efficiency and fairness – we can become both smarter and tougher on crime," Holder said.

The attorney general said 17 states have directed money away from prison construction and toward programs and services such as treatment and supervision that are designed to reduce the problem of repeat offenders.

In Kentucky, legislation has reserved prison beds for the most serious offenders and refocused resources on community supervision. The state, Holder said, is projected to reduce its prison population by more than 3,000 over the next 10 years, saving more than $400m.

He also cited investments in drug treatment in Texas for non-violent offenders and changes to parole policies which he said brought about a reduction in the prison population of more than 5,000 inmates last year. He said similar efforts helped Arkansas reduce its prison population by more than 1,400. He also pointed to Georgia, North Carolina, Ohio, Pennsylvania and Hawaii as states that have improved public safety while preserving limited resources.

Holder also said the department is expanding a policy for considering compassionate release for inmates facing extraordinary or compelling circumstances, and who pose no threat to the public. He said the expansion will include elderly inmates who did not commit violent crimes and who have served significant portions of their sentences.