Even without the revelations of widespread domestic government surveillance, 2013 was hardly a banner year for human rights in the United States, according to a report released this week by Human Rights Watch.

In its World Report 2014, HRW singled out the U.S. criminal justice system as a major source of human rights violations, thanks to systemic problems that range from severe sentencing requirements to the misuse of solitary confinement. The U.S. has the highest incarceration rate of any country in the world. While the overall U.S. population has decreased over the past four years, some 1.6 million people were incarcerated in federal and state prisons at the end of 2012. An additional 700,000 were held in local jails.

The report also pointed out that the criminal justice system is filled disproportionately with men of color, a point both President Obama and Attorney General Eric Holder have mentioned while discussing the need for sentencing reforms in drug cases. Forty-four percent of people serving time in federal prisons for drug crimes are African-American, while they comprise only 13% of the total U.S. population.

Mandatory minimum sentences played a huge part in the prison population boom of the past 30 years and in the explosion of elderly prisoner populations that will continue to increase. A report issued by the American Civil Liberties Union in November found that more than 3,200 people were serving life sentences for non-violent crimes. A December Human Rights Watch report alleged widespread pressure on defendants to plead guilty or face extreme prison terms.

Solitary confinement also drew attention in 2013 when thousands of inmates in California protested against prison conditions there. Prisoners at the Secure Housing Unit at the state’s notorious Pelican Bay State Prison played a major role in that protest. Approximately 400 inmates in the facility have been held in solitary confinement for more than a decade. Extended solitary confinement is recognized as torture by the international human rights community.

The criminal justice system was not the only institution to lock people up with little to no recourse. Immigration detention has skyrocketed during Obama’s presidency, and hundreds of thousands of non-citizens are detained and deported each year. Individuals dealing with immigration proceedings receive fewer civil liberties guarantees than people held in the criminal justice system. The Human Rights Watch report noted that Congress failed to make any meaningful progress toward comprehensive immigration reform last year.

America’s human rights issues are not only related of domestic policy. Despite revived efforts to close the prison at Guantanamo Bay, 152 men remain in limbo there, and that number is unlikely to approach zero anytime soon. Last year saw a widespread hunger strike at the prison that grew to more than 100 prisoners and brought renewed scrutiny to the 12-year-old prison. Obama promised again in May to close the facility. Six detainees have been transfered since August, but even if all 82 of the remaining prisoners who have been cleared for release or transfer leave the prison, serious problems remain.

There are 45 men the U.S. still considers to be too dangerous to ever set free, but who cannot be tried because there is not enough evidence for a trial. Human rights groups in the United States and abroad have repeatedly called on the president to close the prison at Guantanamo and to stop holding people without charging them with a crime.

Obama and his adminstration have made some strides toward addressing the prison system’s major problems. In August, Attorney General Holder announced new sentencing guidelines for drug crimes in an effort to reduce the number of prisoners serving long sentences for non-violent drug crimes. On Jan. 23, Holder also urged Congress to pass the Smarter Sentencing Act, a bill that would give judges greater discretion when making sentencing decisions.

While it is too early to know if U.S. attorneys will follow the spirit of the new guidelines when it comes to charging and sentencing, it could have a serious effect on how large a part the criminal justice system plays in next year’s human rights status report.