Orange Is The New Black author Piper Kerman, whose memoir about her time in federal prison has been adapted into the successful Netflix series of the same name, says she is cautiously optimistic about attorney general Eric Holder's recent promise of a more just justice system.

Kerman joined other experts on US incarceration in New York on Tuesday to discuss American prisons. She and the other panel members – among them journalists covering the US prison system and others who work with people behind bars – tentatively agreed that Holder's proposals could be a sign of improved sentencing structures in a system that has incarcerated 2.9% of the US population and put nearly 5 million adults on probation or parole, as of 2011 (pdf).

"I think it is very powerfully symbolic for the top prosecutor in the country to speak so directly to the senselessness of mandatory minimums and some of the practices that have clearly not worked over the past 30, 40 years," Kerman said. "However, Eric Holder also spoke very eloquently about our need to strengthen our public defense system the year before, and that's really been it."

Holder's proposals were offered in an attempt to curb the enormous US prison population through measures including cutting mandatory minimum sentences for non-violent drug offenders.

Kerman, like her television counterpart Piper Chapman, was indicted for money laundering and drug trafficking charges. Most of that time was served in a minimum-security prison, where the show is currently set, and where prisoners have more freedom to move.

"I didn't really understand to what extent that was sort of a best case situation until I was moved to an Oklahoma City federal transit center on the federal airlift," Kerman said. She said being shackled with hundreds of other prisoners and being physically confined had detrimental psychological effects, even at the end of her sentence.

"Intellectually I knew that I was close to going home, but when I reflect on how powerful the impact of that physical confinement was, on my, certainly my spirits, but also my psychology, by the time I was close to release, I was really skeptical that they were going to let me go, which was really irrational, but was really true," Kerman said.

She endorsed the common prison sentiment: "Do the time, don't let the time do you," but took issue with another: "You walk in alone and walk out alone." Though she said it is difficult to form friendships when one person could have a 24-month sentence and the other a 200-month sentence, it was an important part of her survival.

"There is also, in addition to the things the system tries to take away from you – in terms of your dignity, your femininity, a range of things – there is this sense of loss and fear of that engagement," Kerman said. "But the flip side of that is: I can't imagine doing time – whether short time or long time – without those friendships, without those relationships, that's where you get all your strength from."

Relationships between inmates is an overarching theme of the show, as is racism – an issue that panelists agreed was responsible for the most prominent problems in the US prison system.

Vincent Southerland, senior counsel at the Naacp Legal Defense and Education Fund's criminal justice practice, described the Louisiana State Penitentiary at Angola, where people drive up to the gates and can see black men in prison jumpsuits and white men on horseback with rifles. The prison itself is named after a country slaves were brought from and surrounded by former plantations.

"That history is still with us and so much of it is wrapped up in the way we think about people, the way we dehumanize people, the way we treat people for the transgressions they commit," Southerland said. "Until we are able to deal with all of that and have real conversations about how we work through those issues we all have, there is going to be very slow change in society."

He called Holder's comments great, even if they do turn out to be merely symbolic, and noted that only 10% of prisoners are in federal jails, which means only a small number of prisoners would be affected by these reforms unless states take action.