Washington (CNN) Lisa Dornell loved her job. For 24 years, she sat on the bench in Baltimore's immigration court, hearing hundreds of cases of immigrants trying to stay in the United States.

"It was an honor. It was a privilege to be able to preside over so many different cases and be able to grant relief to people who needed relief," Dornell told CNN in an interview.

But she walked away from that job in April -- a decision that still invokes a wave of emotion when she recalls it. "The toxic environment made it both harder and easier to leave," Dornell said.

Over the past year, in the heat of a border migration crisis, 45 judges have left, moved into new roles in the immigration court system -- which is run by the Justice Department -- or passed away, according to the department. That's nearly double the number who departed their posts in fiscal years 2018 and 2017, when 24 and 21 judges left, respectively, according to data provided by the judges union.

The reasons why individual judges have moved on from their posts on the bench vary, but in interviews with judges who left in recent months, one theme ties them all together: frustration over a mounting number of policy changes that, they argue, chipped away at their authority.

Their departures come as the Justice Department faces a backlog that exceeds 1 million cases. The bogged-down system has led to immigration cases being pushed out years in the future, leaving many immigrants residing in the US unsure if they'll be allowed to stay or be ordered removed.

President Donald Trump has repeatedly criticized the nation's immigration system, specifically taking issue with the practice of releasing immigrants while they await their court dates. To remedy that, the administration has sought to hire more immigration judges. Most recently, the immigration judge corps hit a record high , though the Justice Department still has to contend with judges leaving over policy disagreements.

In a statement to CNN, the Justice Department's Executive Office for Immigration Review spokeswoman, Kathryn Mattingly, said the agency "continually plans for attrition, and both improvements to the hiring process and a policy of 'no dark courtrooms' help minimize the operational impact of (immigration judge) separations and retirements."

The agency doesn't track individual reasons for retirements or departures, Mattingly said.

Immigration judges -- employees of the Justice Department -- are charged with following the policies set by each administration.

"The nature of the job ebbed and flowed as administrations changed," Dornell recalled. "It was always tolerable. We all work with a realization that it's the prerogative of the administration to implement policies as they see fit."

The Trump administration was no exception. Trump's first attorney general, Jeff Sessions, implemented a series of changes to the immigration court system that have continued under his successor, William Barr.

The Justice Department has imposed case quotas, given more power to the director charged with overseeing the courts, reversed rulings, curtailed judges' ability to exercise discretion in some cases and moved to decertify the union of immigration judges.

Over time, those actions prompted immigration judges, some of whom were retirement eligible and had decades of experience, to leave the department despite initial plans to stay longer.

"I felt then and I feel now that this administration is doing everything in its power to completely destroy the immigration court system, the board of immigration appeal and the immigration system in general," said Ilyce Shugall, who served as an immigration judge in San Francisco from 2017 until March of this year. "And I just couldn't be a part of that."

'It started to wear on me'

Over his nearly two-year tenure as attorney general, Sessions transformed the courts by flexing his authority to overrule decisions, hire more immigration judges and set a case quota for judges.

One of Sessions' addresses to the workforce, in particular, resonated with judges. In a June 2018 speech in Washington, Sessions denounced the system, which he believed was encouraging migrants to make baseless asylum claims, and reminded judges of their role in cracking down on those claims.

"You have an obligation to decide cases efficiently and to keep our federal laws functioning effectively, fairly and consistently," Sessions said. Later that day, he issued a ruling that removed asylum protections for victims of domestic violence and gang violence.

"To be honest with you, in that meeting room, there were a number of judges that cheered and clapped when he announced it," said former immigration judge Rebecca Jamil, referring to the ruling that would follow his address. "It was grotesque to me."

Jamil, who had been based in the San Francisco immigration court, had a docket that included migrants who had fled their home countries, claiming they were victims of domestic violence. Sessions' decision took direct aim at those cases.

Another judge in attendance at Sessions' speech, Denise Slavin, recalled jaws dropping. Slavin had become a judge in 1995, serving in Florida before finishing her tenure in Baltimore in April of this year.

Sessions' address and follow-up ruling was among a series of policy changes that began to wear on judges.

"When you've been around that many administrations, you learn to adapt. You see a lot of different things. Nothing like this," said James Fujimoto, a former Chicago immigration judge who started on the bench in 1990 and also retired in April.

In particular, the administration began rolling out changes that dictated the way judges were expected to proceed with cases, thereby tightening control of the immigration courts. For example, the Justice Department said it would evaluate immigration judges on how many cases they close and how fast they hear cases.

Earlier this year, the Justice Department also issued a new rule that gives more power to the director of the Executive Office for Immigration Review . It allows the Justice Department-appointed director -- currently James McHenry -- to step in and issue a ruling if appeals are not completed within a certain time frame.

"It started to wear at me," said Jennie Giambastiani, a former Chicago immigration judge who joined the bench in 2002 and left this year. "The great number of cases coming in and the way it was expected we handle them."

Judge Ashley Tabaddor, president of the National Association of Immigration Judges, told CNN that for the majority of people leaving their roles it's a result of the "hostility and insulting working conditions."

Tabaddor noted that there's been a pattern of new judges either leaving to return to their old jobs or taking other jobs within the government.

"This is not what they signed up for," Tabaddor said, referring to policies designed to dictate how judges should handle their dockets.

Judges who have since left the department expressed similar concern over those policies. Dornell called the situation "intolerable."

Shugall recalled the challenges she had faced in trying to move forward with cases in a way she thought was appropriate. "I felt like as more and more policies were coming down, it was making it harder and harder to effectively hear cases in the way that I felt was appropriate and in compliance with the statute regulations and Constitution," Shugall said.

At an event earlier this year, McHenry rejected criticism that judges are vulnerable to pressures from the attorney general.

"Most judges that we're familiar with, and I don't think that immigration judges are any exception, when they're on the bench, they know what their role is as a judge," he said. "We've had no allegations of anyone reaching down to specific judges telling them, 'You have to rule this way; you have to rule that way.' "

JUST WATCHED Talking about immigration, across the political divide Replay More Videos ... MUST WATCH Talking about immigration, across the political divide 04:31

Justice Department hires new judges

Earlier this month, the Justice Department announced 28 new immigration judges , bringing the number of such judges to more than 465, a record high. The majority come from government backgrounds.

It's not unusual for administrations to hire people who've worked in government, but under the Trump administration, Booz Allen Hamilton, at the direction of the Justice Department's Executive Office for Immigration Review, issued a report recommending that the agency diversify the experience of immigration judges.

The direction of the nation's immigration courts is also a source of concern among immigrant advocate groups. This month, groups filed a wide-ranging lawsuit, alleging that the Trump administration has manipulated the immigration court system to serve an "anti immigrant agenda."

It remains to be seen what changes, if any, are in store for the court system, but some of those who have already left their posts as judges carry guilt for departing, concerned about who may fill their jobs.

"The biggest thing I contended with is who is going to replace me," Jamil said. "I knew I was a fair judge."