President Donald Trump has vowed to take the detention center at Guantanamo Bay and fill it with “some bad dudes.” But he hasn’t yet filled the top two positions in the federal government specifically tasked with overseeing the over 700 former detainees who’ve already been released to ensure they do not become security threats.

The job of monitoring these former Guantanamo Bay detainees and coordinating their transitions to civilian life largely falls to two small special envoy offices formed by President Barack Obama: one at the Department of Defense that reviews detainees considered for release and then tracks the intelligence community’s reports on them, and one at the State Department that helps coordinate communication between detainees and their lawyers, host-country governments, U.S. embassies, and the Department of Defense.

Trump has not appointed a leader for either office, and multiple members of the approximately 10-person office at the State Department have been at least temporarily reassigned, according to current and former State Department officials.

“They are losing critical intelligence about where [former detainees] are now and how they are doing,” said Azmat Khan, a fellow at the New America Foundation who has written extensively on counterterrorism and Guantanamo.

There are only 41 detainees left at Guantanamo. Of the 714 who have been transferred from the Cuba facility or repatriated since the U.S. brought the first detainee there in 2002, most have begun quiet lives in countries like Kazakhstan, Uruguay, and Oman. But the U.S. government claims 121 of the released detainees have since engaged in terrorism, according to a January 2017 semiannual report from the Director of National Intelligence.

Life after being locked up at Guantanamo Bay can be difficult, and lawyers for released detainees told VICE News that they’re concerned the Trump administration is not going to prioritize their clients, which could ultimately create security concerns.

“Having somebody in place at the State Department is important to help with [released detainees] reintegrating,” said Wells Dixon, a staff lawyer at the Center for Constitutional Rights who has represented several released detainees. “That is important for the national security of the United States.”

A representative of the State Department’s envoy office maintained that, for now, the office has sufficient capacity to deal with released detainees even as some members had been reassigned. Until it receives updated guidance, the office is operating as it did before Jan. 20, according to current and former State Department officials.

The rate of detainees engaging in terrorism post-Guantanamo has gone down since the creation of the two envoy offices; only eight of the 182 detainees released during the Obama administration have been found engaged in the fight. “Having that position filled was extraordinarily helpful for released detainees’ lawyers and to make sure the released detainees were being held in safe areas,” Khan said. She added that congressional political pressure also made the Obama administration much more cautious than the Bush administration had been, one of the reasons the detention facility remains open.

The State Department negotiates with host countries to ensure that released detainees get access to things like employment, health care, and housing to ensure a smoother transition to civilian life. In addition, they negotiate security agreements between the U.S. and host countries to restrict travel, arrange surveillance, and ensure the sharing of intelligence information.

Lee Wolosky, U.S. special envoy for Guantanamo Closure at the State Department from 2015 to January of 2017, said that the Obama administration worked hard to ensure a smooth transition for released detainees so that they could live a life “without temptation” to go to the battlefield. “There needs to be some mechanism to monitor these people because monitoring security assurances helped us avoid problems.”

Paul Lewis, the Defense Department’s special envoy for Guantanamo Closure from 2013 to 2017, agreed. “Even if the president doesn’t want to close Gitmo, his administration still has a responsibility to monitor released detainees.”

Even when the offices were fully staffed, some released detainees were not monitored properly or did not get the help necessary to transition smoothly. Mansoor al-Dayfi, a Yemeni citizen who was detained at Guantanamo for allegedly having prior knowledge of 9/11, was sent to Serbia in July 2016 because the U.S. government was unable to negotiate an arrangement to send him back to his war-torn home country. Once in Serbia, Dayfi felt persecuted and went on a hunger strike in hopes of being sent to an Arab country. After speaking to an NPR reporter, Dayfi said that masked men forced themselves into his apartment, pinned him to the floor, and told him not to talk to anyone anymore.

Trump has yet to fill the vast majority of nearly 4,000 political positions throughout the federal government — he’s called many of them “unnecessary” — and he’s proposing a 28 percent cut at the State Department. Those facts, combined with Trump’s stated desire to grow rather than shrink Gitmo, make it appear extremely unlikely that he’ll fill the envoy positions, which literally have “Guantanamo Closure” in their titles.