(CNN) The two prison staff members who'd been guarding the unit where Jeffrey Epstein died by apparent suicide failed to check on him that night for about three hours, The New York Times reported Tuesday , citing several law enforcement and prison officials with knowledge of the investigation.

Officials have cautioned to CNN that they don't know what the staff members were doing during that time and still are trying to pin that down.

Epstein, 66, was in the special housing unit of the federal Metropolitan Correctional Center in New York, in a cell by himself, when he was found dead early Saturday.

Guards are supposed to check on detainees in the special housing unit every 30 minutes, a source with knowledge of Epstein's time at the prison previously told CNN

Epstein had been jailed since early last month, awaiting trial on federal charges accusing him of operating a sex trafficking ring from 2002 to 2005 at his Manhattan mansion and his Palm Beach estate in which he paid girls as young as 14 for sex. He pleaded not guilty to the charges.

One staffer was not currently a regular guard; both were on overtime

Of the two officers who would have had responsibility to monitor Epstein, one was not currently a detention guard but was temporarily reassigned to that post , according to a person briefed on the matter. The guard, a man not identified by officials, had previously been trained as a corrections officer but had moved to another position.

Rules at the Federal Bureau of Prisons allow for people who work in other prison jobs, such as teachers and cooks, to be trained to fill in at posts usually manned by regular guards.

The second staff member on Epstein duty, a woman, was a fully trained guard, according to the person briefed on the matter.

Both guards were working overtime shifts , but it's unclear whether that was mandatory. One person familiar with the matter said both employees had volunteered. Union officials say the overtime was mandatory.

The FBI, one of the agencies investigating Epstein's suicide, is looking into what the guards were doing during their shift. The two guards aren't talking to investigators and have hired lawyers, people briefed on the matter have told CNN.

Most of the staff on duty early Saturday were on overtime, according to numbers provided by a Department of Justice official. From midnight until 8 a.m. ET, 10 of the 18 people there were working extra hours. One other person was scheduled to work but didn't.

On the shift from 4 p.m. Friday until midnight, six of 20 people present were on overtime and one person scheduled to work was absent.

Epstein was no longer on suicide watch

Epstein had been placed on suicide watch last month after an incident in the jail. But after he was given daily psychological assessments, he was taken off suicide watch in late July , a person briefed on the matter said, and went back to the facility's special housing unit, which is a section more restricted than general population.

He initially had a cellmate in the special housing unit, but that person was removed for unknown reasons, the source with knowledge of Epstein's time at the prison told CNN

It's protocol for detainees coming off suicide watch not to be placed alone in a cell, according to the source.

The FBI and Justice Department's inspector general are investigating the earlier incident as part of the broader probe into Epstein's death, it was revealed Wednesday in letters between the judge overseeing the Epstein case and the former warden of the jail where he died.

In the letters dated August 12, US District Court Judge Richard Berman wrote to the warden to inquire about the July 23 incident, saying "to my knowledge, it has never been definitively explained what the BOP (Bureau of Prisons) concluded about that incident."

The warden at the time, Lamine N'Diaye, replied that an internal investigation into the incident had been completed, and that the matter would be reinvestigated as part of the investigations into Epstein's death. N'Diaye said he couldn't divulge any information about the original investigation.

Prison warden has been temporarily reassigned

The warden of the MCC is being moved temporarily as the FBI and the Justice Department's inspector general investigate the circumstances of Epstein's death, the Justice Department said Tuesday.

The move, directed by US Attorney General William Barr, came one day after Barr said there were " serious irregularities " at the New York prison and that he was "appalled" and "angry" to learn of the facility's "failure to adequately secure this prisoner."