(CNN) The Justice Department Inspector General has found "potential systemic issues" in how the department handles sexual harassment and misconduct allegations across its components, according to a memorandum he wrote in May to Deputy Attorney General Rod Rosenstein after reviewing the handling of cases that primarily occurred in years past.

The inspector general's office said it published summaries of 19 "substantiated allegations of sexual harassment and misconduct" that occurred from fiscal year 2012 through fiscal year 2016. In the memorandum, the watchdog warned that "without strong action from the department to ensure that DOJ employees meet the highest standards of conduct and accountability, the systemic issues we identified in our work may continue."

The office released other reports which largely detail concerns about handling of allegations but don't report large numbers of instances of sexual misconduct across Justice Department components.

For example, a May 2017 report posted to the inspector general's website finds "few reported allegations of sexual harassment and misconduct in the Civil Division from FY 2011 through the first two quarters of FY 2016," though the office "identified significant weaknesses in the Civil Division's tracking, reporting and investigating of the 11 sexual harassment and misconduct allegations that we reviewed."

And a 2015 report on the handling of sexual harassment and misconduct allegations by the Justice Department's law enforcement agencies (including the Drug Enforcement Agency, the FBI and the Bureau of Alcohol, Tobacco, Firearms and Explosives) mainly focuses on the way these agencies report misconduct and whether security is compromised as a result of this conduct, rather than the number of sexual harassment allegations.

Read More