Attorney General Eric Holder says he has one regret: his department's court order for Fox News reporter James Rosen's emails labeling him a criminal "co-conspirator."

The outgoing attorney general, who recently announced his retirement, addressed the controversial episode during the "Washington Ideas Forum" on Wednesday. Asked what decision he wishes he could do over, Holder said: "I think about the subpoena to the Fox reporter, Rosen."

Holder was referring to a 2010 search warrant application seeking Rosen's emails. The Justice Department at the time was investigating who leaked information contained in a series of reports by Rosen in 2009 about North Korea's nuclear weapons program.

In the course of seeking Rosen's emails, an FBI agent submitted an affidavit claiming there was evidence that Rosen broke the law, "at the very least, either as an aider, abettor and/or co-conspirator." The affidavit went so far as to invoke the Espionage Act -- pertaining to the unauthorized gathering and transmitting of defense information.

On Wednesday, Holder said that application could have been done "differently" and "better."

"I think that I could have been a little more careful looking at the language that was contained in the filing that we made with the court -- that he was labeled as a co-conspirator," Holder said, while claiming they did that "as a result of the statute."

Rosen, who serves as chief Washington correspondent for Fox News, said in response that the attorney general's comments "scarcely address" his role in the case.

"Throughout this ordeal for my family and me, I have tried to keep my head down and continue covering -- and breaking -- the news. I consider myself blessed to have an employer in Fox News, and a boss in Roger Ailes, who have stood by me and enabled me to remain focused on what matters most to me in professional terms: first-rate journalism," Rosen said in a statement. "At some later point, I may have more to say about this entire controversy, which -- as commentators from across the ideological spectrum have noted -- does indeed raise serious concerns about the state of press freedoms under the present administration. Suffice to say for now that the attorney general's latest comments about my case, like his previous remarks, scarcely address the relevant facts of his conduct."

Though Rosen was never charged, the revelation about the affidavit -- and other details about the Justice Department's tracking of his communications and movement -- prompted outrage from media and free-press organizations. Days before the information about the affidavit was made public, Holder had testified he knew nothing of the "potential prosecution" of the press. House Republicans later issued a formal report accusing Holder of misleading Congress with "deceptive" testimony.

Though the warrant application was from 2010, the incident only became public in May 2013 -- shortly after the Justice Department took heat for obtaining two months' worth of phone records for Associated Press employees.

The cases led to a department review of how it interacts with the media.

Holder, while stopping short of an apology, cited both those incidents on Wednesday in explaining why the department adjusted its policies.

President Obama, in May 2013, also had said he was "troubled" over the impact his administration's leak probes could have on the press. The DOJ review culminated with new guidelines in July 2013 saying members of the media would "not be subject to prosecution based solely on newsgathering activities." They said the agency would exercise the right to use unspecified "tools" to gather evidence from media figures "only as a last resort."

The updated review also called for Congress to consider implementing a media shield law-legislation to protect a reporter's right to refuse to testify. Congress has yet to enact the any legislation on the matter, though several states have similar laws on the books.