David Petraeus speaks onstage during a fireside chat at the Team Rubicon Salute to Service Awards at Skirball Cultural Center on Nov. 7, 2014, in Los Angeles.

Broadwell is Petraeus' former mistress, as well as a former Army Reserve officer who wrote his biography. Petraeus has previously stated he never passed classified documents to Broadwell.

Prosecutors with the FBI and the Department of Justice recommended the charges against Petraeus, a retired four-star general who led U.S. forces in the Afghan and Iraq wars. The recommendation comes as a result of an investigation into whether he gave Paula Broadwell access to his email and other classified documents. The New York Times was the first to report the recommendation.

Attorney General Eric Holder will now have to decide if he will pursue an indictment. If convicted, Petraeus could end up in prison.

Holder said Sunday on CBS' Face the Nation that the decision of whether to charge Petraeus would be made at the "highest levels" of the Justice Department.

"The determination in any case is made at the time that all of the evidence has been acquired, all of the evidence has been reviewed, when it has been gone over with people who are the subjects of the investigation and with their lawyers and so at the appropriate time, the proper people within the Justice Department will make determinations as to what, if any, action should occur," Holder said.

Holder did not comment on many details of the case, saying he could not do so because it is an ongoing investigation.

The recommendation to charge Petraeus comes after a years-long investigation. Following a high-profile military career, Petraeus was appointed to head up the CIA in 2011 but resigned in 2012 when news broke of his affair with Broadwell. The affair was exposed during an investigation into a security breach of Petraeus' email.

Jesseyln Radack, the director of national security and human rights at the Government Accountability Project, told BuzzFeed News she thinks it's "more than fair" that Petraeus face charges.

"This is a justified leak case, especially in light of Petraeus' hypocrisy in condemning former CIA official John Kiriakou for 'leaking' the name of a torturer," she said. "I'm not sure why Petraeus leaking classified information to his mistress is any worse than Leon Panetta leaking classified information for the production of a Hollywood movie — both should be prosecuted."

Radack, a former whistleblower who has publicly defended other famous leakers including Edward Snowden and Thomas Drake, described a double standard in place when trying leak cases.

"While I'm uncomfortable with the use of the Espionage Act to punish unauthorized disclosures, the fact of the matter is that the only national security and Intel officials prosecuted for leaking classified information have been whistleblowers trying to expose fraud, waste, abuse and illegality."

Trevor Trimm, Executive Director of the Freedom of the Press Foundation, said prosecuting leaks is never necessary, arguing that administrative repercussions worked as an adequate policing mechanism for decades.

"I don't think prosecuting leaks is ever really necessary, unless the person is 'leaking' to a foreign government for personal gain. There are plenty of motivations for government officials to never leak to the media, including the loss of pay, loss of their job, the loss of their security clearance and their career. This is primarily how the government policed leaks for decades," Trimm said Friday evening.

But Trimm argued that if the government is committed to pursuing leak cases, "they should do so fairly and across the board — no matter one's rank in the military or place in the government."

"We know, for example, the last two CIA directors besides Petraeus, Leon Panetta and John Brennan, have leaked classified information to the media, but they faced no punishment. Yet lower-level CIA officers like John Kiriakou and Jeffery Sterling have been charged with felonies and faced jail time for extremely similar alleged acts, in cases many consider whistleblowing," he added.

Jack Goldsmith, a Harvard law professor and former Department of Justice and Department of Defense attorney during the Bush administration, argued Petraeus' case is unique in that high level officials are rarely prosecuted for leaking information.

"In the past the government has gone after lower-level officials for leaking or mishandling classified information to the press, but it has never prosecuted senior officials, who often reveal classified information, either on purpose or inadvertently. Petraeus might be an exception because he leaked the information to his mistress rather than the press," Goldsmith said, noting that despite that distinction "the law is the same regardless of the recipient of the leak. But there is at least a weak moral case for sometimes leaking classified information to the press (for purposes of government accountability), and no such case in Petraeus' situation."