The Pentagon is considering retroactively demoting retired Gen. David Petraeus after he admitted to giving classified information to his biographer and mistress while he was still in uniform, three people with knowledge of the matter told The Daily Beast.

The decision now rests with Secretary of Defense Ash Carter, who is said to be willing to consider overruling an earlier recommendation by the Army that Petraeus not have his rank reduced. Such a demotion could cost the storied general hundreds of thousands of dollars—and deal an additional blow to his once-pristine reputation.

“The secretary is considering going in a different direction” from the Army, a defense official told The Daily Beast, because he wants to be consistent in his treatment of senior officers who engage in misconduct and to send a message that even men of Petraeus’s fame and esteemed reputation are not immune to punishment.

Pentagon spokesperson Peter Cook told The Daily Beast that Carter had requested the information ex-Army Secretary John McHugh had when he made his recommendation on the matter, before reaching a final decision. McHugh had recommended taking no action against Petraeus.

“The Department of the Army is still in the process of providing the secretary with information relevant to former‎ Secretary McHugh’s recommendation,” Cook told The Daily Beast. “Once the secretary‎ has an opportunity to consider this information, he will make his decision about next steps, if any, in this matter.”

Carter could also recommend other actions that don’t result in Petraeus losing his fourth star. Or the defense secretary could simply allow the Army’s previous recommendations to stand.

Petraeus, arguably the most well-known and revered military officer of his generation, retired from the Army in 2011 with the rank of a four-star general, the highest rank an Army officer can achieve. If Carter decides to strip Petraeus of his fourth star, he could be demoted to the last rank at which he “satisfactorily” served, according to military regulations.

Reducing Petraeus’s rank, most likely to lieutenant general, could mean he’d have to pay back the difference in pension payments and other benefits that he received as a retired four-star general. That would amount to hundreds of thousands of dollars over his retirement. According to Pentagon figures, a four-star general with roughly the same years of experience as Petraeus was entitled to receive a yearly pension of nearly $220,000. A three-star officer would receive about $170,000.

Petraeus didn’t respond to a request for comment.

But the financial pain to Petraeus isn’t likely to be severe. He has confided to friends and acquaintances that he’s making a hefty sum from his job at a private equity firm and through speaking fees.

The demotion in rank would be a bigger, lasting blow, and take from Petraeus the rare achievement he’d set his eyes on many years ago.

At any given time, there are only 12 four-star generals in the Army, the largest of the services. By the time he was a colonel, in the mid-1990s, many thought Petraeus was destined to be one of them.

The U.S. military has, on several occasions, demoted generals, increasingly for improper personal contact and not for poor battlefield decisions. But rarely does it demote four-star generals, in part because there are so few of them. It’s also more common to reduce the rank of more junior officers than of top generals.

If Petraeus were demoted, it would mark another spectacular fall. Petraeus stepped down as director of the Central Intelligence Agency in 2012 after his affair with Paula Broadwell, a writer and current Army reservist, was revealed. At the time, Petraeus had been frequently mentioned as a possible Republican presidential candidate in 2016.

Petraeus pleaded guilty last year to giving Broadwell eight notebooks that he compiled while serving as commander of U.S. forces in Afghanistan and that he knew contained classified information. The notebooks held some of the most sensitive kinds of military and intelligence secrets, including the identities of covert officers, intelligence capabilities, quotes from high-level meetings of the National Security Council, and notes about Petraeus’s discussions with President Obama.

After leaving Afghanistan, Petraeus took the books back to his home in Virginia and gave them to Broadwell just three days before he retired from the Army. She later returned them. No classified information appeared in her biography, All In: The Education of General David Petraeus, officials have said.

Petraeus could have faced felony charges, including for lying to FBI investigators, but was allowed to plead guilty last year to a misdemeanor count of unauthorized handling classified information. He avoided a prison sentence but received two years probation and a $100,000 fine.

But that was not the end of the matter. Last year, the FBI gave Army investigators information that the bureau had come across as it was closing up its own investigation of Petraeus, the defense official and one former U.S. official told The Daily Beast.

The information, the FBI believed, might be of interest to the Army, the defense official said. The Army investigated and decided “there was nothing new here that should change his retirement” and “recommended that there be no change” to his four-star rank, the official said. Last month, it went to the secretary of defense for final approval.

Army personnel regulations say that an officer doesn’t automatically retire with the highest rank he or she achieved while in uniform (PDF). And even though Petraeus had already been officially retired, through a process known as grade determination the Army can retroactively reopen his case and consider whether to demote him “[i]f substantial new evidence discovered contemporaneously with or within a short time following separation could result in a lower grade determination.”

The regulations also state that if “an officer’s misconduct while still on active duty is documented,” including by “conviction after retirement,” a new grade determination may be completed. Petraeus hadn’t yet retired when he gave Broadwell the classified information.

The Army received the information from the FBI that prompted this new review more than four years after Petraeus had retired. The Defense Department was also running its own investigation into Petraeus’s relationship with Broadwell and what classified information he gave her at the same time the FBI and federal prosecutors were pursuing their case. That may explain why the Army decided it had seen nothing new in the information it received last year from the FBI and decided not to recommend a demotion.

But Carter is said to be concerned that because he has recommended other generals be reduced in rank for actions not becoming an officer, he’ll be seen as inconsistent if he doesn’t do the same for Petraeus. The decision is as much about timing and politics as it is Petraeus’s own transgressions.

“This is about Ash Carter, not David Petraeus,” the defense official said.

Last November, Carter removed his senior military aide, Lt. Gen. Ron Lewis, for personal misconduct, and referred the matter to the Pentagon’s inspector general for investigation. Lewis was demoted a rank, to a major general.

Lewis was a longtime and influential aide to the secretary, and his removal and punishment signaled Carter’s commitment to maintaining upstanding behavior among the military’s generals. The exact nature of Lewis’s misconduct has not been announced, but military officials have suggested he was involved in an improper personal relationship.

While few are familiar with Petraeus’s potential demotion, those who are aware of it said they were surprised that he could be punished years after the scandal was presumably put behind him and after he pleaded guilty to mishandling classified information. No general in recent history has been demoted years after scandal swirled around him or her.

Those who know and have worked with Petraeus describe him as a man of extraordinary capabilities and ambition. He received his fourth star in 2007 and then served in several prestigious and demanding assignments, including commander of U.S. Central Command, the commanding general of all ground forces in Iraq, and later as commander of ground forces in Afghanistan.

Petraeus’s unorthodox thinking and willingness to buck conventional strategy was seen as key to the U.S. victory over insurgents and jihadists in Iraq during the so-called troop surge of 2007 and 2008. His reputation was so esteemed that there was talk of giving him a fifth star—a largely symbolic gesture that was highly unlikely—or renaming the road to Petraeus’s alma mater, the U.S. Military Academy, after him.

Should Carter choose to knock Petraeus down to the rank of a three-star general, he will have a chance to appeal his case to the secretary, but Congress doesn’t have to be informed of the decision, the official said.

There is no deadline on Carter to make a decision.

The last commander to lose rank for professional misconduct was Brig. Gen. Janis Karpinski, who was demoted to colonel in 2005 for the Abu Ghraib prison scandal in Iraq. The last four-star general to be demoted was Gen. William Ward, who retired as a three-star in 2012 amid allegations he misspent government money on himself and his family.