Of all the questions swirling around the widening farce of General David Petraeus's downfall, one stands out: who is the rogue FBI agent, now dubbed Agent Shirtless, and why was he so determined to bring down the director of the CIA?

Since Petraeus's sudden resignation caught Barack Obama by surprise last week, over an affair with his biographer that the FBI knew about for months, the mood in Washington has swung from regret at the loss of a respected soldier and intelligence chief to disbelief and mirth amid almost daily revelations over the scandal, including the irony of the CIA director being brought down by covert surveillance. Astonished members of Congress have also raised questions about the timing of the resignation.

The latest twist has dragged Petraeus's successor as US and Isaf military commander in Afghanistan, General John Allen, on to the stage after the Pentagon said he is being investigated for "potentially inappropriate communications" with a woman, Jill Kelley, whose complaint about harassing emails from Petraeus's mistress, Paula Broadwell, set in motion the investigation that forced the CIA director from office. Those communications involved more than 20,000 pages of emails, some of them "flirtatious", and other documents over two years.

The White House said Barack Obama retains his "faith in General Allen", suggesting that, for the the moment at least, the administration does not feel there is enough there to warrant his resignation. But Allen's appointment to become the next supreme allied commander of Nato is on hold for the investigation.

Some Pentagon officials strongly deny Allen had an affair with Kelley but the sheer volume of emails has raised questions about just what kind of relationship the pair had.

The denials did not stop a Taliban spokesman, Qari Yousuf Ahmadi, from attempting to exploit the US army's evident embarrassment.

"I can say that this shows how shameless all the American troops are, starting from the top commander to the soldiers. Having unlawful relations with a woman is very normal in America," he said.

But, for now, the principal focus in Washington is on Petraeus and the part played by the FBI in his resignation, including why it was the president knew nothing about the investigation until hours before the general quit.

The White House spokesman, Jay Carney, said the president was surprised to hear about Petraeus's affair and that it was for the FBI to explain why Obama was not informed about the months-long probe of the CIA director until last Wednesday.

"The fact of the matter is there are processes in place to handle these kind of things. They are playing out appropriately," he said.

The swirl of revelations has left members of Congress dumbstruck and scrambling to understand just what is going on as intelligence committees held closed hearings on Tuesday.

"Disturbing is the word that has come to my mind since all of this has come to light," said congresswoman Debbie Wasserman Schultz, chair of the Democratic National Committee. "Really, I think it goes without saying if you're the director of the CIA, if you are a four-star general in the United States army, that you have to hold yourself to a higher standard. And you can't put yourself in a compromising position."

Petraeus's leadership of the CIA began to unravel when Broadwell, his former mistress and writer of an aptly named biography, All In: The Education of General David Petraeus, shot off six anonymous harassing emails to Kelley.

Kelley, described by the military as a civilian volunteer who runs social events for service personnel at a Tampa, Florida, military base, was friends with the Petraeus family. They regularly appeared at the same social and charity events, as did Allen who was serving with Petraeus at US central command.

The Washington Post reported that the Petraeuses also attended "intimate family gatherings" at the Kelleys. The CIA director was close enough to Kelley to write a letter in support of her "psychologically unstable" twin sister who was involved in a child-custody fight.

A former member of the general's military staff told the Washington Post that Kelley was a "self-appointed" go-between for US central command officers in Florida and Middle East government officials. Kelley and her siblings were born in Lebanon and brought to the US as small children.

Jill Kelley at home on Tuesday. Photograph: Chris O'Meara/AP

The same official told the paper that "Kelley's presence was often a bit puzzling to the Petraeus staff but the officer said there was never any indication that her relationship with the general was anything more than social".

Broadwell appears to have feared there was more to it than that. One of her messages accused Kelley of touching Petraeus under a table in an intimate manner and warned her to stay away from the general.

Kelley turned to an old friend in the FBI whose name has yet to be revealed but which will surely come out. It wasn't clear the emails amounted to the kind of threatening behaviour the agency would investigate but the agent and Kelley had a history. He was so taken with her that some months earlier he had taken a picture of himself topless and sent it to Kelley, a married woman.

Agent Shirtless, as he swiftly became known, pressed for an investigation even though cyber crimes were not his area of authority. The FBI Tampa office began a probe but Kelley's agent friend kept up the pressure to dig deeper, to the point where he was told to back off, according to the Wall Street Journal.

The agency apparently trod carefully, mindful of the FBI's long history of effectively acting as a secret police with snooping into the private lives of politicians and activists, and blackmail and corruption, under the four decade long leadership of J Edgar Hoover until the 1970s.

But the FBI swiftly discovered Broadwell, who is also married with children, was responsible for the harassing emails and from reading other messages in her account concluded she was having an affair with Petraeus. The pair avoided leaving a trail of emails by sharing the passwords to their accounts, and writing messages left in the draft folder for the other to read but which were never sent.

The FBI interviewed Broadwell and discovered classified documents on her computer but were unable to prove they came from Petraeus. The agency concluded no crime had been committed and was apparently content to let the matter drop. Agent Shirtless was not.

The FBI agent was himself under investigation over his conduct and warned to stay away from the case altogether after the topless picture incident was discovered. So he turned rogue and went elsewhere.

The agent's motives are not entirely clear, but he was described by officials as having a particular "world view" which appears to have been hostile to Obama. He apparently saw an opportunity to embarrass the president shortly before the election and approached a Republican member of Congress, David Reichert, who passed the information to the Republican majority leader, Eric Cantor. Cantor spoke to the agent and then contacted the FBI director, Robert Mueller.

"I was contacted by an FBI employee concerned that sensitive, classified information may have been compromised and made certain director Mueller was aware of these serious allegations and the potential risk to our national security," said Cantor.

That appears to have forced the FBI's hand. Mueller kicked the matter upstairs to the director of national intelligence, James Clapper, on the grounds that Petraeus's affair could open him to blackmail, as unlikely as that seemed, and was therefore a national security matter. Clapper called the CIA director and told him he had to resign.

But that was not until two days after the presidential election.

Congressman Peter King, chairman of the House of Representatives homeland security committee, has demanded to know why Obama was not notified about the investigation earlier – and whether that effectively amounted to a coverup to save the president from embarrassment before the vote.

"It's almost unprecedented for the FBI to investigate the director of the CIA. If they were going to do that they should immediately have gone to the attorney general and also to the president of the United States because David Petraeus was a key part of the president's foreign policy team," he told CNN.

King called the FBI "derelict in its duty".

No Washington scandal is complete without a conspiracy theory. Broadwell delivered one on a plate last month when she told students in Denver that the death of the US ambassador to Libya, Chris Stevens, in an attack on the American consulate in Benghazi was linked to the CIA holding prisoners at a nearby annex.

"I don't know if a lot of you have heard this, but the CIA annex had actually taken a couple of Libyan militia members prisoner and they think that the attack on the consulate was an effort to get these prisoners back," Broadwell said.

Petraeus had been due to testify to Congress this week over the security failings that led to the death of Stevens and three other Americans, including two CIA security men. Once he resigned, the agency said it would send another official to give evidence.

The CIA has since denied Broadwell's claim but it led to speculation that Petraeus may have been the source even though the pair was no longer romantically involved by then.

King has questioned whether Petraeus's resignation was intended to undermine the congressional probe into the Benghazi killings which is likely to be highly embarrassing to the White House.

The chairwoman of the Senate intelligence committee, Dianne Feinstein, said she will call Petraeus to testify in any case.

Within hours of the Pentagon's investigation of Allen being announced, another army general was demoted over extravagant spending and unauthorised expenses. General William "Kip" Ward was tripped of one of his four stars for misuse of military funds when he was head of the US's Africa command. He has also been ordered to repay $82,000.

• This article was amended on 14 November 2012. The original caption referred to General David, rather than John, Allen. This has been corrected.