The attackers behind the nation-state espionage tool known as Flame accidentally left behind tantalizing clues that provide information about their identities and that suggest the attack began earlier and was more widespread than previously believed.

Researchers have also uncovered evidence that the attackers may have produced at least three other pieces of malware or variants of Flame that are still undiscovered.

The information comes from clues, including four programmers' nicknames, that the attackers inadvertently left behind on two command-and-control servers they used to communicate with infected machines and steal gigabytes of data from them. The new details about the operation were left behind despite obvious efforts the attackers made to wipe the servers of forensic evidence, according to reports released Monday by researchers from Symantec in the U.S. and from Kaspersky Lab in Russia.

Flame, also known as Flamer, is a[highly sophisticated espionage tool discovered earlier this year](https://contextly.com/redirect/?id=PV7X8ia750&click=inbody "Meet "Flame", The Massive Spy Malware Infiltrating Iranian Computers") that targeted machines primarily in Iran and other parts of the Middle East. It's believed to have been created by the United States and Israel, who are also believed to be behind the groundbreaking Stuxnet worm that aimed to cripple centrifuges used in Iran's nuclear program.

The new clues show that work on parts of the Flame operation began as early as December 2006, nearly six years before Flame was discovered, and that more than 10,000 machines are believed to have been infected with the malware.

Although the 2006 date refers to the development of code used in the command-and-control servers and doesn't necessarily mean the malware itself was in the wild all of this time, Vikram Thakur, a researcher with Symantec Security Response, says the details are still troubling.

"For us to know that a malware campaign lasted this long and was flying under the radar for everyone in the community, it's a little concerning," he says. "It's a very targeted attack, but it's a very large-scale targeted attack."

The two security firms conducted the research in partnership with BUND-CERT, the federal computer emergency response team in Germany, and ITU-IMPACT, the cybersecurity arm of the United Nation's International Telecommunications Union.

Although the attackers clearly were part of a sophisticated nation-state operation, they made a number of mistakes that resulted in traces of their activity being left behind.

According to data gleaned from the two servers the researchers examined:

At least four programmers developed code for the servers and left their nicknames in the source code.

One of the servers communicated with more than 5,000 victim machines during just a one-week period last May, suggesting the total victims exceed 10,000.

The infections didn't occur at once, but focused on different groups of targets in various countries at different times; one server focused primarily on targets in Iran and Sudan.

The attackers stole massive amounts of data - at least 5.5 gigabytes of stolen data inadvertently left behind on one of the servers was collected in one week.

The four pieces of malware used different custom protocols to communicate with the servers.

The attackers used a number of means to secure their operation as well as the data they stole - although they left behind gigabytes of purloined data, it was encrypted using a public key stored in a database on the servers and an unknown private key, preventing the researchers and anyone else without the private key from reading it.

The attackers, perhaps suspecting that their operation was about to be uncovered last May, attempted a cleanup operation to wipe the Flame malware from infected machines.

Flame was discovered by Kaspersky and publicly disclosed on May 28. Kaspersky said at the time that the malware had targeted systems in Iran, Lebanon, Syria, Sudan, Israel and the Palestinian Territories, as well as other countries in the Middle East and North Africa. Kaspersky estimated at the time that the malware had infected about 1,000 machines.

The malware is highly modular and can spread via infected USB sticks or a sophisticated exploit and man-in-the-middle attack that hijacks the Windows Update mechanism to deliver the malware to new victims as if it were legitimate code signed by Microsoft.

Once on machines, Flame can steal files and record keystrokes, as well as turn on the internal microphone of a machine to record conversations conducted over Skype or in the vicinity of the infected computer.

Previous research on Flame conducted by Kaspersky found that Flame had been operating in the wild undetected since at least March 2010 and that it might have been developed in 2007.

But the new evidence indicates that development of code for the command-and-control servers - servers designed to communicate with machines infected with Flame - began at least as early as December 2006. It was created by at least four programmers, who left their nicknames in the source code.

The Flame operation used numerous servers for its command-and-control activities, but the researchers were able to examine only two of them.

The first server was set up on March 25 and operated until April 2, during which it communicated with infected machines from 5,377 unique IP addresses from more than a dozen countries. Of these, 3,702 IP addresses were in Iran. The country with the second largest number was Sudan, with 1,280 hits. The remaining countries each had fewer than 100 infections.

The researchers were able to uncover the information, because the attackers had made a simple mistake.

"The attackers played with the server settings and managed to lock themselves out of it," says Costin Raiu, senior security researcher for Kaspersky.

Left behind on the locked server were the http server logs, showing all of the connections that came in from infected machines. Researchers also found about 5.7 gigabytes of data stored in a compressed and encrypted file, which the attackers had stolen from victims' machines.

"If their collection of 6 gigabytes of data in a span of ten days in March is indicative of how prevalent their campaign was for multiple years in the past, they probably have terabytes of information that they collected from thousands and thousands of people across the globe," says Symantec's Thakur.

The second server was set up on May 18, 2012, after Kaspersky had discovered Flame, but before the company had publicly disclosed its existence. The server was set up specifically to deliver a kill module, called "browse32," to any infected machine that connected to it, in order to delete any trace of Flame on the machine. It may have been set up after the attackers realized they'd been caught.

Raiu says the attackers may have realized Flame had been discovered after a honeypot machine belonging to Kaspersky reached out to the attackers' server.

"Around the 12th of May, we connected a virtual machine infected by Flame to the internet, and the virtual machine connected to the [attackers'] command-and-control servers," he says.

Five hours after the server with the kill module was set up on the 18th, it received its first hit from a machine infected with Flame. The server remained in operation only about a week, communicating with a few hundred infected machines, says Symantec.

Four Coders

The four programmers who developed code for the servers and left their nicknames in the source code, were part of a sophisticated operation that likely included multiple teams - a coordinating team tasked with choosing the specific targets to attack and examining all of the stolen data that came in from them; a team responsible for writing the malware and command modules; and a team of operators for setting up and maintaining the command-and-control servers, sending command modules out to infected machines and managing the stolen data once it came in.

Symantec and Kaspersky have redacted the nicknames of the four coders and identified them only by the first initial of their nicknames - D, H, O, and R.

Thakur said he's never seen nicknames left in malware except in low-level malware that's unsophisticated. But the coders in the Flame operation were clearly a higher caliber than this.

"Maybe they just never expected their server to reach the wrong hands," he says. "But considering that Flamer has links to Stuxnet and DuQu, we would have expected not to see these names. But, at the end of the day, they're human."

Of the four coders, "D" and "H" were the more significant players, since they handled interactions with infected computers, and were responsible for creating two of the four protocols the servers used to communicate with malware on infected machines.

But "H" was the most experienced of the group, responsible for some of the encryption used in the operation. Raiu calls him a "master of encryption," and notes that he was responsible for implementing the encryption of data stolen from victim machines.

"He coded some very smart patches and implemented complex logic," Kaspersky writes in its report. "We think [he] was most likely a team lead."

"O" and "R" worked on a MySQL database used in the operation ,as well as the development of cleanup files that were used to wipe data from the servers.

Based on timestamps of activity on the servers, Symantec thinks the coders were based in Europe, the Middle East or Africa.

Command-and-Control Server Setup

The command-and-control servers hosted a custom web application the programmers developed – called NewsforYou – to communicate with infected machines. Through the application's control panel, the attackers could send new modules to infected clients and retrieve stolen data.

The password for the control panel was stored as an MD5 hash, which the researchers were unable to crack.

The control panel was disguised to resemble a content management system that a news organization or a blog publisher might use, so that any outsider who gained access to the panel wouldn't suspect its real purpose.

While command-and-control servers used by cybercriminal groups generally have showy control panels with the words "bot" or "botnet" clearly labeled on them, making their malicious purpose immediately apparent, the control panel used in the Flame operation was bare-bones in design and used euphemistic terms to conceal its real purpose.

For example, file directories for storing malicious modules to send to infected machines were named "News" and "Ads." The "News" directory held modules meant to be sent to every machine, while "Ads" held modules intended only for select machines. Data purloined from infected machines was stored in a directory called "Entries."

The attackers also used a number of security features to prevent an unauthorized party who gained control of the server from sending arbitrary commands to infected machines or reading stolen data.

Often, criminal control panels use a simple point-and-click menu of options for the attackers to send commands to infected machine. But the Flame operation required the attackers to create a command module, place it in a file with a specially formatted name and upload it to the server. The server would then parse the contents of the file, and place the module in the appropriate location, from where it could be pushed out to infected machines.

Symantec said this complicated style, which prevented server operators from knowing what was in the modules they were sending to victims, had the hallmarks of "military and/or intelligence operations," rather than criminal operations.

Similarly, data stolen from victim machines could only be decrypted with a private key that was not stored on the server, likely so that server operators could not read the data.

The researchers found evidence that the command-and-control servers were set up to communicate with at least four pieces of malware.

The attackers refer to them, in the order they were created, as SP, SPE, FL, IP. "FL" is known to refer to Flame, a name that Kaspersky gave the malware back in May, based on the name of one of the main modules in it (Symantec refers to the same malware Flamer). IP is the newest malware.

Aside from Flame, none of the other three pieces of malware have been discovered yet, as far as the researchers know. But according to Raiu, they know that SPE is in the wild because a handful of machines infected with it contacted a sinkhole that Kaspersky set up in June to communicate with machines infected with Flame. They were surprised when malware that was not Flame contacted the sinkhole as soon as it went online. They only recently realized it was SPE. The SPE infections came in from Lebanon and Iran.

Each of the four pieces of malware uses one of three protocols to communicate with the command-and-control server - Old Protocol, Old E Protocol, or SignUp Protocol. A fourth protocol, called Red Protocol, was being developed by the operators but had not been completed yet. Presumably, the attackers planned to use this protocol with yet a fifth piece of malware.

Clean-Up and Coding Gaffes

The attackers took a number of steps to delete evidence of their activity on the servers, but made a number of mistakes that left telling clues behind.

They used a script called LogWiper.sh to disable certain logging services and delete any logs already created by those services. At the end of the script, there was even an instruction to delete the LogWiper script itself. But the researchers discovered the script contained an error that prevented this from occurring. The script indicated that a file named logging.sh should be deleted, not LogWiper.sh, leaving the script behind on the server for the researchers to find.

They also had a script designed to delete temporary files at a regularly scheduled time each day, but the script had a typo in the file path, so that it couldn't find the tool it needed to erase the files.