The Secret Service and FBI are investigating an increasing number of attacks on US retailers' data, including the massive breach of Target last year that affected more than 40 million debit and credit card accounts. Investigators won't talk about the Target probe. Instead, the Secret Service pointed to O'Neill's investigation that began in 2010 as an example of how they go about solving such crimes. The chase for the hackers took three years. It uncovered what US federal prosecutors described in court records as a "massive, international computer hacking and credit card data theft scheme". 800 stores The conspirators hit more than 800 US stores from 2009 to 2011, stealing data from in excess of 150,000 credit card accounts and inflicting losses to financial institutions conservatively tallied at $US12.5 million, according to interviews with the agent, his supervisors and US Justice Department prosecutors, as well as a review of court filings. O'Neill's green eyes and sly grin mask an intensity for the hunt that relied on a mix of high-tech sleuthing and traditional police work, with some creativity sprinkled in: the agent even went undercover online as an "attractive, independently wealthy waitress".

"These hackers are sophisticated," said US Secret Service Agent Ed Lowery, who is in charge of the agency's criminal division. "The type of individual we are talking about – the highest-level cybercriminal – they don't leave bread crumbs." The son of a former Secret Service agent who investigated counterfeiting rings in Philadelphia, O'Neill joined the service in 1998 after a post-college stint at ESPN. He specialised in cybercrimes, except for four years on Vice-President Dick Cheney's security detail. 'Subway case' They never expected to get caught. Secret Service Agent Matt O'Neill In the agency's four-agent Manchester, New Hampshire, office, he juggled five or six cases at a time. None were as big as what became known as the "Subway case".

It began on a February afternoon in 2010 with calls from banks. American Express and Citibank reported fraudulent activity on accounts that had one thing in common – purchases made at a Subway sandwich shop in Plaistow, New Hampshire, a town of 7609 people about 40 kilometres south-east of Manchester. American Express reported that 36 compromised credit cards had been used at the Plaistow Subway; Citibank said it had suffered $US80,000 in losses tied to cards swiped at the store. Within days, O'Neill and a New Hampshire state trooper were inspecting the store's computer. It was clear it had been hacked through the internet, and the attacker had planted a "key logger" program onto its hard drive. Acting like a vacuum cleaner, the program sucked up the data from credit and debit cards swiped through the store's magnetic reader. The investigators determined the stolen data was only stored briefly on the computer before being uploaded to a website, ftp.tushtime.info. Romanian beetle A password embedded in the software code – Carabus05 – provided a clue to its source. When O'Neill Googled the word, he discovered it was Romanian for beetle. Russia, Romania and other eastern European countries are hotbeds for hackers.

"The important thing in these cases isn't so much: How did they get in?" O'Neill said. "It's: Where did the data go? It's the destination that matters." Representatives of Subway reported multiple hacks of other stores and the stolen data was flowing to several "dump sites" such as ftp.tushtime.info. Armed with search warrants, O'Neill saw the stolen data was flowing from the dump sites to computer servers scattered across the US. Agents tracked down those servers – one belonged to a law office, another a dentist – but none were still being used by the hackers. Then, in July of 2010, O'Neill got lucky. One of those computer servers belonged to New Harrisburg Truck and Body in Mechanicsburg, Pennsylvania. With permission from the shop's innocent owner, agents in August 2010 placed a "sniffer" on his computer. Truck shop

The sniffer revealed that the hackers were using the truck shop's computer to store and use their "tools" – malicious software that scanned the internet for vulnerable computers, allowed them to break into those computers, steal the data, upload it to a dump site, download it to the truck shop's computer and then zap it around the globe. The hackers were careful. They masked their identities by using anonymous email and chat accounts. They hid their location by routing through other servers in Europe. Even so, O'Neill suspected they were in Romania. They chatted in emails in the language, and the agent managed to track some of their computer activity back to the country. It was now late 2010 and O'Neill was getting concerned that the hackers would figure out he was monitoring them. The Secret Service was caught in a catch-22: agents wanted to keep watching in secret to find out the hackers' identities yet had an obligation to alert customers, retailers and financial institutions that the accounts had been hacked. "Their clients were emailing them and saying, 'Why are you cheating me? These are bum cards,'" O'Neill said, of the messages he secretly read.

Solid lead Finally, in late October, agents picked up a solid lead: in an online chat, a hacker mentioned that his computer had been seized and his house raided by Romanian police investigating his cyber activities. O'Neill called his Romanian counterparts and provided them with the information. In less than a day, they gave O'Neill the hacker's identity: Adrian Tiberiu Oprea, a 26-year-old who had studied computer science and lived in the Black Sea port city of Constanta. Romanian authorities told O'Neill they were investigating Oprea for hacking retailers in eastern Europe. From the hackers' emails and social media posts, O'Neill found the identity of one Oprea's customers: a Romanian living in France named Cezar Butu, 27. A third member of the conspiracy was harder to identify. In January 2011, O'Neill was examining more than 15,000 emails from an anonymous account when he found two that stood apart. They were from a personal email and had attachments that were core to the scam: a program that masked the hackers' activities in their victims' computers, and a trove of stolen credit card numbers.

Email mistake The hacker had mistakenly used his personal account to forward himself the information. The misstep was enough for O'Neill to finger Iulian Dolan, 25, a third Romanian. O'Neill and the federal prosecutors still weren't optimistic that they could put the Romanians on trial. Though the US has an extradition treaty with Romania, getting the country to hand over suspects was far from guaranteed. "I thought our best case scenario would be that we would approach Romanian law enforcement and hope we could convince them to prosecute these people, assuming we could ever able to identify them," said Mona Sedky, a prosecutor in the US Justice Department's computer crimes division. "I never in a million years thought they would see the inside of a US courtroom." O'Neill and his Romanian counterparts discussed his options, which amounted to taking the risk of trying to extradite the men, or finding a less official way to arrest them. "They basically said, 'Do whatever you can do legally to get them to the United States,'" O'Neill recalled.

Ladies man Research showed the thieves had obvious weaknesses: Dolan was an online gambler, and Butu was a ladies' man. To capture Dolan, O'Neill became Sarah, a marketing specialist for a Connecticut casino who invited the Romanian to a poker tournament. For eight months, O'Neill emailed Dolan, sometimes late at night from his home where he was on paternity leave with a baby boy. His wife was understanding: she's an agent with the FBI. "Dolan seemed kind of like a lonely guy," O'Neill said. "And, yes, there was some gentle flirting." When Dolan finally walked off the plane in Boston on August 13, 2011, he was carrying a gold necklace for "Sarah" and six boxes of grape-flavoured condoms. "He was being optimistic," O'Neill said.

Meanwhile, O'Neill was also masquerading as Chrissy, a wealthy waitress for a restaurant chain known for its scantily-clad servers, who had met Butu during a recent sojourn through Europe. Wealthy waitress O'Neill's gambit this time was that "Chrissy" had enjoyed meeting Butu and hoped to re-establish contact. Eventually, "Chrissy" invited Butu to visit her in the US. Butu took the bait, arriving in Boston the day after Dolan. "You tell them what they want to hear, within reason," O'Neill said. He took the traditional route with Oprea. The US government sought the Romanian's extradition. It worked: Oprea was arrested in December 2011 and sent to New Hampshire in May of last year.

All three pleaded guilty to hacking-related charges, admitting they hit more than 800 US stores, about 250 of which were Subways. In interviewing Dolan and Oprea, O'Neill determined they didn't target Subway. It was just luck that so many got hacked. There are about 25,000 Subways in the US and many had poor online security, O'Neill said. As for the hackers, they didn't make much profit – Oprea, the ring leader, made only $US40,000. He paid a steep price for the estimated $US12.5 million in losses inflicted on financial institutions and the $US5 million Subway spent upgrading its cyber security systems. Oprea was sentenced in September to 15 years in US federal prison. Butu got 21 months behind bars, and Dolan received a 7-year sentence. "They weren't stealing from Romanians," O'Neill said, "so they never expected to get caught." Bloomberg Follow IT Pro on Twitter