Ever since news broke that thieves stole more than 40 million debit and credit card accounts from Target using a strain of Point-Of-Sale malware known as BlackPOS, much speculation has swirled around unanswered questions, such as how this malware was introduced into the network, and what mechanisms were used to infect thousands of Target’s cash registers.

Recently, I spoke at length with Tom Arnold and Paul Guthrie, co-founders of PSC, a security firm that consults for businesses on payment security and compliance. In early 2013, these two experts worked directly on a retail data breach that involved a version of BlackPOS. They agreed to talk about their knowledge of this malware, and how the attackers worked to defeat the security of the retail client (not named in this story).

While some of this discussion may be geektacular at times (what I affectionately like to call “Geek Factor 5”), there’s something in here for everyone. Their observations about the methods and approaches used in this attack point to an adversary that is skilled, organized, patient and thorough.

So you first saw BlackPOS at a retailer in early January 2013?

Tom: Yes, it did seem like a game changer at that point because of the way it hooked into the POS system. By that I mean the fact that it hooks into the POS process, and it’s not just a general memory scraper like some people have stated.

Help me understand the distinction between a memory scraper and malware that runs completely in memory?

Tom: Well, it’s very specific. It’s what’s called an inter-process communications hook. If you look at a memory scraper — so if you were to dump the memory on your laptop right now — you would get this big old blob of information and you would have to filter through it to find what you’re looking for. But this [malware] is very specific: It’s very much designed for the POS system it’s running on, because it knows exactly where to hook and where the memory location is going to be when the data it’s looking for is flowing through it. But if you look at what it captures, it captures only the track data [information stored on the magnetic strip on the backs of cards]. So, it’s actually very, very sophisticated and that’s why I think this isn’t just a teenage hacker who put this together in his basement. I think this is a more sophisticated development effort. [HP last week published some interesting, uber-geeky details about the memory behavior of the version of BlackPOS found at Target].

Paul: It certainly hooks into a specific process, but we did not figure out if it was just good at scraping the memory of that process, or whether it actually altered the process to hook into the code somehow. That part of the code was obfuscated and not reviewed during the engagement.

What did you think of the iSight paper?

Tom: The iSight paper was good, and what it described was very similar to what we saw in the first variants of BlackPOS. But it didn’t talk about how it appeared on the network or where it came from or how a retailer might defend themselves against it.

In your prior experience with BlackPOS, has it been used against the same POS that Target uses? A source who seemed to have a clue told me that their setup was somewhat homegrown.

Tom: That I don’t know. I haven’t really looked at what Target uses. With a homegrown system, the problem is if you’re going to build a process hook, you have to have a test environment, you have to know what you’re looking at and what memory addresses to go after, and that’s not exactly something that gets published.

Paul: Target has a homegrown POS.

So you think it’s likely they were using some off-the-shelf equipment and software? Wouldn’t it be enough for the attackers in this case to have obtained a physical point of sale device that was once used at Target?

Tom: If they have one, sure, that would be different. I don’t know if they’re using a commercial product. A lot of the big retailers use commercial products and will customize those with their back end system. But at the front end and what happens at front of the house…a lot of those are just retail off-the-shelf applications. A lot of those retailers, when you have a hard disk that breaks on one of the [checkout] lanes, they’ve got an outsourced service provider of that POS that comes out there with a new hard disk and fixes it.

How do the bad guys break in, and how do they actually get the malware pushed out to the point-of-sale?

Tom: A lot of the POS systems use whitelisting software of some kind, such as Bit9 or SolidCore. Those two companies are the two you see most out there in the industry.

Paul: It could also come in through the point-of-sale application update process.

By whitelisting, you mean sort of the opposite approach of antivirus, right? As in, if the file or program isn’t approved by the whitelisting software, it simply won’t run on the system?

Tom: The software update processes at a point-of-sale that is running one of those [whitelisting applications] has to come through one of the software update channels and has to be reviewed for the update and approved. And when it’s approved, the whitelisting software says okay this patch is approved to come online.

It’s probably a good idea at this point to make sure we’re defining the terminology in a uniform way. When you think of point-of-sale device, are you talking about the cash register, or the card swipe terminal, or…

Tom: I’m using point-of-sale to mean the payment application that is running on the cash register. The vast majority of those devices, when you check out at the grocery store or large retailer, those are just PCs, and yes they’re mostly running Windows XP or WEPOS as their operating system. But above that, you have what I call the point-of-sale, or the point-of-sale application, and that’s the stuff that the cashier is interfacing with at the time you’re checking out. It’s a software application, running as multiple pieces of software inside the register itself. And whitelisting is put on to protect the register from any sort of unsolicited modification. A lot of the attacks before this [whitelisting became more broadly used] involved where you corrupt a sales clerk to put a USB stick in the cash register and infect the PC with some malware. But by using a whitelisting software, the USB stick will not work and the operations personnel back in the head office will get notified that something isn’t right with that register.

If they get past a whitelisting system, doesn’t that suggest that someone on the inside would have to be involved?

Tom: No, not really. You have to also consider the distribution server that distributes the point-of-sale software.

Paul: Right… three possible ways… it could come through a legitimate update channel, or the retailer was lax in their update procedures, or the attackers hacked the console of the whitelisting software and just whitelisted it themselves.

So when you look at a hacked system, what can you determine about the state of the POS software on the affected systems?

Tom: Well, there can be huge problems if the retail software vendor does not provide SHA-1 hashes or even MD5 hashes of their software. There would be no way to tell whether what you were receiving had not been tampered with..

How does the update process typically work?

Tom: Basically the retailer would receive notification of a patch, and go off to an outside server and pull it down. Very similar to you getting notified of a patch being available on a Windows machine. Except in these situations, when you’re dealing with critical retail software and installations, patching is very formal process, because uptime and reliability is very important. So they test it. They would install the patch on their internal lab servers, run and test it to verify that everything would run smoothly. Then they would then certify the patch when it was ready to run, and then distribute it to stores 1-5 tonight, and then go 6-10 then next day, or whatever their mechanism was to distribute them. But it would typically take them 2-3 months to finish the distribution to all the stores.

One of the questions still outstanding in the Target breach, which is where did this stuff come from and how did it distribute itself across the network? Do you have insights or best guesses on that? And wouldn’t the mechanism like the one you just described be the most likely way…some kind of POS update mechanism?

Tom: Well, if they were not running whitelisting software…maybe, yes. But you know the other thing was that the malicious software we have dealt with has sort of a worm-like characteristic to it. It would keep trying to worm its way around the organization. If it found itself already installed on a machine, it was bright enough to elect that machine and it would start seeking a way out of the organization. When we looked at this in the past, it was very difficult for us to determine how they tried to exfiltrate the information. I think what they did was…since they could find a host — remember, this is an early variant of what was seen at Target — they sort of elected a machine that became the exfiltration point and started doing the exfiltration. Worked exactly the same way, in that the data was encrypted, and moved off to an exfiltration point, and from there it was pulled off to Mother Russia.

Paul: Once they have a password that works on every system, they can use that to spread the malware. There would be nothing that that would stop a POS device in store 12 from talking to a register in store 23.

What type of encryption did they use?

Paul: RSA-512 bit.

But this information wasn’t encrypted by the point of sale or card swipe device at that point?

Tom: Well, remember this was taken off the memory, so it was encrypted from the card terminal, but when it got internally so the [cash register] could handle it, the system itself would decrypt it, and pass it to another process that used for communications, and then it would go off network to get the authorization. And the process hook would grab the data at the point that it gets decrypted out of memory. Then the malware used its own key to encrypt it.

So at that point are there two parallel streams of data here?

Tom: Yes. One good, one bad. And then the malware itself, would then encrypt that data, and pass it to where it was going to aggregate it from all these infected point-of-sale devices, and then the aggregation server would encrypt it again for exfiltration. So it’s pretty sophisticated stuff.

So the bad guys in the scenario that you describe were double-encrypting their stolen data?

Tom: Yes. And that’s why….a lot of people ask the question, hold on a second. Wouldn’t intrusion detection systems flag this; if they see credit card numbers floating around the internal network and moving outbound, they classically raise the alarm. Yet no alarms would go off, if it was encrypted. In the case of Target, it sounds like they were using those encrypted packets to hide from data loss prevention and intrusion detection systems.

Did the forensic report from iSight say the data stolen from them was encrypted in the Target breach? I don’t recall reading that.

Tom: Well, if they were using BlackPOS, I’m pretty certain it was encrypted. Because, if Target is following PCI best practices, there’s no outbound traffic moving at that point, and I would expect that they would have an intrusion detection system in there, because that would be required. And in that case, the traffic would have become visible as card data leaving the network. Now there’s nothing that says you have to do data loss prevention like that [in the PCI standard] but a lot of big organizations do.

We were never able to figure out how they egressed the data. The only protocols that were allowed outbound were DNS. So if they wrapped the stolen data up in outbound DNS packets, maybe. We never solved that problem, we were still looking when the engagement ended. We could see it was doing a lot of the work, but we couldn’t see it was actually leaving the organization.

How is that even possible? That you would not have been able to see how they offloaded the card data?

Tom: There were a lot of modules inside of BlackPOS that were anti-forensics modules, so it cleaned up a lot after itself. It did look for a specific time window for when it would actually move data. There was a whole timing vector as to when it would turn itself on, and then it would start seeking things. We put the software in a lab, but it never [sent data outbound] so we couldn’t quite figure out how. In my client’s case, the firewall rules outbound were very strict, and we couldn’t figure out how it was getting the data out of the organization.

You said earlier that this malware had something of a worm-like capability. Can you talk more about that?

Tom: The iSight report alluded to this when it said there are a whole bunch of hacking and forensics tools in there, and it really did have everything but the kitchen sink in there in terms of tools. The actual transfer mechanism seemed to be a VBSCript. When it got involved, it would very gently start mapping the network, and so that it became aware of what was in the network with it. And then it would start seeking out other point of sale devices. It was very careful, and it took its time doing this. It would attempt to communicate with those other POS devices. And if one of those devices was already infected, the two would form a handshake and a bond, and the two of them would look for the third, fourth and fifth, and so on.

If it would see like a manager’s laptop, it would try to push itself there, but then when it got to that laptop – if it successfully infected it — and it didn’t see the point of sale software, it would run its anti-forensics software and completely destroy itself, so that there was no trace of it at all. We actually took the software and forced an infection onto one of our test laptops, and it only lived for about 15 seconds before it completely destroyed itself. And then it went through and did a whole bunch of anti-forensics work before it destroyed the anti-forensics module. Very cute program. It cleaned up after itself pretty nicely.

So this is why I think as an old software engineer, I was looking at it saying someone did a lot of work to make this thing run properly, and they did a lot of testing. This is not just one guy. If it is the work of just one guy, he’s absolutely brilliant.

ANALYSIS

From talking with Paul and Tom — and in discussing the Target case with security experts from other retailers — a few things seem clear (these are my personal takeaways, not theirs). Firstly, many retailers only update their point-of-sale systems according to a well-planned schedule (a schedule, by the way, which typically happens well before Black Friday — the busiest and most profitable day of the year for most retailers). As a result, depending on the size of the retailer, that update process can take weeks, even months. Their experience suggests that whoever broke into Target was inside Target’s network for quite some time before the point-of-sale malware started stealing card data on Nov. 27, 2013.

Secondly, many reporters and readers have been asking what retailers like Target could have and should have been doing from a security perspective. I don’t have much information on what Target or other retailers were or were not doing, but assuming the attackers typically take months to orchestrate and execute these attacks, it stands to reason that more quickly detecting these intrusions would help quite a bit. According to Verizon’s 2013 Data Breach Investigations Report, 66 percent of breaches that Verizon responded to in 2012 took months or more to be detected. Too often, the breaches aren’t even first detected internally. It’s worth noting that Target’s case, the company’s CEO acknowledged being alerted to the breach on Dec. 15 by law enforcement. In the case of the breach at Neiman Marcus, which exposed some 1.1 million credit and debit cards, the company said it learned of the breach on Jan. 1, 2014, although the actual card thefts occurred between July 16 and Oct. 30, 2013.

Finally, I have to wonder how many times this scene played out in 2013: An individual forensics firm analyzes a sophisticated retail breach involving point-of-sale malware – collecting mountains of interesting and useful data about the threats, threat actors and their methods — but at the end of the day has no mechanism by which to share that information with others in the retail and security community. It’s telling that most of the details about the malware and methods used in the Target breach were first published here on this blog. That’s not a brag: That’s me being incredulous at how the industry as a whole still sucks at sharing important information.

I hope that this interview helps other digital first responders, and that it fosters a more public debate about the malware and miscreants responsible for what appear to be a large number of very similar data breaches that will no doubt continue to come to light this year via additional victim disclosures (willing or otherwise).

Tags: Bit9, Black Moon, BlackPOS, iSight, Paul Guthrie, PSC, SolidCore, target breach, Tom Arnold