For ten days in March, millions were caught in the same massive spam campaign.

Each email looked like it came from someone the recipient knew: the spammer took stolen email addresses and passwords, quietly logged into their email account, scraped their recently sent emails and pushed out personalized emails to the recipient of that sent email with a link to a fake site pushing a weight loss pill or a bitcoin scam.

The emails were so convincing more than 100,000 people clicked through.

We know this because a security researcher found the server leaking the entire operation. The spammer had forgotten to set a password.

Security researcher Bob Diachenko found the leaking data and with help from TechCrunch analyzed the server. At the time of the discovery, the spammer’s rig was no longer running. It had done its job, and the spammer had likely moved onto another server — likely in an effort to avoid getting blacklisted by anti-spam providers. But the server was primed to start spamming again.

Given there were more than three million unique exposed credentials sitting on this spammer’s server — hosted on intelimost.com , we wanted to secure the data as soon as possible. With no contact information for the spammer — surprise, surprise — we asked the hosting provider, Awknet, to pull the server offline. Within a few hours of making contact, the provider nullrouted the server, forcing all its network traffic into a sinkhole.

TechCrunch provided a copy of the database to Troy Hunt. Anyone can now check breach notification site Have I Been Pwned to see if their email was misused.

But the dormant server — while it was still active — offered a rare opportunity to understand how a spam operation works.

The one thing we didn’t have was the spam email itself. We reached out to dozens of people to ask about the email they received. Two replied — but only one still had a copy of the email.

“The same mail appeared on three occasions,” said one of the recipients in an email to TechCrunch. “The subject was related to an email I had sent previously to that person so the attacker had clearly got access to his mailbox or the mail server,” the victim said.

The email, when clicked, would direct the recipient through several websites in quick succession to determine where they were located, based off their IP address. If the recipient was in the U.S., they’d be pushed to a fake CNN site promoting a bogus health remedy. In this case, the spammer was targeting U.K. residents — and most were directed to a fake BBC page promoting a bitcoin scam.

The spammer had other servers that we had no visibility into, but the exposed server revealed many of the cogs and machinery to the operation. The server, running an Elasticsearch database, was well-documented enough that we found one of the three spam emails sent to our recipient.

This entry alone tells us a lot about how the spam operation worked.

Here’s how it works. The spammer logs into a victim’s @btinternet.com email account using their stolen email address and password. The scammer pulls a recently sent email from their victim’s email server, which feeds into another server — like inbox87.host and viewmsgcs.live — tasked with generating the personalized spam email. That email incorporates the subject line of the sent email and the target recipient’s email address to make it look like it’s being sent from the real person.

Once the message is ready to send, it’s pushed through a proxy connection, designed to mask where the email has come from. The proxy server is made up of several cell phones, each connecting to the internet over their cellular connection.

Each spam message is routed through one of the phones, which occasionally rotates its IP address to prevent detection or being flagged as a spammer.

Here’s what that proxy server looks like.

Once the spam message leaves the proxy server, the spam message is pushed through the victim’s own email provider using their email address and password, making it look like a genuine email to both the email provider and the recipient.

Now imagine that hundreds of times a second.

Not only was the spammer’s Elasticsearch database leaking, its Kibana user interface was also exposed. That gave the spammer a detailed at-a-glance look at the operation in action. It was so granular that you could see which spam-sending domains were the most efficient in tricking a recipient into clicking the link in the spam email.

Each spam email includes a tracker in the link that fed information back to the spammer. In bulk, that allows the spammer to figure out which email domain — like outlook.com or yahoo.com users — is more likely to click on a spam email. That can also indicate how an email provider’s spam filter acts. The greater number of clicks, the more likelihood of its spam going through — allowing the spammer to target specific email domains in the future.

The dashboard also contained other information related to the spam campaign, such as how many emails were successfully sent and how many bounced. That helps the spammer home in on the most valuable logins in the future, allowing them to send more spam for lower bandwidth and server costs.

In all, some 5.1 million emails were sent during the 10-day campaign — between March 8 and March 18, with some 162,980 people clicking on the spam email, according to the data on the dashboard.

It’s not the first time we’ve seen a spam operation in action, but it’s rare to see how successful it is.

“This case reminds me on several other occasions I reported at some points in the past — when malicious actors create a sophisticated system of proxying and logging, leaving so much tracks to identify their patterns for authorities in the investigations to come,” Diachenko told TechCrunch. “This shows us — again! — how important a proper cyber hygiene should be.”

What’s clear is that the spammer knows how to cover their tracks.

The language settings in the Kibana instance suggested the spammer may be based in Belgium. We found several other associated spamming domains using data collected by RiskIQ, a cyberthreat intelligence firm, which scours the web for information. Of the domains we found, all were registered with fake names and addresses.

As for the server itself, the provider said it was possibly hacked.

“This was a resold box and the customer already responded to the abuse forward saying it was supposed to have been terminated long ago,” said Awknet’s Justin Robertson in an email to TechCrunch.

And we still can’t figure out where the email addresses and passwords came from used to send the spam. Only 45 percent of emails were already in Have I Been Pwned, ruling out the possibility that all of the passwords were stolen from credential stuffing.

Since the hosting provider pulled the spammer’s server offline, several of their fake sites and domains associated with the spam campaign no longer load.

But given the spread of domains and servers propping up the campaign, we suspect the sunken server is only a single casualty in an otherwise ongoing spam campaign.

Got a tip? You can send tips securely over Signal and WhatsApp to +1 646-755–8849. You can also send PGP email with the fingerprint: 4D0E 92F2 E36A EC51 DAAE 5D97 CB8C 15FA EB6C EEA5.