Summary: the timing of the crash and the continued glaring vulnerability of Verrit’s web server make poor planning a far more likely explanation.

UPDATE 9/12/2017: I’ve edited this post to reflect new information I received from an engineer yesterday. Verrit’s original IPv4 address was hit with a minor UDP NTP amplification attack shortly after their launch. Peter also shared this email today from his provider which provides similar information. I remain extremely skeptical that Verrit was the target, and the new information shows with certainty that it was not “sophisticated and persistent”. Quite the contrary; it wasn’t persistent because the site has remained up since it changed IPs, and it wasn’t sophisticated because, among other reasons, the attack bandwidth is such that it could have been mounted by one person with a single 10 megabit connection. However, given the way NTP amplification works, it still could have been enough to cripple a Linode slice/VPS and possibly even saturate its inbound port(s), especially in combination with the massive flood of legitimate traffic that was inbound at the same time. Without a view into exactly how many clicks were generated toward the Verrit server, it’s hard to determine what percentage of the server’s traffic was legitimate. It might sound like a dodge to say “we’re both right”, but we really are. Peter is right about the NTP attack. My facts (if not my implication) all remain technically sound and well-founded. This is a textbook example of a bungled site launch, and I think if I got Peter drunk enough he’d admit that. Even if Verrit really was a target that evening, it could easily have been prevented with the most basic digital groundwork. Also, my key point becomes more painfully obvious by the hour: Verrit is not being targeted by malicious forces intent on silencing “the 65 million”, and I continue to believe that mythos forms an unfortunately large part of Peter’s narrative. However, in my original post, I strongly implied that the Verrit server fell victim entirely to legitimate traffic. It’s now clear that was not the case. The NTP event definitely contributed to the server’s troubles, and I apologize to Peter and Leela for implying otherwise. While my facts and key points were correct, I’d nonetheless like to atone for the implication itself. So, I’ve written a check for $500 to the International Rescue Committee, the non-profit Leela mentioned earlier in the week.

Recently, I’ve earned myself a bit of a reputation for calling out tech BS that emanates from certain political activists on Twitter. For the last few months, I’ve been occasionally pestering Louise Mensch about her wilder claims, especially those based entirely on humorously flawed interpretations of technical “clues”. Her fans — allowing their confirmation biases to feast on any and every morsel of dreck served up about Trump — react with passion, accusing me of being a Russian agent or a “Trump bro”.

So I want to start by explaining what motivates me to be a gadfly about this stuff, and hopefully pre-empt a few “omg why do you care so much?” questions in the process.

I view Donald Trump as an existential threat to the Great American Experiment, and when his detractors are provably full of shit, it hands him and his enablers everything they need to bury the terrifying truth under the weight of our own lies. When these lies are exposed — and most of them will be — he can say to America, “See?! More fake news!”

You can accuse me of being politically motivated — and I suppose, in the way I’ve described above, I am — but you can’t call me an enemy of the cause. I’ve criticized Trump for years, worked for Bill Clinton, raised a lot of money for Barack Obama, and voted for Hillary Clinton last year.

If you truly care about America’s future, you should want a rigorous public dialogue based entirely upon facts, scrubbed of lies, innuendo, and partisan victimhood narratives. And, you should want to see an equally honest and impartial judicial process, with the President and his comrades supported by the best possible defense. It’s precisely because I want America to survive that I find hyperbole so odious.

Now that we have that out of the way, let’s explore the claim that Verrit was targeted by a “sophisticated and persistent” DDoS attack. The best place to start is with a basic explanation of a DDoS attack. There are many types, and they’re beyond the scope here. Cloudflare has an excellent explainer if you’re so inclined, and their CEO has written extensively about their technical approach. In a nutshell, DDoS means hitting the target server with so many requests that it runs out of resources to handle them. This results in legitimate traffic going unanswered. Anonymous, the hacktivist collective, is well-known for using this technique to cripple target organizations. Their attack on the Daily Stormer is a recent example. You’ll notice they were targeting IP addresses. Remember this for later.

There are myriad ways to attack a site, and not all of them require a direct IP address. But by far the easiest is directly via its IP address. Armed with that, knocking a site offline is fairly trivial. That’s why companies like Fastly boast about their ability to cloak server IPs from even the most sophisticated attackers. With no IP, life gets much harder for the bad guys.

When you type a domain name into your web browser, the browser does a “lookup” with DNS (the Internet’s “phone book”) to find out which IP address to connect to. If a site is behind a protective network like Cloudflare or Fastly (this is called reverse proxying), then — assuming the site owner has carefully configured their DNS — it’s much harder to discover the web server’s IP. That’s because the IP that’s returned for lookups is an IP for the reverse proxy (the service’s network), and not the actual web server.

The use of something called Anycast diffuses those special IP addresses across thousands of high-bandwidth servers around the globe. That makes mounting an attack many orders of magnitude more difficult, because in order to attack one customer, they have to attack the service’s entire global network. This disparity is hard to overstate: it’s like the difference between beating up an out-of-shape drunk and taking on an entire U.S. Army brigade.

There are many other reasons why this diffusion is good: perhaps the most impactful is edge caching, which serves content (videos, images, web pages, etc) from a server close to the end user. A video that’s going viral can be served to users in Tokyo from an edge server in Tokyo, London in London, etc. This dramatically improves the user experience (much faster load times) and substantially reduces the load on the origin server, because each edge server only needs to ask the origin server for the file once, then it will serve the file on its behalf to users in that region.

There are many ways for attackers to figure out a target server’s IP, but by far the easiest is for the site owner to tell them. And the silliest way to do that is with public DNS entries that point directly at the origin server, such as “mail.mysite.com” or “forum.mysite.com”. Cloudflare specifically warns customers not to make this mistake.

As of this writing, Verrit’s DNS zone file is still offering up its origin server’s IP address to anyone who asks. I’m not going to publish it, but it can be discovered in ten seconds by anyone with a shred of technical know-how.

Now let’s look at the timeline. On Sunday afternoon, Hillary tweeted her Verrit endorsement. As it began spreading, the site quickly became unreachable. Many observers didn’t need a blowhard like me to tell them what was going on; all they needed was Occam’s razor, which says Verrit fell victim to bad planning, and the server simply wasn’t prepared for the crush of traffic. “Slashdot Effect” is the original term for this, but I like Reddit’s “hug of death” better because that’s what it really is — there’s so much inbound love that it kills you.

When Hillary tweeted Sunday afternoon, Verrit had absolutely no edge caching or DDoS protection in place. We know that because the IP history of the Verrit domain tells us so:

(for you nerds out there, the Amazon IP is for Amazon EC2, not Cloudfront)

This means, indisputably, that until sometime Monday, 100% of the clicks from Hillary’s tweet (and anyone else linking to Verrit) from anywhere in the world were all routed directly to one lonely little web server sitting in a rack in Newark, New Jersey. That’s right — every click, every page, every image — all had to be served from that one box. Launching a site that’s expected to get national attention with no DDoS protection or edge caching in place is absolutely crazy.

Peter was likely one of the first people to notice his site had shit the proverbial bed, and presumably kicked his shadowy tech team into action, spending the evening hours frantically signing up for Cloudflare and doing many other things they should have done weeks before.

On Monday, he blamed the blackout on a “sophisticated and persistent DDoS attack”. The first I heard of this (and the first I heard of Verrit) was when someone, in the wee hours of Tuesday morning, retweeted this into my timeline:

So I asked, politely, how he knew.

His answer was more than a little opaque:

Bored and sleepless, I did a little sniffing around. I noticed their sudden Sunday night transition to Cloudflare — and discovered what I mentioned earlier: their web server’s IP address was still completely exposed. Wide open. Anyone who wanted to DDoS them could do it in a moment. That singular fact is critical to this story: with the server’s direct IP address, if any “sophisticated attacker” wanted the site offline, the site would be offline. If it was a persistent attack targeted at Verrit, then changing the server’s IP without cloaking the new IP would provide, at best, a very brief reprieve.

But publicly pointing out the gaping security hole seemed irresponsible until I gave him a chance to plug it. I immediately asked Peter to DM me. He did, and I gave him step by step instructions on how to fix it. He thanked me and told me he’d pass it along to his tech team.

My skepticism was noticed by a few journalists, and a couple of them wanted details. So, half a day later, I messaged Peter again to tell him I wanted to defend my skepticism but didn’t want to provide an attack vector (however obvious this one may be).

That’s when things got a little weird. Peter asked if I planned to tell the journalists what I’d figured out. I said yes, because it was the only way to justify my prior public comments. He said he was “working on a lot of issues and [was] underwater” and that I was “free to discuss whatever [I] want”. Then, in what felt like a misguided attempt to quell my meddling, he told me he was engaging “the authorities” and there were things I “may not be aware of”. Then, he blocked me.

That bizarre exchange took my confidence level from 98% to 99.9%.

In conclusion, Peter is asking us to believe that “five [unnamed] engineers from two [unnamed] firms .. including [an unnamed] cyber security expert” worked to fend off a “sophisticated and persistent attack”, yet left the most obvious DDoS vectors wide open, even days after it was pointed out to them. He’s asking us to believe Verrit was the victim of a paralyzing digital assault, yet he still hasn’t bothered to lock the front door. He’s asking us to believe that dark forces are out to silence Verrit and “the 65 million”, but these forces suddenly felt guilty Monday afternoon, turned off their ion cannons, and went home.

Either that’s all true, and I’m wrong, or Hillary gave Verrit a “hug of death” and Peter’s just winging it. You be the judge.