Time Warps, Mining Exploits, Denial of Service, and More!

Cryptocurrency enthusiasts are keen on telling ordinary civilians how safe and secure the Blockchain protocols powering their favorite coins are. Indeed, major cryptocurrencies like Bitcoin and Ethereum have maintained their security quite well — better, arguably, than any other digital asset/payment system in history — which is pretty remarkable, considering that they are unbacked digital money free from any single party’s control with an effective multi-billion dollar bounty on their proverbial heads.

Many, however, will go a step further, and declare said cryptocurrencies to be literally “unhackable.” This is, at the very least, a tactical error, since the proliferation of the “unhackable” meme forces the enthusiast into some awkward positions when and if certain events unfold. Like, say, a hack.

In such an event, it seems that, if nothing else, an explanation is in order.

Last month, an as-of-yet unidentified attacker was able to severely compromise Verge, a relatively small, privacy-focused cryptocurrency. The mystery hacker managed to dominate the network on three occasions for intervals of several hours at a time over the course of April 4th-6th, preventing any other user from making any payments. Worse, in that interval, they were able to generate what is effectively counterfeit Verge at a rate of 1,560 Verge coins (roughly $80) per second, minting what amounted to over a million dollars worth of the currency.

No need to beat around the bush — this was a disaster. The thing was hacked to high heaven.

But who’s to blame? Is this a case of human error on the part of the Verge developers, an undermining of crypto fundamentals, or something in between? Could such a thing happen again, perhaps to larger currencies, and if so, what can be done to prevent it?

With these sorts of breaches, many details inevitably remain murky. However, in this case, the fundamental exploits can be fairly clearly understood. Onward:

Timestamp Spoofing (Or: Honest Mistakes vs. Dangerous Lies)

The root of the exploit is something that would appear, prima facie, to be a bug, but is actually a deliberate feature: the ability to create “inaccurate” timestamps. In blockchain protocols, individual transactions (usually payments from one party to another) are grouped together into a single block, which is then confirmed as a whole. Every block comes with a timestamp of its creation date. Even when a blockchain protocol is functioning properly, the ordering of these timestamps may sometimes be out of sequence; i.e., block 100 may have a timestamp that actually comes after block 101. This is because, in decentralized networks that obstinately refuse to grant any special authority to third parties, accurately enforcing time synchronization is no simple matter. Given the unpredictable variance in the time it takes for data to propagate through the peer-to-peer network, it’s entirely possible for block times to appear “out of order,” even when all parties are being perfectly honest. In other words, it’s only fair to allow some degree of flexibility; in the case of Verge (before the hack, anyway), the protocol allowed nodes to “disagree” about the current time by a window of, at most, two hours.

The entry point for the hacker was to start spoofing timestamps, submitting blocks that appear to be from the past, but are still within the allotted two-hour window, and thus eligible for acceptance by the other nodes.

Why this would ultimately matter for network security has to do with the nature of proof-of-work mining.

Mining Difficulty (Or: Gates Only Work When They’re Raised)

Keeping the Verge network decentralized requires ensuring that fairly small-scale devices (macbooks, say) can participate in running the network’s software. This, in turn, means limiting the volume of payment activity on the network; i.e., setting a clear target block time (and in turn, a limit on the network’s transactions per second). For Verge, the target is one block per 30 seconds. Now, one might well ask, given that the network is decentralized, how could this be enforced? What’s stopping parties from submitting blocks at a much faster rate? This is no trivial problem; given that an accepted block earns its submitter a block reward, it’s in the submitter’s incentive to confirm blocks as fast as possible.

The answer, in short, is proof of work mining. For a block to be considered valid by the network, it must contain a solution to a cryptographically difficult computational problem derived directly from the data in the block itself. The nature of this problem is such that the difficulty can be freely adjusted. The target block time for Verge is one block every 30 seconds, and the difficulty of mining blocks is constantly being adjusted based on the current rate of block confirmations; if more people decide to devote more mining power to generating Verge blocks, and more blocks get mined faster, the protocol increases mining difficulty and block submission is throttled. Conversely, as mining power lowers and block time increases, mining is made easier. Thus, when properly functioning, even as messy real-word factors change — economic fluctuations, market prices of crypto, energy markets, empires rising and falling, etc — the Verge network is perpetually reacting and guiding the network to our target block-rate equilibrium.

The algorithm that Verge uses to calculate the current difficulty is known as Dark Gravity Wave; it involves taking a weighted average of the rate of block confirmations over a moving 30-minute window. It’s a bit complex, and the details don’t really matter here — what matters is this: mining difficulty is a function of recent block frequency, and running calculations on block frequency naturally involves looking at blocks’ timestamps.

And hence the problem: if enough faulty timestamps are getting created, all bets are off. And this is what the hacker did — examining the blockchain data reveals that throughout the duration of the hack(s), every other block was submitted with a timestamp roughly one hour before the present time, tragically confusing the protocol’s mining adjustment algorithm. If the protocol were sentient and fluent in English, it would be saying something like “Oh no! Not enough blocks have been submitted recently! Mining must be too difficult — let’s make it easier!” Since timestamps were continuously being spoofed, the protocol continuously lowered the difficulty, until mining got laughably easy. To give a general idea, the average difficulty in the hours before the initial attack was 1393093.39131, while during the attack, it got as low as 0.00024414, a decrease in difficulty of over 99.999999%. Lower difficulty in submitting a block means more blocks get submitted— in this case, roughly a block every second.

The cleverness of this attack is in how it circumvents the barrier of mining difficulty instead of attempting to burst through it. If the security provided by mining power is a gate surrounding the network — a gate that’s far too strong to break through and too high to climb over — this hack gets past it by finding a way to lower it so close to the ground that it can be stepped over.

If it isn’t already obvious, this is, in and of itself, bad news. A violation of the intended protocol this blatant is simply not a good look. Additionally, the drastically increased block submission rate means lots of more newly mined Verge than the protocol had allotted, which is bound to bother you if you’re the sort of economist who has a thing for sound money with predictably high stock-to-flow ratio.

However, lowering the difficulty is only half the story; in isolation, it wouldn’t actually give the attacker any advantage. With the difficulty drastically lowered, mining blocks does become easier for the attacker, but it also becomes easier for everyone else; miners are still competing against each other just like before. What we would expect to see is that although, yes, blocks get mined faster, the identities of the successfully miners should be just as distributed and democratic as before. Or, put another way, no matter the difficulty, a single attacker would still need 51% of the mining power to dominate the network, which is just as hard as it was to do before the attack.

However, this hacker did indeed take over the entire network, and was able to do so with far less than 51% of the hash-rate. What enabled them to do this is the second component of this exploit, which has to do with Verge’s use of multiple mining algorithms.

Verge Mining (Or: When Five Algorithms Aren’t Better Than One)

Generally, blocks in proof-of-work based cryptocurrencies are mined by a single algorithm, the most common being SHA-256. Verge, however, allows miners to use any of five different algorithms (for those dying to know, the algorithms Verge uses are Scrypt, X17, Lyra2rev2, myr-groestl and blake2s.) The rationale for using multiple mining algos is something like this:

Some critics of Bitcoin argue that over time, the Bitcoin mining industry has gotten too specialized and centralized; most mining of the currency, for example, is performed by Bitcoin ASICs, devices created for the sole purpose of mining the currency, and much of Bitcoin mining is performed by a few mining pools — groups of miners that amalgamate their resources together and share the rewards proportionally. These trends towards different types of “centralization”, they say, are antithetical to the fundamental value proposition of permissionless cryptocurrencies. Having a coin use multiple mining algorithms purports to be a bulwark against these trends. The argument goes that controlling five different algorithms in terms of hardware, industry, and resource management, is bound to be harder than controlling just one, pushing the Verge mining economy in the more distributed, decentralized direction.

So here’s the deal: the only way for this to properly function — and “properly functioning” in this case includes maintaining 30 second block times, keeping all five algos economically sound for miners, and preventing one of the five algos from dominating (which would render the whole experiment pointless) — is to have each algorithm have its own difficulty parameter that gets adjusted independent of the other four. Which is to say, Scrypt’s mining difficulty is adjusted to hit 30 second block equilibrium, as is X17’s, and so on.

What this means is that our timestamp forger didn’t actually lower the difficulty of mining for the whole network; he only lowered it for those mining with one of the five algorithms — Scrypt, it turns out. So while the Scrypt miners now all enjoy comically easy mining difficulty, the miners utilizing the other four algos are stuck having to work just as hard as before, rendering all of their hash-power effectively useless for securing the network. Crucially, this meant that the attacker only had to mine with the Scrypt algorithm and only had to compete against the others doing the same; thus, required hash-power for our attacker to dominate goes from over 50% (dominating the whole network) to over just 10% (dominating the other Scrypt miners).

Now at this point, things do get a bit speculative, but it would appear that the situation was a lot worse than even that. The “10%” estimate stems from the fact that given the nature of mining difficultly adjustment, roughly the same amount of economic resources should be applied to each of the five mining algos. Reality, however, often has a way of stubbornly refusing to conform to the axioms of free market economics. According to discussions within the community, various factors — the existence of Scrypt ASICs that had yet to be deployed, the spare resources available for rent via Nicehash, etc — meant that Scrypt would, at the time of the attack, have been far cheaper to dominate than any of the other four algos. One can safely assume the required hash-rate ultimately comes to far less than 10%; some back-of-the-napkin estimates on reddit place it was as low as 0.4%.

So, in sum: timestamp spoofing made it possible to drastically lower mining difficulty; Verge’s use of five algorithms meant that one could lower the difficulty of just one of them, thus making it far easier to override the whole network; the economic/industrial status of this one particular mining algorithm made it even easier to dominate still; and finally, the drastically decreased block-times ensuing from the low difficulty made the attack roughly 30 times more profitable than it would otherwise be.

Lessons Learned

What can be gleaned from all of this? The short-term aftermath of the hack was predictably messy and confusing. After a rough few days, the core developers deployed some quick bug-fixes, which may or may not have had mistakes embedded in them, and the network eventually underwent a hard fork, which may or not have initially been an accident. As for the response in the world at large, in the week following the hack, the price of Verge increased by 30%, and in the following week, it was announced that Verge would be accepted as subscription payment for pornhub.com. Exactly how the largest protocol-level hack of a cryptocurrency in recent memory could preceed said cryptocurrency increasing in price and then announcing a partnership with the most trafficked porn-site on the internet is a question I’m forced to leave open-ended; my personal pet-theory is that it has something to do with the fact that the world makes no sense and human beings are all completely out of their goddamn minds.

But I digress. In the grander scheme of things, there may indeed be some useful take-aways here:

First off, the trick of using timestamps to artificially lower difficulty is one that has actually been known about long enough to get its own name — the “time-warp” exploit. One can find discussions of the attack vector on Bitcoin forums from years ago. In some sense, what made the attack work in Verge is their use of the newer, fancier, Dark Gravity Well difficulty adjusting algorithm, in which difficulty is tweaked with every new block, as opposed to, say Bitcoin, where difficultly is tweaked only every 2016 blocks. While adjusting difficulty in such spaced out, discrete increments may at first glance seem like an odd design decision, this hack makes it clear that it’s actually a security precaution; should there be some way to slightly lower mining difficultly, in Bitcoin, the assailant can only do so once every 2 weeks, rendering the results negligible, compared to Verge, where they can do so once every 30 seconds.

Interestingly, one of the purported benefits of Dark Gravity Wave is specifically that it’s supposed to be immune to the time-warp exploit. Given how decisively such a claim has now been disproven, other currencies using it ought to be, at minimum, a bit nervous.

As for the use of five mining algos: while, from a distance, this appears to be a worthy experiment in economically encouraging decentralization, it again introduces new complexities which inevitably increase the likelihood of unforeseen vulnerabilities emerging.

In both cases, this hack presents a strong argument for tending towards sticking to things proven to work and to be wary of overcomplicating things and thereby introducing unnecessary risks when people’s financial assets are involved. Which, I suppose, means two points for team Bitcoin.

There’s a larger point to be made here: software developers, as much as we’re loathe to admit it, are, in the final analysis, only human. Even when underlying security principles seem perfectly sound, there’s always room for error. Unexpected attack vectors emerge; trade-offs get poorly estimated; and then, of course, there’ll always be good old fashioned bugs. That software doesn’t always work the way we want it to, and that such malfunctions can lead to the loss of funds, shouldn’t be particularly shocking to anyone in 2018. But when that software is, in fact, money, it’s worth an extra layer of precaution.

Given that most of us don’t have the spare time to conduct a thorough code review of every project we invest in, the best guards against disaster are to put more trust in things which a proven track record of working properly, and whose development ethos veers on the conservative side. And should you want to stake some funds in more experimental, move-fast-and-breaks-things-and-raise money style projects, at least be aware of the risks involved. Most importantly, pressure from the community for due diligence in trying to mitigate future disasters is ultimately the best defense. I believe it was George Santayana who said “Those who fail to learn from the security holes of the past are doomed to reimplement them, ” words to heed by lest we unexpectedly find ourselves on the verge of doin’ the time warp again.

Correction: The window of difficulty retargeting was originally reported as being equal to the allowed time drift (2 hours). In fact, the retargeting window is actually 30 minutes, which, for those keeping track at home, is in fact LOWER than the allowed time drift.

Update (5/22): Seems we are, indeed, doin’ the time warp again; the same vulnerability is being exploited again as we speak, though word is that this time the hacker(s) used two mining algos instead of one. Stay tuned!

Update: Read the follow-up post on the second hack here

For inquiries: daniel@theabacus.io