Intro

The purpose of this writeup is to describe a proposed new Bitcoin Cash (BCH) difficulty adjustment algorithm (DAA) known as “wtema”, which stands for “Weighted Target Exponential Moving Average”. An implementation of wtema is available in Kyuupichan’s difficulty adjustment algorithm simulator [1] .

This DAA is a candidate for possible inclusion in the May 15 Bitcoin Cash protocol upgrade [2] . Whether it will be included or not is undecided. This article meant to provide information that may be helpful in deciding whether it should be included.

Some general considerations and requirements for designing a difficulty adjustment algorithm will first be presented, then the history of Bitcoin Cash difficulty algorithms will be discussed, and finally the specifics of wtema will be presented.

Requirements

As explained in a previous Medium article on the Bitcoin Cash difficulty adjustment algorithm [3], there is no perfect theoretical solution to the problem of difficulty targeting when two coins share the same proof of work (POW). This is because one of the two coins will always be at least slightly more profitable to mine, so there is an incentive for 100% of the hash rate to mine only one of the chains at any point in time. In practice, however, experience has shown that with a sufficiently responsive difficulty target, the profitability will be “close enough” for the hash rate to reach an equilibrium mining both coins.

Some of the general goals of a difficulty adjustment algorithm are:

Adjust difficulty to hash rate such that the mean block interval over time averages to the target (600 seconds for Bitcoin Cash).

Avoid sudden changes in difficulty when hash rate is fairly stable.

Adjust difficulty rapidly when hash rate changes rapidly.

Avoid oscillations from feedback between hash rate and difficulty.

Be resilient to attacks such a timestamp manipulation.

Be simple enough to easily implement across multiple independent codebases.

Some of these goals lead to competing incompatible requirements. The main tradeoff involved is between responsiveness to changes in hash rate, and susceptibility to random fluctuations.

Since the requirements consist of tradeoffs between different goals, it is difficult and maybe impossible to specify a set of well-defined requirements to achieve the design goals. Instead of attempting to specify requirements, the following approaches may be more appropriate:

Provide clear analysis of the desired properties, and challenges in achieving them, so that the tradeoffs can be well understood. In particular, it is important to understand the economic incentives. An imperfect system with aligned incentives can achieve stability over time, whereas a system with misaligned incentives is likely to deteriorate over time. Look at real-life examples of different DAAs, in order to gain an appreciation for what is important in practice. Use simulation to test DAAs under a variety of circumstances and threat scenarios, some of which can be more extreme than scenarios that have been historically observed.

Selfish Mining and Greedy Mining

The tradeoff in stable difficulty target versus fast adjustment manifests itself as a tradeoff between two incentive problems: Selfish mining and Greedy mining.

Greedy mining refers to the situation when two coins share the same proof-of-work. “Greedy miners” are those miners who rapidly switch to mine the most profitable coin at any moment. In contrast, “benevolent miners” are miners who continue to mine the coin even when it is less profitable at a particular moment. If 100% of miners on a minority chain are “greedy”, then the chain will likely die, as it will stop being extended as soon as price fluctuations make it less profitable.

A good illustration of the need for benevolent mining was provided in the early days of Bitcoin Cash, when it still used the Bitcoin fixed-period difficulty adjustment algorithm, with the added Emergency Difficulty Adjustment (EDA). Immediately after the Aug 1st fork, BCH and BTC shared the same difficulty, even though the BCH price was much lower. Luckily, ViaBTC and an anonymous “benevolent miner” mined enough blocks at a relative loss in order for the EDA to be triggered, and Bitcoin Cash survived.

The smooth operation, and maybe even the survival, of either chain will depend on the existence of Benevolent miners. This is especially true for the minority chain, which is in a more vulnerable position compared to the larger hash rate of the more valuable coin. The difference in revenue between Greedy miners and Benevolent miners should be considered a type of security vulnerability, since it punishes the more desirable behavior. A difficulty adjustment algorithm that minimizes this difference is desirable.

After the Nov 13th BCH protocol upgrade, a new, more responsive, difficulty adjustment algorithm took effect [4] . This new DAA greatly reduced the revenue difference between Greedy miners and Benevolent miners, and thus greatly lowered the systemic risk of chain death for the BCH chain.

“Selfish mining” describes another vulnerability, which, confusingly, is completely unrelated to “Greedy mining”. Selfish mining was described in a paper by Emin Gün Sirer [5] , and is well explained in a Bitcoin Magazine article by Vitalik Buterin [6] . The Selfish mining attack describes a way that a cabal of miners could theoretically collude to increase the orphan rate of competing miners, thereby increasing the cabal’s proportion of mined blocks. One of the criticisms leveled at the Selfish mining theory is that although it increases relative revenue, it does not increase absolute revenue until the difficulty adjusts downward due to the increased orphan rate. This means that a difficulty adjustment algorithm that adjusts more rapidly could theoretically increase the feasibility of Selfish mining. It is unclear whether Selfish mining is a significant problem or not. It is thought to be easily detectable, but there is no recorded case of it ever occurring.

Timestamp Manipulation

All major difficulty adjustment algorithms currently in use across various altcoins use timestamps to calculate the difficulty target. Since it is impossible for the algorithm to “know” the real time, it must rely on the timestamp in the block header. This timestamp is just a number that is meant to correspond to real time, but this correspondance can only be enforced loosely on the network, for various reasons.

Since the market incentives enforcing timestamp accuracy are not very strong, it is important that the DAA have the following properties:

It should be somewhat resilient to inaccurate timestamps, including occasional “outlier” timestamps. It should not create incentives for timestamps to have particular values. An example of this would be an algorithm that adjusted the reward according to the inter-block time, which would create an incentive to skew timestamps to higher values.

Requirement (1) can be handled in two ways. The first way would be to reject blocks with “outlier” timestamps when doing the calculations. For example, the current DAA uses a “median-of-three” method to pick the endpoints of the sample period, thereby helping to avoid basing the calculation on outlier timestamps.

The second way to to mitigate the effect of inaccurate timestamps is to base the difficulty of the timestamps of many blocks, with each particular timestamp having only a small effect on the final result. This was the approach taken by dgenr8’s formerly-proposed “wt-144” algorithm [7] , and the approach taken by the wtema algorithm.

Motivation and Background

The current DAA, in effect since Nov 13, 2017, is working well. It seems that the 144-block sample that it uses is a good size. This sample size makes it responsive enough so that large discrepancies in mining profitability do not persist for long, and stable enough that random “luck” in mining does not cause the difficulty to jump around too much. The fact that the algorithm is based on a sound theory of estimating hash rate based on chain work means that it is good at accurately targeting the 10-minute block interval.

The main problem with the current DAA is that since it is based on a uniform fixed sample, it is sensitive to the beginning and end points. In Kyuupichan’s Python simulation, it is called “cw-144”, since the calculation uses “Continuous Weight” over the range of the sample. Being sensitive to the end point may make sense, since the most recent block can be expected to have more recent information about hash rate. But the fact that the calculation is sensitive to the beginning point 144 blocks from the chain tip is unfortunate. This means that unusually fast or slow blocks will not only have an immediate impact when they are mined, but will also have an “echo” effect on the difficulty calculation as they leave the sample 144 blocks later.

These “echo” effects can lead to difficulty deviating from accurately representing the most recent hash rate, which ends up costing benevolent miners more money than a more accurate target would.

This “144-block echo” problem was predicted in the discussion and modeling leading up to the implementation of the DAA, and was one of the motivations for different approaches. Kyuupichan’s “k-1” algorithm avoided the oscillation problem by limiting the target changes to small “nudges” up or down based on a small 6-block sample [8] . Since k-1 was based on “nudges”, it does not have a satisfying theoretical guarantee to target the 10-minute block interval, and instead relied tuning several parameters to aim at the target.

Dgenr8’s “wt-144” algorithm used a similar fundamental method as cw-144, using chain work and time to calculate a target based on estimated hash rate. The difference is that instead of calculating it as a single sample over the entire range, wt-144 calculated it for each inter-block period. These individual calculations were then be summed over the sample range. This algorithm removed the sensitivity to the beginning of the sample range by adding a linear weighting to the sample period, so that the effect of early blocks faded away gradually, rather than having a hard cutoff. One of the downsides of wt-144 was that the need to loop and sum over the 144-block sample seemed inelegant.

The wtema algorithm combines elements of all of these algorithms, converging to a design that is simple, theoretically robust, and which outperforms all the alternatives in simulation.

Implementation

Conceptually wtema is very similar wt-144. It calculates targets based on each individual inter-block period, and sums them with a weighting applied. Differences from wt-144 are:

Instead of a linear weighting increment, wtema applies an exponentially decreasing weight to each inter-block period going back from the most recent. There is no beginning to the sample period. The algorithm sums over a set of inter-block periods going back indefinitely.

That sounds unwieldy, how is the algorithm supposed to sum over a set without a starting point? The trick lies in a property of exponential moving averages: that they can be calculated recursively [9] . The way this works is that as elements are added to the sum, you simply scale the prior sum up to that point to represent all prior elements.

For the wtema difficulty target calculation, this formula is:

next_target = SBT * α + prior_target * (1 - α)

Where SBT is the single-block-target based on the last block interval, prior-target is the target of the prior block (which is the exponential average up to that point), and α is a parameter set to 1/144.

The value of 1/144 chosen for parameter α in wtema corresponds to a “half-life” of 100 blocks. This value was chosen through simulation and observation of real-life network conditions in the period after Nov 13, 2017 since the new DAA was implemented. It seems to provide a low variance in target values, while still being responsive enough to hash rate changes.

SBT is calculated based on estimated hash rate, as represented by work done, and block time. It is calculated by the formula:

SBT = prior_target * block_time / ideal_block_time

Where block_time is the inter-block time according to the block timestamps, and ideal_block_time is the targeted time of 600 seconds.

Since prior-target corresponds to the amount of work that needs to be done to solve the block, that means that prior_target * block_time should correspond to the estimated hash rate of the network. Then, reversing the process, we divide this estimated hash rate by the desired block time to yield the difficulty target that should correspond to producing blocks at the desired intervals.

The formula for SBT can be substituted in to yield:

next_target = (prior_target * α / ideal_block_time ) * (block_time + (1 / α - 1) * ideal_block_time)

For the actual calculation, the reciprocal of α is used, and the calculations are done division first, so that integer math can be used while maintaining precision, and avoiding overflows with large targets in regtest.

Specification

Let B_n be the Nth block in a Bitcoin Cash Blockchain. Let “block_time” be equal to the difference in UNIX timestamps (in seconds) between B_n and B_n-1 Let “prior_target” be the difficulty target of B_n Calculate an interim value for next_target as the integer result of prior_target divided by (600 * 144) Calculate next_target as the interim value multiplied by (block_time + (600 * 143)) Let max_change be prior_target with right bit-shift of 3 If next_target is greater than (prior_target + max_change), then let next_target equal prior_target + max_change If next_target is less than (prior_target - max_change), then let next_target equal prior_target - max_change The target difficulty for block B_n+1 is then equal to the lesser of next_target and 0x00000000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

Conclusion

The wtema difficulty algorithm is theoretically sound, robust in simulation, and simple to implement. Based on the study and investigation that has been done, it seems to outperform the many other options that have been considered, and to outperform any existing difficulty algorithm used by altcoins. It would reduce the revenue disadvantage of Benevolent mining versus Greedy Mining. The current Bitcoin Cash DAA seems to be performing adequately under current network conditions, but we would expect wtema to perform better under harsher conditions that may or may not occur. The main question to consider is whether the benefits of wtema are worth the disruption of changing difficulty algorithm again.

Appendix - Algorithm Comparison

cw-144: target calculated based on estimate of hash rate calculated from work done and time measured in a single sample over a ~144 block interval. This is the current DAA of the Bitcoin Cash network.

k-1: Does not calculate a hash-rate estimate, uses small “nudges” to adjust target toward a desired inter-block interval.

wt-144: Based on hash-rate calculated from work done and time for each inter-block period. These estimates are then weighted by inter-block time, and combined according to a linear weighting over 144 blocks. This makes the algorithm have to loop and sum values over the sample range.

wtema: Combines most of the best qualities of the previous algorithms. Calculation is based on chain work and time for each inter-block interval, which means it accurately targets the 10 minute interval over long time periods. The values are summed over blocks extending back indefinitely. The calculation is simplified due to a property of exponential sums, resulting is a calculation resembling the k-1 “nudge”, where the size of the nudge is variable.

References