The following is hopefully a technical and approachable write-up of some exciting progress made by CoZ and NEO teams on Friday 25th August to resolve some odd behaviour on the NEO Blockchain.

Introduction

This post assumes some familiarity with NEO, its ecosystem and its DBFT Blockchain design, but not too much. Where possible, I will state the relevant design assumptions for the Blockchain and explain why it exists and any issues.

If this post starts to get too technical, and you just want to know a summary, please check out the reddit post: https://www.reddit.com/r/NEO/comments/9a4y0v/moving_forward_spam_attacks_and_improvements/

Setting the scene

NEO and Ethereum are the two leading smart contract platforms if you look at the size of the network, number of tokens, size of development eco-system etc. But NEO has one key advantage over Ethereum, which is speed.

NEO has an expected block time of 10–20 seconds and that is a final confirmation time too. Because NEO uses DBFT for blocks rather than Proof of Work mining or Proof of stake, every signed block produced by Consensus nodes is the finished article, no forks etc. Ethereum has a similar block time but because of their consensus algorithm, you must wait 3–6 blocks before being confident that the block will be accepted into the master chain.

This gives a 3–6x speed-up in basic terms.

It is worth noting at this point that a bug in NEO’s code caused an extremely rare fake forking event on 16th August that caused some mayhem but was fixed the same day. Forking is not by design, that was purely a bug.

The other interesting thing about NEO is that it never mandated any fees for transactions or smart contract calls (known as InvocationTransactions). Instead, the idea was that paying fees would give you higher priority in the case where the Blockchain needed to pick a subset of transactions. But generally, all transactions could get through without any problems.

Introducing SPAM

Free transactions are great until someone decides to take advantage of your generosity. A SPAM attack involves creating thousands of transactions and flooding the network with valid transactions (usually tiny transfers) and thus making it hard to distinguish the normal user free transactions from the attackers.

Of course, the NEO team had already thought of this as both an eventuality and real threat.

NEO’s defence design

NEO created a defence design (a set of parameters and heuristics) to try and mitigate the effectiveness of a SPAM attack. I’m going to show you what they are (as far as I know), and also where they are in the Github repo. One of my goals with this article is to show you that the core NEO library is more approachable than people think and you too can get an approved Pull Request which is awesome.

#1 Only allow 20 Free Transactions per block (21 including Miner)

this.MaxFreeTransactionsPerBlock = GetValueOrDefault(section.GetSection(“MaxFreeTransactionsPerBlock”), 20, p => int.Parse(p));

MaxFreeTransactionsPerBlock is a heuristic that limits each Block to only have a certain number (20 currently) of free transactions. That means that a SPAM attacker can only get 20 transactions into the block. Whilst many people have pointed out this is a rather simplistic defence, it should be effective.

#2 Only allow 500 transactions per block (499 + 1 Miner)

this.MaxTransactionsPerBlock = GetValueOrDefault(section.GetSection("MaxTransactionsPerBlock"), 500, p => int.Parse(p));

MaxTransactionsPerBlock limits each block to a certain number of transactions (currently 500 including the MinerTransaction). This is to ensure blocks have a high chance of being processed quickly (imagine a huge block that takes so long to produce and propagate that it slows the block time down).

#3 Only keep at most 50,000 candidate transactions in the Mempool

public const int MemoryPoolSize = 50000;

The Memory Pool is a pool of transactions that are submitted for including in future blocks. To further ensure that a very large SPAM attack doesn’t cause problems for nodes (remember, nodes are just computers — they have limited memory, CPU and space like everyone else), the design limits the number to 50K transactions.

#4 Prefer transactions with the highest NetworkFee to TransactionSize ratio

This is a design heuristic that appears in a number of places and also is a common prioritisation metric for other blockchains too. The Network fee is the fee that you can add to your transaction. The transaction size is the number of bytes it takes to represent your transaction on the blockchain. Remember, the brilliance of Bitcoin and blockchain is that sending 0.0001 BTC and 1000 BTC is just an array of bytes (of very similar length) so the cost of doing so in terms of memory and processing is almost identical.

As a result, the NEO system aims to prioritise transactions that pay the most Fee per byte used. Obviously, this design still means any fee > free (remember this for later!).

#5 If there are more than 50K transactions in the Mempool, keep the ones with the highest Fee/Size score, and discard the rest

This only happens if there are 50K transactions in the Mempool, which should be rare currently except under attack. In the future, with heavy normal usage, we may need to adjust this threshold.

private static void CheckMemPool()

{

if (mem_pool.Count <= MemoryPoolSize) return;

UInt256[] hashes = mem_pool.Values.AsParallel()

.OrderBy(p => p.NetworkFee / p.Size)

.ThenBy(p => new BigInteger(p.Hash.ToArray()))

.Take(mem_pool.Count - MemoryPoolSize)

.Select(p => p.Hash)

.ToArray(); foreach (UInt256 hash in hashes)

mem_pool.Remove(hash);

}

The key thing in the code fragment is

OrderBy(p => p.NetworkFee / p.Size)

#6 If there are more than 500 Candidate Transactions in the Mempool, then pick the ones with the highest Fee/Size ratio to go into the next block

transactions = array.OrderByDescending(p => p.NetworkFee / p.Size).Take(Settings.Default.MaxTransactionsPerBlock - 1);

Rank by Fee/Size and then take the top 499 of them (plus Miner Transaction).

This makes sense, you want the best transactions in the next block.

#7 Of the top 499 transactions, accept all fee transactions but block free transactions after 20

private IEnumerable<Transaction> FilterFree(IEnumerable<Transaction> transactions)

{

int count = 0;

foreach (Transaction tx in transactions)

if (tx.NetworkFee > Fixed8.Zero || tx.SystemFee > Fixed8.Zero)

yield return tx;

else if (count++ < Settings.Default.MaxFreeTransactionsPerBlock)

yield return tx;

}

Conclusion

If you made it all the way here, congrats. Hopefully you agree with me that the Defence design makes sense so far. Now we turn to the subtle bug that derailed it all..

Integer Division — the bane of our lives

Everything was going well with using Integers (specifically BigInteger and Fixed8) until we tried to do division. C# handles integer division like most languages, by ignoring the remainder. Hence, 1/3 in integer division = 0.

Code snippet from developer Belane illustrating this

So now we have an issue. Our assumption was that we were ranking transactions by Fee/Size, but even for transactions with a fee, this calculation can equal 0 (hence free and fee look the same during ranking!).

This meant that our design assumptions #5 and #6 were not being handled correctly. Instead, the SPAM attack transactions were making it into the Top 499 ranking because the code incorrectly treated them the same as Fee transactions.

The consequence of this is that when it came to #7 (picking 20 free TX from the 499 and the rest are paid), nearly all of the 499 were free transactions.

That is why we ended up with so many blocks with just 21 transactions (when we know there are lots more fee transactions out there).

NEOScan screenshot showing many successive blocks with close to 21 transactions each

Since #5, #6 and #7 were not behaving as expected, the following code also caused an issue:

else

yield break;

This simple two line code assumed that the list that was passed into the FilterFree method was already ranked by Fee and then by Free transactions. So if the count of Free transactions exceeded 20, then the rest of them must be Free and rather than processing, you can simply end early.

However, since the ranking was incorrect, the list was actually ‘random’, so Fee transactions that made it into the Top 499 were still sometimes being ignored because of this break clause.

Speedy resolutions and final reflections

Once we had started to understand exactly what was going on, the CoZ community and NEO team (Erikzhang, in particular), acted fast to update, review and approve changes.

A tale of three fixes

A) Hal0x2328’s Break fix

First up, Hal0x2328 made the first fix which was approved by Erikzhang within 4 minutes! This removed the “else yield break” part of the method.

B) Belane and Jseagrave21’s Ranking Fix

The key breakthrough came when CoZ members figured out how to use secondary rank in C#:

To secondary rank transactions by Network Fees. So if Integer(Fee/Size) > 0 then those are prioritised. But if Integer(Fee/Size) = 0 but Fee > 0, then those would still be prioritised ahead of Free transactions.

C) Applying the same idea to Mempool

Using the same idea, applied the secondary ranking concept to Mempool when deciding which transactions to discard above 50K.

Tying this all up, it was essentially three commits and less than 30 characters.

It boils down to:

.ThenByDescending(p => p.NetworkFee)

I was personally very impressed with the speed at which Erik and the NEO team reviewed, approved, tested and then merged our submissions into the NEO repository. This was a great example of collaboration between NEO and CoZ.

Since the code affects the core NEO library (neo-project/neo) and the consensus plugins (neo-project/neo-plugins), a new release will need to be generated and rolled out as soon as possible.

I look forward to seeing the network go from strength to strength.

#SmartEconomy

Final Notes

Special thanks to Fabian (fabwa) who encouraged me to stick with it even when the situation seemed to get tougher and tougher, and also working with Joshua Seagrave (jseagrave21) has been great fun.

Pull Requests: