We’ve seen many people asking questions about what’s going on with the EOS Mainnet in the last few days. Why are there so many forks and missed blocks?

First, we want to provide a bit of background. Every transaction on the EOS Mainnet takes CPU to process, and the total amount of CPU consumed for all transactions has been steadily increasing. This increase in the total amount of CPU consumption comes from higher usage of network resources.

As the CPU need increases, block producing infrastructure providers need to make changes to accommodate accordingly. For EOS Nation, that means buying more hardware and then splitting the workload across more machines. Usually our users don’t notice anything – we just balance the increased load across more servers.

Although it’s unclear at this time what exactly triggered the problem, it’s safe to say that the seemingly ever increasing CPU usage was a contributor. Not all of the infrastructure was fast enough to deal with the changes, and some block producers couldn’t keep up with the load.

Server CPU is consumed every time a node exchanges blocks with another node. The more nodes you exchange blocks with, the more CPU it takes. If you peer with fewer nodes, you save on CPU. This seems like a straightforward fix.

However, as an active block producer, when you produce a block you want to get it to everyone else who needs it as quickly as possible. So you want to peer with at least the 3 block producers in the schedule before you and the 3 after you. This also includes peering your API nodes needed to service your applications as well as peering the P2P endpoint you provide to the public.

The solution has mostly been optimizing connections between block producers to be as efficient as possible. Each block producer is adjusting to their own unique situation: for example, some have modified various EOSIO settings, others have upgraded their EOSIO binaries to the latest 1.8.9 release and some have done both.

A block producer cannot simply optimize their own infrastructure and be done with it. All block producers need to work together to come to a common solution. Over the last week or so, block producers from around the world have been working together to improve the situation. Block.one has also been very responsive, communicating with us on a daily basis and looking over our logs in order to hopefully gain valuable insights from the data.

Significant progress has been made to reduce the number of micro-forks and missed blocks and block producers continue to work to improve things.