It was great to see the strength of the teams represented in Taipei. Among the participants:

The Ethereum Foundation research team, of course. Much of the thought-leadership in sharding research is coming from Vitalik and Justin Drake, as well as others like Hsiao-Wei Wang, Karl Floersch, and Vlad Zamfir.

The Geth client development team.

The Parity and Web3 foundation teams.

The Trinity (Py-EVM) team, also from the Ethereum Foundation.

A team from Status developing a mobile client in the Nim language.

The Prysmatic Labs team — working on a sharding implementation in Go.

Other individual researchers such as Phil Daian and Leonardo Bautista-Gomez.

The workshop discussion ranged far and wide over the three days.

On the sharding front, we had in-depth discussion of the concepts from the new specification. As an example, in today’s Ethereum network, every node is responsible for three distinct functions: (1) participating in consensus on ordering transactions, (2) executing those transactions to update the state, and (3) making those transactions and the updated state available to the rest of the network (data availability). In a sharded network, these functions could be split among different participants so as to optimize various features of the network as follows:

Proposer nodes could be responsible for assembling transactions into blocks which they propose as the canonical history.

Collator nodes check that the data in the blocks offered by proposers is available and then add them to the shard’s blockchain.

Executor nodes could be responsible for updating clients on the state of the blockchain (e.g. their account balances) on demand. This allows for a kind of “lazy evaluation” in which only calculations related to data that is actually needed are executed, and also perhaps for “alternative execution engines.”

This is quite different from today’s Ethereum Mainnet, but something like this is likely necessary in order to balance efficiency and security in a network where not every node can be a client of every shard. In order to maintain efficiency, the idea is that proposer and executor nodes could remain synced with a small number of shards, but in order to maintain security, the collator nodes (which actually write to the blockchain) can be shuffled between shards quite frequently. This avoids shard-takeovers by a small subset of participants.

We also discussed the various infrastructure needed to make all this work, the shard manager contract, stateless clients, and the peer-to-peer network layer, among other things.

In addition to working on the scalability infrastructure, it was also clear that there is a significant pent-up demand for innovation on Ethereum, unrelated to scalability. Perhaps the implementation of sharding could be a chance to bring in some other big innovations. So we also spent time on other long-standing topics like the replacement of the Ethereum Virtual Machine (the EVM) with eWasm, older topics like account abstraction, and controversial ideas like storage rent.

Berlin, June 2018: Sharding Meets Proof-of-Stake

Many of the concepts discussed in Taipei were very new, and teams continued to evaluate them after the workshop. Over the following weeks, a couple of trends emerged. First, that there were some weaknesses with the specifics of the proposals discussed (we published one critique). Second, there were some very interesting developments on other fronts, most notably in cryptography, that could enable a big refactoring of the sharding model without losing efficiency or security.

With all the new developments to discuss, it was time to hold another sharding workshop. This time, we were kindly hosted in Berlin in June 2018 by the team from Status as part of the client developers’ conference they organized.

We were happy to be unexpectedly joined at the workshop by the Casper FFG (proof-of-stake) team. During the three or four weeks leading up to the event, another huge change to the specification had been proposed: why don’t we build Sharding and Casper together on a common platform?

It was becoming evident that some of the new Sharding design choices had commonalities with the planned Casper FFG work that had been progressing independently (as per the now abandoned EIP-1011). Both require validator deposits (stakes), both rely on access to random numbers, both have fault proofs and slashing mechanisms, both make use of aggregate signatures. In view of these commonalities, it was proposed that both Sharding and Casper be built on a common infrastructure known as the Beacon Chain. An additional advantage would be taking the work of running Casper and Sharding off the existing Mainnet, which might struggle to sustain the extra load.

Discussions in Berlin confirmed that we all agreed that this was a positive and practical approach to getting both projects delivered.

Beyond the project planning, once again, a wide range of new ideas were discussed at the workshop. We had sessions on new cryptographic primitives such as zkSTARKs and alternative hash functions, we discussed proofs-of-custody, and we looked at options for random number generation, with the current front-runner being a RANDAO with a verifiable delay function (VDF).

Today: Towards Ethereum 2.0

So where does all this leave us in August 2018?

I hope you get a sense from the above that these last six months have seen an explosion in research into scaling Ethereum, and to a large extent the dust has yet to settle.

But the general direction is clear. Development and delivery of both Sharding and Proof of Stake will take place on a new blockchain platform (Ethereum 2.0), coupled back to the current Main Chain which will continue to run as-is.