Technology

Consensus mechanism: dBFT

Byzantine Fault Tolerance mechanism is a universal solution for distributed systems. Neo proposes dBFT (delegated Byzantine Fault Tolerance) consensus algorithm based on pBFT (Practical Byzantine Fault Tolerance) algorithm. dBFT algorithm determines validator set according to real-time blockchain voting, which effectively enhances the effectiveness of the algorithm, bringing block time and transaction confirmation time savings. dBFT2.0 as an upgraded version was released in March 2019, which improves robustness and safety by introducing 3-stage consensus as well as a recovery mechanism.

System Model

There are two types of Neo nodes in the network: one is the bookkeeping node, which is responsible for the consensus communication with other bookkeeping node to generate new blocks; the other is the ordinary node, which does not participate in the consensus, it can transfer and make transactions. The bookkeeping nodes are generated through voting by the entire network users.

In Neo, a relatively small number of bookkeeping nodes are selected through voting to perform PBFT consensus and generate new blocks, and then the new blocks are released to the whole network to reach a consensus across the network.

Consensus (bookkeeping) Node: This node participates in the consensus activity, make a block proposal and vote. Ordinary Node: This node, but does not participate in the consensus activity. Speaker (Unique in each round): The Speaker is responsible to create and transmit a proposal block to the system. Delegates (Multiple): Delegates are responsible for voting on the proposal block. The proposal will be accepted if more than 2f+1 consensus nodes vote it. f is the limit of Byzantine nodes. Validator candidate: Nodes participating in elections. Candidate nodes for consensus activity. View: The dataset used during one consensus activity. The view number start from 0 in each round, and the number will increase when it failed to reach consensus in a round. Source

The Algorithm

Neo algorithm ensures security as well as usability. With erroneous nodes in the consensus making no more than ⌊ (N−1) / 3 ⌋ , the functionality and stability of the system is guaranteed. In it, N = |𝑅| suggests the total number of nodes joined in the consensus making while R stands for the set of consensus nodes. Given F = ⌊ (N−1) / 3 ⌋ , f stands for the maximum number of erroneous nodes allowed in the system. In fact, the total ledger is maintained by bookkeeping nodes while ordinary nodes do not participate in the consensus making. This is to show the entire consensus making procedures.

All consensus nodes are required to maintain a state table to record current consensus status. The data set used for a consensus from its beginning to its end is called a View. If consensus cannot be reached within the current View, a View Change will be required. Neo identifies each View with a number v, starting from 0 and it may increase till achieving the consensus.

Neo identifies each consensus node with a number, starting from 0, the last node is numbered N − 1. For each round of consensus making, a node will play speaker of the house while other nodes play congressmen. The speaker’s number p will be determined by the following algorithm: Hypothetically the current block height is h, then 𝑝 = (ℎ − 𝑣) 𝑚𝑜𝑑 N, p’s value range will be 0 ≤ 𝑝 < N .

A new block will be generated with each round of consensus, with at least N − F signatures from bookkeeping nodes. Upon the generation of a block, a new round of consensus making shall begin, resetting v=0.

General Procedures

A round of consensus consists of 4 steps, as shown in the figure above.

Speaker starts consensus by broadcasting a Prepare Request message,

Delegates broadcast Prepare Response after receiving the Prepare Request message,

Validators broadcast Commit after receiving enough Prepare Response messages,

Validators produce and broadcast a new block after receiving enough Commit messages.

View Change

In case of the following scenarios, the Change View Request will be broadcasted attempting to replace speaker:

The transaction verification fails

Time is out while waiting for Prepare Request or Prepare Response

Recovery Mechanism

When creating Change View Request, if there are not enough active consensus nodes (sum of nodes with Commit sent and fault nodes is greater than F), consensus nodes will broadcast Recovery Request message to update the local consensus context. Upon receiving Recovery Request, if certain conditions are met, a consensus node will generate and broadcast Recovery Message.

Fault Tolerance of dBFT2.0

A dBFT2.0 consensus system with N validators can tolerate at most F abnormal nodes. Each consensus phase (Commit, Change View, block generation, etc.) requires at least M nodes to reach consensus [M = N-F]. As long as the amount of normal validators is not less than M, the consensus process will go on smoothly. For example, just 4 − ⌊ (4−1) / 3 ⌋ =3 normal validators required can keep alive a consensus system where N= 4.

Single Block Finality of dBFT2.0

Neo’s dBFT 1.0 algorithm was susceptible to a single block fork in rare cases of network latency. dBFT2.0 fixes this problem, hence there is no possibility of forking since then. The mechanism is described as follows:

To generate a new block, it is required to collect Commit messages from at least M different validators for corresponding block proposal.

A validator will never change its view after broadcasting Commit message.

Hence the success of block generation means:

There are already at least M validators having signed the block proposal and broadcast Commit messages. Moreover, these validators won’t change the view in current consensus round.

The rest of the validators are insufficient to produce another different block.

Therefore, the finality of the new block can be guaranteed at a given height.

Source

Smart contract system: NeoContract

Neo’s smart contract system consists of three parts:

1. NeoVM — Universal Block Chain Virtual Machine:

NeoVM is a lightweight, general-purpose virtual machine whose architecture is very close to the JVM and .NET Runtime, similar to a virtual CPU that reads and executes instructions in the contract in sequence, performs process control based on the functionality of the instruction operations, logic operations and so on. It has a good start-up speed and versatility, is very suitable for small programs such as smart contracts, can also be ported to non-blockchain systems, or integrated with the IDE to provide an optimal development experience. NeoVM’s functionality can be extended, like introducing a JIT (real-time compiler) mechanism, thereby enhancing the efficiency of the implementation.

2. InteropService — Interoperable Services:

Used to load the blockchain ledger, digital assets, digital identity, persistent storage area, the file storage system of NEO (NeoFS), and other underlying services. They are like virtual machines that are provided for virtual machines, enabling smart contracts to access these services at run time to achieve some advanced functionality. Through this low-coupling design, NeoVM can be ported to any blockchain or even non-blockchain system used, increasing the utility of the smart contracts.

3. DevPack — Compiler and IDE plugin:

DevPack includes the high-level language compiler and the IDE plug-in. Because NeoVM’s architecture is very similar to JVM and .NET Runtime, the compilers in DevPack can compile Java byte code and .NET MSIL into NeoVM’s instruction set. Java / Kotlin, C# developers do not need to learn new languages and will be able to immediately start developing smart contracts in VS, Eclipse and other familiar IDE environments. This greatly reduces the learning curve for developing smart contracts, allowing us to easily build a vibrant community around NeoContract.

NeoContract can create a smart contract call tree through static analysis before running a smart contract. Through the deterministic call tree, the Neo node can dynamically fragment the smart contract to achieve theoretically unlimited expansion , which overcomes the “jamming effect” caused by the static fragmentation of other block chain systems.

Cross-chain interoperability agreement: NeoX

NeoX is a protocol that implements cross-chain interoperability. NeoX is divided into two parts: “cross-chain assets exchange protocol” and “cross-chain distributed transaction protocol.”

Cross-chain assets exchange agreement:

NeoX has been extended on existing double-stranded atomic assets exchange protocols to allow multiple participants to exchange assets across different chains and to ensure that all steps in the entire transaction process succeed or fail together. In order to achieve this function, Neo need to use NeoContract function to create a contract account for each participant. If other blockchains are not compatible with NeoContract, they can be compatible with NeoX as long as they can provide simple smart contract functionality.

Cross-chain distributed transaction protocol:

Cross-chain distributed transactions mean that multiple steps of a transaction are scattered across different blockchains and that the consistency of the entire transaction is ensured. This is an extension of cross-chain assets exchange, extending the behavior of assets exchange into arbitrary behavior. In layman’s terms, NeoX makes it possible for cross-chain smart contracts where a smart contract can perform different parts on multiple chains, either succeeding or reverting as a whole. This gives excellent possibilities for cross-chain collaborations and we are exploring cross-chain smart contract application scenarios.

Distributed Storage Protocol: NeoFS

NeoFS is a distributed storage protocol that utilizes Distributed Hash Table (DHT) technology. NeoFS indexes the data through file content (Hash) rather than file path (URI). Large files will be divided into fixed-size data blocks that are distributed and stored in many different nodes.

The main problem with this type of system is the need to find a balance between redundancy and reliability. NeoFS plans to solve this contradiction by means of token incentives and the establishment of backbone nodes. Users can choose the reliability requirements of the file. Files with low reliability requirements can be stored and accessed for free or almost free. Stable and reliable services for files with high reliability requirement will be provided by backbone nodes.

NeoFS will serve as one of the InteropService interoperability services under the NeoContract system, enabling smart contracts to store large files on the blockchain and set access for those files. In addition, NeoFS can be combined with digital identity so that digital certificates used by digital identities can be assigned, sent, and revoked without a central server to manage them. In the future, the old block data can be stored in NeoFS, so that most of the full nodes can release the old data for better scalability and at the same time, ensure the integrity of historical data.

Anti-quantum cryptography mechanism: NeoQS

The emergence of quantum computers poses a major challenge to RSA and ECC-based cryptographic mechanisms. Quantum computers can solve the large number of decomposition problems (which RSA relies on) and the elliptic curve discrete logarithm (which ECC relies on) in a very short time. NeoQS (Quantum Safe) is a lattice-based cryptographic mechanism. At present, quantum computers do not have the ability to quickly solve the Shortest Vector Problem (SVP) and the Closest Vector Problem (CVP), which is considered to be the most reliable algorithm for resisting quantum computers.

Source

NEO3

The cornerstone of the team’s efforts is NEO 3.0, which will be a robust blockchain implementation with high throughput, enhanced stability and security, an optimized smart contract system, and a feature-packed infrastructure set for diverse business application scenarios.

On the flip side, the team keenly recognizes governance’s pivotal role in the long-term evolution of a blockchain as the common good collectively owned by stakeholders and more broadly the entire surrounding communities. In 2019, they are actively collaborating with experts from academia, industry, and community to explore various governance mechanisms including liquid democracy, futarchy and some others that emerged in recent times. In many cases the economic model is tightly interlaced with governance mechanism, therefore they will be treated as an integrated system. NEPs regarding on-chain governance changes will be published if satisfying outcomes are achieved after extensive research and simulation.

New features in NEO 3.0

a) dBFT 2.0

In dBFT 2.0, the team added a recovery mechanism that greatly improved the stability of the consensus algorithm. In the rare occurrence of a network failure or a node failure, a quick recovery is expected.

The development of dBFT 2.0 began in Q4 of 2018 and was completed in Q1 of 2019. It will soon be deployed to the main network of NEO 2.x.

b) Pricing Model

There are two native tokens on the Neo blockchain, namely neo and gas. Gas is used to pay transaction fees and smart contract execution fees.

Currently, the relatively high cost of deploying and running smart contracts leads to a reluctance in smart contract usage and development. The current pricing model becomes a significant obstacle in the gas application scenarios, and thus hinders the continuous growth of DApp development and usage on Neo platform.

In NEO 3.0, the team will address this issue by significantly reducing the deployment and execution costs of smart contracts, thereby expanding the application scenarios of gas and increasing the number of DApps. Prior to the NEO 3.0 implementation, credible projects can apply for grants from the Neo Foundation with contract deployment costs.

c) Internet Resource Access

NEO 3.0 will have a built-in oracle implementation that allows smart contracts to access Internet resources during execution. Inconsistencies between nodes when accessing Internet resources can be resolved thanks to the security and efficiency of dBFT 2.0.

With this feature, developers can easily develop more sophisticated or scenario-specific oracles based on Neo, and develop more diverse DApps that rely on external data.

d) P2P Protocol

In NEO 3.0, the team will redesign the P2P protocol, add support for the UDP communications protocol, and enable compression options. This is expected to greatly improve the TPS and stability of the network.

e) NeoVM

In NEO 3.0, NeoVM will be completely decoupled from the blockchain and become a pure virtual machine. There are several benefits to this:

- Easy implementation of native contracts.

- Application scenarios of NeoVM outside the blockchain.

- Smooth Integration of NeoVM into any IDE and easy debugging of smart contracts without loading blockchain data.

In addition, NeoVM will also include some new features, such as support for static members, exception handling, and more.

f) Simplified Architecture

Currently, there are two methods to create assets on Neo. The first is to create a global asset with RegisterTransaction, and the second is to create a contract asset with a smart contract. In practice, global assets are rarely used, and most applications create contract assets due to their flexibility and functionality. Since global assets are not integrated with smart contracts, managing global assets in smart contracts are very difficult.

For these reasons, the team doesn’t need to continue to support global assets in NEO 3.0. In NEO 3.0, all assets are created in smart contracts, including neo and gas.

By removing global assets, it becomes possible to unify all transaction types. Currently, in NEO 2.x, there are 9 different transaction types. These transaction types are either related to a particular application scenario or provide more niche functionality. For example, RegisterTransaction and IssueTransaction are related to the creation and distribution of global assets. Since global assets will no longer exist, these related transactions are rendered obsolete.

Other transactions dealing with complex application scenarios will also be removed and replaced with interop services in smart contracts. As a result, there is only a need for a single transaction type in NEO 3.0, which is used to execute smart contracts.

The team has also proposed a simplified validation model that will greatly improve the speed of transaction verification, and allow these validations to be performed concurrently.

With these architectural adjustments, the performance of NEO 3.0 base layer will be substantially increased by orders of magnitude. However, this new architecture can lead to incompatibility with NEO 2.x. In order to minimize the impact of this incompatibility, the team plans to delay the development of any incompatible features until all NEO 2.x compatible features have been developed.

g) NeoFS

NeoFS is intended to be primarily used by DApps for data storage and as a Content Delivery Network. Moreover, NeoFS can be used to create private distributed storage systems for SMEs, which use regular servers or clusters (data centers), and for storing large amounts of unstructured IoT data.

The use of smart contracts is proposed to control the distribution of rewards from data owners and publishers to participants that are hosting data. The Neo protocol can be extended for deeper integration with NeoFS, allowing storage on data nodes instead of blockchain ledger. A topic of further research is the potential of decreasing the cost of smart contract deployment along with storing data and files to be used by smart contracts. In addition, NeoFS could be used to store old block data instead of full nodes, further increasing the scalability of Neo.

NeoFS contains a scalable data placement method. Fine control over object location and minimal data movement in case of storage node failures are achieved by using a subset of a network map and storage policy rules for object placement, along with Rendezvous hashing for node selection.

In addition, the proposed platform uses a novel zero-knowledge data validation method based on homomorphic hashing to minimize data transfers. This helps to maintain network scalability by minimizing computational costs on storage node and validation nodes, and ensures a large number of parallel interactions.

h) NeoID

Digital identity refers to the identity information of individuals, organizations, and other entities that exist in electronic form. Blockchain brings a new way to define identities and the relationships between them.

NeoID is a decentralized identity protocol built on Neo. It empowers users and organizations to have better control of their identities and delivers a higher degree of trust and security to the smart economy.

It consists of three main parts: Trust Model, Privacy Model and Game Model. The Trust Model describes the rules of trust in this distributed network. The Privacy Model describes the privacy protection scheme for users’ online data. The Game Model describes the benefits and penalties of actions within the trust network. These three parts provide a mathematical model to abstract the real world, forming the basis of NeoID.

NeoID will not only support a decentralized identifier issuance model but will also be compatible with the X.509 level certificate issuance model.

Source

Launch of NEO3 TestNet and NEO3 Preview1

As a big milestone in the development of NEO3, Neo has launched the NEO3 TestNet and released NEO3 Preview1 this September.

With the launch of NEO3 Preview1, Neo has taken a tangible step forward in terms of advancing NEO3 while enabling its developer communities to begin testing and exploring NEO3 upgrades.

To accelerate enterprise-grade blockchain innovations for the future, NEO3 will empower developers by completely revamp NEO2 to deliver a scalable platform with higher throughput, enhanced stability and security, an optimized smart contract system, and a feature-packed infrastructure set.

Major features and improvements included in NEO3 Preview1:

Architecture Optimization

- Neo is moving away from UTXO (Unspent Transaction Output) to an Account based transaction model in NEO3. All assets will be created in smart contracts in NEO3, to fully utilize smart contract functionalities.

- Smart Contract enhancements

Native contracts for NEO, GAS, and Network Policy that can be invoked by other contracts: auto-claimable GAS

Contracts will now require application manifest and ScriptHeader to describe its properties, including: application Binary Interface (ABI) and contact method descriptor and permissions.

New smart contract APIs: access to notifications and JSON serialization.

-NeoVM is now decoupled from the Neo blockchain and can be used independently.

-Transaction optimizations

Single transaction type for all blockchain transactions

Scoped witness (scope for signature usage)

Transaction outcome is now being stored

Transaction validation by block height

-Blocks optimizations

Introduced maximum size limit for blocks

Introduced maximum size limit for witness for better spam attack preventions.

2. Stability enhancements

dBFT2.0:

1/3 fault tolerance

One-block finality

Recovery mechanism

3. Pricing model adjustment

Overall fee structure adjusted to greatly reduce system fees, contract deployment costs and NeoVM instruction costs (for information on current fees click here).

Network fees will be applicable for all transactions.

4. Performance enhancements

Auto compression mechanism on p2p network.

Source