NEL

Project composition

This project is called LightDB

It’s composed of three parts

Lightdb.lib is a layer of package for rocksdb. The main modification is to append the saved data type, and the concept of the table.

Https://github.com/lightszero/lightdb.lib

Lightdb.server is a remote database, the server version of Lightdb

Https://github.com/lightszero/lightdb.server

Lightdb.SDK is provided by the client to access LightdbServer. Currently, only the C# version is provided. The version of Typescript will be also provided later. The communication protocol is relatively simple. Any solution that supports websocket can be connected.

Https://github.com/lightszero/lightdb.sdk

Project Features

Let’s look at the features of the project,

First, the data type and table concept have been added.

Second, there is a server, using websocket communication, friendly to js.

Third, the data is stored in the taskblock unit to facilitate mutual authentication and synchronization of the database.

In fact, the most important feature is the snapshot reading. The demand for snapshots comes from the transactionalization of data processing, or the atomicity of a batch of operations.

Traditional database supports transactionalization to a certain extent, while the KeyValue database supports transactionalization to a lesser extent.

The embedded database used by NEO, LevelDB provides a very easy to read snapshot with a small cost, because of the LSM storage method it adopts.

This also caused the storage part to be very dependent on Snapshot in the implementation of NEO.

The original intention of creating this project is partly to transform NEO’s storage into network storage, and you can use light nodes to directly find the network database to execute InvokeScript.

That is to transform the NEO’s mode of “one node, one process” into “one node ,one cluster” mode.

The basis for accomplishing this goal is that this network database requires a similarly low-cost snapshot reading support.

So I chose rocksdb as the basis, and rocksdb is an improved version of facebook based on leveldb magic.

So why is the concept of data types and tables added? Because the things stored in leveldb are all byte[], and many times we don’t know what byte[] obtained in neo we use, unless there is the corresponding convention.

I think this is very inconvenient, so the unit we store is not a byte[] but a structure DBValue, which can describe what data type it is, be it integer, string, byte[], bigint, decimal, and so on.

This structure also reserves the function of appending information to each value, and also records the timestamp (storage height) of the final modification of this value.

As for the table, it’s easier to understood. This is a natural requirement for using leveldb. The keyvalue database is a dictionary, but the things we deposit are logically stored into several dictionaries, transaction dictionaries, utxo dictionaries, etc. .

We just provide the natural requirements of this sub-dictionary directly at the database level.

Then there is the network layer, and websocket is chosen as the communication method, which is based on the consideration of web page is the future. Because the storage function is isolated, you can even write a chain, and it runs entirely on the web.

Of course, I don’t think so much yet, but I can foresee that more and more businesses will run in the network. Because of the emergence of this database, at least 50% of the query requirements of NEL can be directly completed by js without the background server.

For example, blance of of nep5 assets and the confirmation of trading UTXO data.

Third, the storage of the database mimics the structure of the blockchain, with taskblock as the unit, which is connected end to end. After all, the original name of the project was called lightchain.

Later, I thought that the name is a bit mundane, so i decided to call the database as DB.

Why to use taskblock as the unit? Because it is easy to synchronize. Database read and write separation is the inevitable trend of performance expansion, and each operation is recorded. From the database, as long as each block is obtained from the main database, it is executed once, and the synchronization is completed.

Right, there is a bit like the NEO blockchain. Our write access is controlled not by the password, but the private key,which is fully compatible with NEO.

Finally, there is a very important feature called “ checkpoint”, which is a gift from rocksdb that can quickly generate a new copy of the database locally. In this way, we have more ways to make snapshots of on-chain data. The traditional method is to use the crawler to crawl the data that are to be snapshotted. It usually climbs to the specified height, and stop. Now there is another means: agreed time-based snapshot. A checkpoint can back up the whole database when it comes to the agreed time.

However, when the amount of data reaches the G level, what is the performance of this checkpoin still needs to be confirmed by the test.

How to use the project

First, open the server

Get lightdb.server, generate

Look at the config before starting

Server_port service port

Server_storage_path database storage path

Server_type read and write separation, slave mode has not been developed yet, only Master

Storage_maindb_magic provides a magic string so that this library can stand alone

Will be written to the library, after the database is created, it is meaningless to change this value.

Storage_maindb_firstwriter_address provides a NEO address, and he will be the first person to have write access. In fact, he has not developed the function of appending the writer. He is the only one.

Will be written to the library, after the database is created, it is meaningless to change this value.

Then start the server. It supports Linux and windows. We tested in win10, unbutu and centos. it needs to install the dependent library under linux, please refer to the “ Supported platforms section” in the description of facebook rocksdb project: https://github.com/facebook/rocksdb/blob/master/INSTALL.md

If the database has not been created in the path, otherwise it will open:

The console currently has only two functions: db.state and db.block.

Db.block will parse the block, we can see what each block has done , all the value areDBvalue, you can view the type with the DBValue structure.

Client usage

First reference lightdb.sdk, you can download the source, you can nuget

New a Client object

Register, disconnect event, then Connect

If the connection is successful, it is OK

You don’t need to log in, you can read it as you please. When you write, you need to sign your name first.

Ping and dbstate

All read operations are done via snap

So first obtain a snapshot

Then you can getblock

Getblockhash

Getvalue

client.Post_snapshot_getvalue(snapshotid, tableid, key);

It is more complicated to write data.

First create a writetask object, all writes are done through the writetask object.

Then create a signdata object and sign it.

Then post these two things to the server. Have a corresponding private key to complete the signature and successfully write the data.