As blockchains grow with each authenticated transaction, querying and reading history and state data proves to be a growing challenge. In order to tackle this challenge, we have been working on a series of History Tools designed to make data more accessible for EOSIO based blockchains. You can find more technical details and documentation in the History Tools repository on GitHub. Documentation for History Tools can be found here.

We have previously released the State History Plugin that makes accessing blockchain data easier. The state history plugin saves blockchain state data to a new flat-file format and exposes a websocket interface to read the block/state data. With the alpha release of the History-Tools solution, we take this one step further and introduce database fillers to read the state data and populate PostgreSQL and LMDB databases that can be queried with the specialized client and server WASMs.

Rich user experiences and reliable applications built on EOSIO require tools that allow applications to efficiently access and query data from the blockchain and smart contracts. By nature, blockchain systems store data in a sequential manner that doesn’t easily lend itself to scalable random access query and retrieval systems.

As this is an alpha release, we expect to iterate changes throughout the development process. It is our hope that feedback from members of the community will help to provide us additional guidance as we roll out these tools.

WASM-Query Language

WASM-Query Language (WASM-QL) implements a client-server architecture to design and execute queries in WASM. This implementation pattern allows contract authors to design and code queries using the same tooling used to create contracts and leverage the client-server query architecture. It also minimizes the back-and-forth communication required between client and server optimizing speed and efficiency. Once enabled, this system would make it possible to inspect historical states at any given block height. In addition, since replication in this system is relegated away from nodeos towards low memory-intensive query processes, it is more conducive to scalability.

This offers a more streamlined approach for the general-purpose query engines that enable applications to gather data from the EOSIO smart contracts. In addition, authors of contracts and applications can design their own queries with the same toolset that contracts are designed with, providing overall greater synergy.

There are two types of queries that wasm-ql-pg supports:

Binary queries are the most flexible. A client-side WASM packs the query into a binary format, a server-side WASM running in WASM-QL executes the query then produces a result in binary format, then the client-side WASM converts the binary to JSON. The toolset helps authors create both WASMs.

An emulation of the /v1/ RPC API.

In time, we may choose to drop the client-side WASMs and switch the format of the first type of query to JSON RPC, Graph QL, or another format. We appreciate community feedback as we make these decisions, so please comment in the repository with the recommendation you support.

Fill-pg

Fill-pg is designed to offer applications a lightweight solution to gather all pertinent data they would normally receive from monitoring the main chain. The information gathered includes:

Header information from each block

Transaction and action traces, including inline actions and deferred transactions

Contract table history, at the block level

Tables which track the history of chain state, including

Accounts, including permissions and linkauths

Account resource limits and usage

Contract code

Contract ABIs

Consensus parameters

Activated consensus upgrades

Fill-pg cuts dead weight by excluding blocks from the equation since they don’t include the inline actions and deferred transaction data that applications need. It supports both full and partial history so users can decide if they want to query the entire chain or a subsection.

Fill-lmdb, wasm-ql-lmdb

This pair operates identically to fill-pg and wasm-pg, however, data is stored using lmdb as opposed to postgresql; as an embedded database lmdb might provide administration advantages.

Making Blockchain Data Accessible

In order to deliver seamless user experiences for more complex applications that maintain the integrity of blockchain data, developers need robust access to deterministic databases that can be queried like their non-blockchain counterparts. Blockchain nodes have a narrow scope of query options which causes problems to arise when complex applications try to store and retrieve indexed data. Traditionally, this left developers with two options; either processing the data within the application or storing sorted data on the blockchain itself which can be inefficient and expensive.

By reducing the back-and-forth between application databases and the EOSIO blockchain we’re following through on providing superior tools for Data Access Scalability, outlined in the EOSIO Strategic Vision. As we continue to receive feedback, we’ll continue to optimize methods that can reduce resource loads and help developers create more efficient application architectures.

Stay Connected

We appreciate the efforts of those who continue to help test and provide input on these tools as they evolve to best serve the EOSIO community. If you would like to offer feedback and work more closely with our team to improve EOSIO for developers, you can send our developer relations team an email at developers@block.one.

Remember, you can also keep up-to-date with future announcements by subscribing to our mailing list on the new EOSIO website. We are excited to be regularly improving the usability of the software for EOSIO developers as we continue to lay a foundation for the mass adoption of blockchain technology.