DDAO: Decentralized Data Access Object

This technology is being developed by the QtumX team to enhance the current state of decentralized data management. The authors of this research are Zheng Yi and Li Zhen.

Abstract

We propose a library which is used to access various decentralized systems with uniform interfaces, to perform CRUD (Create, Read, Update, Delete) operations. Based on this library, developers can easily manipulate various decentralized systems and build decentralized applications without knowledge about the underlying technology.

Introduction

Data Management refers to the process of collecting, organizing, storing, processing, transmitting and retrieving different types of data. It is an important application area of computer science. One of its goals is to efficiently store and manage large amounts of complex data using computers so that people can quickly get access to the data. Another goal is to extract and derive valuable information from the data and then use the information as a guide for action and decision making.

At present, enterprises mainly use centralized databases and file systems for data management. Large Internet companies develop their systems, such as GFS (Google File System), SQL Server, etc., and deploy local services. Small and medium-sized companies connect to cloud services such as AWS, Aliyun, and delegate data management tasks to these third parties. Local and cloud data services have their application scenarios, and they maintain the basic data of many companies.

Centralized storage is currently the primary solution for data management, but there are serious storage problems in centralized databases and file systems around the world. Enterprises often store all their data and files in centralized databases and file systems. However, centralized storage solutions often attract attacks from hackers, or an accident may ruin them. Enterprises sometimes have to face the loss of all their data resources and never get them back. Some businesses highly rely on data and such kind of damage may lead to huge risks. Although cloud service providers promise backup data, it has not solved the fundamental problem.

Decentralization technologies, such as blockchain and IPFS, provide a new type of solutions to data management. By using encryption and consensus, data privacy and security are guaranteed. Since data is stored on many nodes, there is no need to worry about data loss caused by the failure of one single node, which improves security and reliability. Encryption and segmentation technologies control the read access and guarantee data privacy. At the same time, in order to ensure scalability, cluster technology is used. A cluster is a large group of nodes that store and manage data together, and it allows new nodes to join or expand into more clusters.

However, at present, with the increasing number of decentralized systems, developers have to face high costs in the selection, learning, development, deployment, and maintenance of these systems. Therefore we proposed the DDAO (Decentralized Data Access Object). Our goal is to develop a common library to access various decentralized systems with uniform interfaces, to perform CRUD (Create, Read, Update, Delete) operations. Through this library, developers can quickly read and write data in various decentralized systems and build their applications without having to care about the underlying technology.

Related Work

With the development of blockchain technology and digital currency market, decentralized data storage and management technologies have received attention from research fields and investment markets.

Decentralized File System

As a decentralized system, blockchain backs up data at every node, so it is not suitable for storing large-size files, such as pictures, videos, and audio. In current blockchain applications, such kind of files are still stored in servers held by developers. This way leads to the centralization of these applications, which violates the essential meaning of blockchain.

Decentralized file systems mainly include IPFS (InterPlanetary File System) [1], Swarm [2], Storj [3], and so on. Their basic method is to split the uploaded file into small pieces and then store them on nodes. When downloading the file, you can use the hash value of the file as an address to make a request to a node, and then you will get all the data pieces from the nodes and combine them into the whole file. In this way, one file can be separately stored on many nodes in a network, which achieves decentralization.

One problem with these technologies is: how do we keep more nodes online to provide download services? Hence, some projects propose to add node incentive mechanisms to the decentralized file system so that online nodes can obtain a revenue. In this way, the decentralized file system becomes a cloud storage service, in which users need to purchase the service during uploading and downloading processes so as to reward the nodes that provide the storage service. Several projects that use this structure include Filecoin [4], Wolk [5], Fluence [6], and Sia [7].

Decentralized Database

The decentralized file system enables decentralized storage of data, but after the data is uploaded, it cannot be further modified or retrieved, which limits its application in data management. Traditional centralized data management services usually use database systems such as MySQL or Redis to execute CRUD operations of data. Therefore, similar database services are required in the decentralized area.

Several projects are have researched decentralized databases, including BigchainDB [8], Bluzelle [9], Ties.DB [10] and so on. Their basic idea is to upload database write requests to the blockchain through transactions, and then use database engines such as MongoDB to perform data indexing and process read requests. Using blockchain to build decentralized databases, effectively utilizing the mature decentralized architecture of blockchain systems, and this has significant advantages in stability and security.

These projects are currently in their initial stages, and before their implementations mature, using smart contracts as a database service is also a viable alternative. Some projects are based on Ethereum + IPFS to develop their applications, using smart contracts written in Solidity for data management. Solidity supports data structures such as List, Map, Struct, etc., which are sufficient for common data management scenarios. These items are fully enumerated in Awesome IPFS [11].

System Description

Decentralized systems have a variety of implementations, leading to high learning costs for developers. This article will design a common library for connecting upper-layer DApp applications to lower-layer decentralized systems, enabling CRUD operations for multiple decentralized systems. Developers can quickly build their data management applications through this library without having knowledge about the underlying technology. At the same time, based on this library, a cloud service can be built to provide online CRUD interfaces for numerous decentralized systems, eliminating the cost of deploying and maintaining decentralized systems for developers.

The system can be separated into three layers, namely DDMI (Decentralized Data Management Infrastructure), DDAO (Decentralized Data Access Object), and DApp (Decentralized Application), which will be introduced as follows.

DDMI

The DDMI layer contains a variety of decentralized systems, including blockchains, decentralized databases, decentralized file systems, and more. These decentralized systems can be launched locally by the developer or deployed on a remote server, and then connected through a descriptor with a uniform format. For example, the format of name://user:password@ip:port, that is, connecting to a local Qtum node with port 13889 by qtum://admin:123456@127.0.0.1:13889, or connecting to an IPFS node in LAN with 8080 port by ipfs://myname:mypasswd@192.168.1.2:8080.

DDAO

The DDAO layer is the core module proposed in this paper. Its main functions include (1) establishing connections to various decentralized systems, (2) abstracting to get compatible CRUD interfaces. For different types of decentralized systems, abstract interfaces are different:

For blockchain systems, interfaces contain sending transactions, querying transactions, creating contracts, reading and writing contracts. For decentralized database systems, database interfaces such as create, find, insert, and update are included. For decentralized file systems, main interfaces such as file upload, download, and encryption are included.

The design here needs to take into account the compatibility of different systems, so only some common functions are abstracted. At the same time, the interface used to send the original commands is offered, so that developers can request some unique interfaces of each system.

DApp

The DApp can be JavaScript scripts running on browser or programs running on a server. They may also be stored in the decentralized file system and later obtained through a specific address as an entry. The DApp initiates the DDAO instance, connects local or cloud decentralized systems, and performs CRUD operations on these systems, so as to realize data management applications.

Multiple DDAO instances can be initiated within one DApp, in order to connect to multiple different decentralized systems. This allows data interaction with various decentralized systems in one application, enabling more features compared with traditional smart contract-based DApps. For example, cross-chain data transmission among multiple blockchains, or data visualization based on blockchains and file systems.

Application Scenarios

A large number of application scenarios is the best way to describe the value of DDAO. Based on DDAO, many decentralized applications will become easier to develop and implement.

Decentralized Content Sharing

DDAO can be used to build a blog, photo, music, video sharing platform. By using DDAO’s write interfaces, users can publish texts, images, videos and other content to a decentralized file system such as IPFS, and then store their metadata (such as content address, release time, user ID, etc.) into a smart contract or a database. The DDAO read interfaces are used to filter, sort, and display the content. The advantage of such a system is that the contents published by users is decentralized and cannot be tampered with.

Decentralized Trading Platform

Based on DDAO, you can build a trading platform like Taobao and Amazon. When a merchant uses DDAO write interfaces to upload its product information, the text and image are stored in a decentralized file system, and the metadata (such as price, quantity, classification, keywords, etc.) are stored in a smart contract or a database. When a customer browses the platform, DDAO read interfaces are used to search and filter products. Finally, the customer uses DDAO to connect to the blockchain to create orders, pay orders, and complete orders. The advantage of this platform is that the product information and transaction process are transparent and the payment process is convenient.

Decentralized Cloud Storage Service

Storage service providers register their information into a blockchain or a database via DDAO. When a user uploads a file, he needs to use blockchain tokens to purchase the service, and then uploads the file to a decentralized file system through DDAO, while the file information and address are stored in a smart contract or a database. When downloading, users browse the file information and address by using DDAO, and then download the file from the file system. The tokens consumed by the user are used to reward storage service providers, forming a complete economic ecosystem.

Summary

Decentralized data management has broad application prospects. Although, at present, decentralized databases and file systems are still under development, and immature infrastructure limits its application, it is conceivable that these systems will be implemented one after another in the near future. DDAO will also be widely used just like the DAO in current centralized systems.

References

Contact:

Zheng Yi (zhengyi@qtum.org)