BigchainDB is a database with added blockchain characteristics, that can store data to support decentralized applications. Think of a MongoDB database (which is what BigchainDB uses behind the scene) wrapped up in a burrito of blocks, transactions, votes and cryptography to allow for a high speed database that is also immutable and can be run by frenemies. The BigchainDB software is run by the application manager, but also other semi-trusted parties to form a resilient network of database hosts.

I have been working on architecting and implementing the use of BigchainDB as part of the indose.io platform. Like most work with decentralized technology at the present time, this has been exciting, challenging and at times, very frustrating *sigh… Today I want to share a few of the insights that we at Indorse have learned about working with BigchainDB.

1. Know why you are using BigchainDB

A crucial step we have found, before touching any code has been to figure out why we are using BigchainDB and what our objectives were. For Indorse we are using BigchainDB to add the following characteristics to our database:

Autonomy : If the entire Indorse team were to get hit by a bus (let’s hope not) and our AWS cloud account was to run out of gas then the Indorse platform would go on.

: If the entire Indorse team were to get hit by a bus (let’s hope not) and our AWS cloud account was to run out of gas then the Indorse platform would go on. Ownership: At Indorse, we are BIG on ownership. Users, not Indorse, own their data.

At Indorse, we are BIG on ownership. Users, not Indorse, own their data. BFT: BigchainDB allows us to share our database with our partners in a way that gives us the peace of mind that it is very difficult for the database to be compromised, even if one of our partners decided to try and attack our system.

BigchainDB allows us to share our database with our partners in a way that gives us the peace of mind that it is very difficult for the database to be compromised, even if one of our partners decided to try and attack our system. Transparency: We wanted to give users a transparent system, but as I’ll discuss below, we have had to make compromises.

It was also important for us to recognise some of the characteristics of blockchain that would cause headache for us: running on top of a blockchain structure BigchainDB is slower and less efficient, and the immutability of data has legal implications.

2. The balance between privacy, transparency and ownership

We quickly realized that the trifecta of full privacy, transparency and ownership could not all be achieved at the same time.

A cluster or network of BigchainDB nodes is supposed to run a decentralized database where all nodes have a consensus on the state of the database. In fact, much like Bitcoin, the software has been designed to allow for a public API to such a database. A network of BigchainDB nodes, called the InterPlanetary Database (IPDB), is currently live and running such a public database as a service.

Using a public cluster like IPDB gives users the ultimate ownership and transparency but the obvious problem is that everyone else can see other people’s data and that’s definitely not good enough.

The solution could be to encrypt that data, but then Indorse loses the ability to query the data for the benefit of the users (such as to help recruiters find prospective employees). We thought about running a database in parallel that was private and unencrypted but realized that to do this we would require the knowledge of the encryption key of the user’s data, compromising their ownership.

Stuck in this trilema, the way we are currently thinking is running an unencrypted private database cluster with an API over the top that manages the privacy of the users. Transactions can still be signed client side to give users full ownership, but as you can see there is only visibility of their own data.

I should also note at this stage that we consider BigchainDB to be a medium term tool. We are expecting to transition to a truly decentralized platform. We are investigating and keeping up with updates on projects such as Status, SWARM, Filecoin and Bluzelle, amongst the many others.

3. Bigchain isn’t ready, yet!

No it’s not production ready, but it’s pretty close and aside from a few hours spent pulling out hairs, it works pretty well. I don’t claim to be an expert on BigchainDB, but I would like to offer the following insights into the components:

The core node software: Written in python, this run’s pretty well. I’m pretty happy that I have yet to have to go into software repo to debug anything. It is a journey to figure out what is going on but the documentation is a very good guide. While the nodes function as expected, it came to our attention that BigchainDB is currently not working as a true P2P system. It is in fact leveraging the networking features of MongoDB replica sets to replicate data. The BigchainDB team are well aware of this and are planning on moving to a real P2P voting system using Tendermint but were unable to tell us when. Until they achieve this milestone BigchainDB is definitely not a product.

Written in python, this run’s pretty well. I’m pretty happy that I have yet to have to go into software repo to debug anything. It is a journey to figure out what is going on but the documentation is a very good guide. While the nodes function as expected, it came to our attention that BigchainDB is currently not working as a true P2P system. It is in fact leveraging the networking features of MongoDB replica sets to replicate data. The BigchainDB team are well aware of this and are planning on moving to a real P2P voting system using Tendermint but were unable to tell us when. Until they achieve this milestone BigchainDB is definitely not a product. Setting up a multi node cluster with SSL: Without SSL this was not too hard, again, it was just a matter of figuring out what BigchainDB was trying to do. Once it got to adding SSL certificates on top we ran into trouble. After nearly three days trying to set it up using our own VM’s in AWS we have had to put SSL on hold for now until we go through their more advanced guides using some funky container cluster managers. It is a bit of a shame that it’s so difficult to create a cluster in our own system but I attribute this still being early days for BigchainDB.

Without SSL this was not too hard, again, it was just a matter of figuring out what BigchainDB was trying to do. Once it got to adding SSL certificates on top we ran into trouble. After nearly three days trying to set it up using our own VM’s in AWS we have had to put SSL on hold for now until we go through their more advanced guides using some funky container cluster managers. It is a bit of a shame that it’s so difficult to create a cluster in our own system but I attribute this still being early days for BigchainDB. The Nodejs driver: This is probably the least developed component. What the Nodejs driver does support it does work, mostly, however there are a few features in the core node software that are not available. What made this hard was a lack of documentation, so going through the driver code was needed to do some debugging.

4. Allowing users to access Bigchain client side without knowing about it

The Indorse platform is trying to create a seamless experience that minimizes the user’s exposure to the technical components of blockchain that they are not used to. We allow users to sign on with the conventional username and password and don’t expose transactions or private keys or any of that jargon to them.

At the core of the Indorse platform is the Ethereum blockchain, which is being used to store tokens of value and execute the logic that processes this value. However BigchainDB is not compatible in the slightest with Ethereum. In fact Ethereum and BigchainDB use a completely different cryptographic algorithm to generate their key pairs (Ethereum uses the secp256k1, while BigchainDB uses curve 25519).

We wanted some mechanism that worked client side that would allow us to generate and store two separate key pairs. To add to the complexity, we needed that user be able to recover their keypair in the event that our naive user reset their phone, or their phone was lost or stolen. We found that the Edge wallet gave us exactly that solution! User’s can now sign up to Indorse, have two key pairs generated and backed up in a trustless way that allows them to recover, now isn’t that neat!