This post also available at: https://blog.louie.lu/2017/10/31/iota-iri-rocksdb-data-storage-structure/

0x1 Overview

In IOTA IRI, it uses rocksdb as its database with column family to separate different data stored. Here is the introduction about the IRI rocksdb data storage structure and how to manipulate in Python.

0x2 Column Families

In IOTA rocksdb, just think column families as a sheet to separate data stored. No high technique inside here. Also, need to notice that some of the columns are using merge operation with string append operator in “,”. That is, a value may contain multiple items, and separate with “,”.

Above table is the column families list of IOTA IRI rocksdb:

0x3 Convert between trytes-string and bytes

A big issue between a trytes-string and the database is, in the database, it stores not the “trytes-string” with bytes, but the “trytes-string” convert to bytes. Sounds strange, ugh? (more exactly, is “trits” convert to bytes)

Without any implement details, IOTA using trytes-string as its represent method, and we need to save trytes-string into database, there is a mapping between trits and bytes that can convert trytes-string into bytes.

Given an example in Python, consider the transaction hash (in trytes-string):

UNUK99RCIWLUQ9WMUT9MPQSZCHTUMGN9IWOCOXWMNPICCCQKLLNIIE9UIFGKZLHRI9QAOEQXQJLL99999

its trits will be:

[0, 1, -1, -1, -1, -1, 0, 1, -1, -1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 1, 0, 0, 0, 1, -1, -1, 0, 0, 1, 1, 0, 1, -1, -1, 0, -1, 0, 0, 0, -1, -1, 0, 1, 1, 1, 0, 1, -1, -1, 1, -1, 0, 0, 0, 1, 1, 1, 1, -1, -1, -1, 0, -1, 1, 0, -1, -1, 0, 0, 0, 1, 0, -1, 0, 1, -1, 1, -1, 0, 1, -1, 1, 1, 1, 1, -1, 1, -1, -1, -1, 0, 0, 0, 0, 0, 1, -1, -1, 0, 0, -1, -1, 0, 1, 0, 0, -1, -1, 0, -1, 0, -1, -1, 0, 1, 1, 1, -1, -1, -1, 1, -1, -1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, -1, 0, -1, -1, 1, 1, 0, 1, 1, 0, 1, 1, -1, -1, -1, 0, 0, 1, 0, 0, 1, -1, -1, 1, 0, 0, 0, 0, 1, -1, 0, 0, 1, 0, -1, 1, 1, -1, 1, -1, 1, 1, -1, 0, 0, 0, 1, 1, -1, 0, 1, 0, 0, -1, 0, 0, 1, 0, 0, 0, -1, 0, -1, 1, 0, 0, 0, -1, -1, -1, -1, 1, -1, 0, -1, 0, -1, 0, -1, 0, -1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

taking a trist to bytes convert, then the result will be:

b'\x8e\x9c\x04\x00\x08\xdf]\xe1\xdc^\xeau\xda\x96\x1b>>C\xf3\xca\x93\xb2\xa5t\xbd\x1a\x1c\xe8p\xf3\xcb\x02\xfadeP\x19N\xe5\x02)\xa53o\x0c\x00\x00\x00' .

0x4 Why convert trytes-string to bytes?

Each byte can contain 5 trits, so there is a 60% space reduce for this conversion.

0x5 Why do you need to know about IOTA IRI rocksdb

I want to create another IOTA node implement without java, thus I need to know about the layout of IRI rocksdb.

0x6 Anyway to manipulate with rocksdb data?

Currently, only java and rust (official implement) have the correct way dealing with database. I’m working on Python and mostly done (mlouielu/iota-python).

If you want to manipulate with Python, you will need to setup python-rocksdb, this is the version that I hack about adding column family with string append operator for IOTA (and, there is a hardcode include path in setup.py). And a converter is here in mlouielu/iota-python.

Manipulate IOTA rocksdb with Python: