Build a Redis Compatible Cluster based on TiKV

What is Redis?

Redis, a fast, open source, in-memory key-value data store, can be used as a database, cache and message broker. It provides rich data structures like String, Hash, Set and Sorted Set, which can help the users build their products easily. It is a high-performance, maybe one of the fastest database in the world. Although it saves data in memory, it supports persistent and async-replication which can guarantee data safely.

Limitations of Redis

Redis is cool, but it still has some problems:

Memory is expensive, and not unlimited. You can’t save huge data in one machine. Async-replication can’t keep data consistent. The transaction mode can’t guarantee the ACID compliance. Although it supports clustering, it can’t support cross-node distributed transactions.

Sometimes, we need a more powerful database which supports:

Rich data structure, like string, hash, set, sorted set, etc. High performance Data strong consistency Horizontal scalability Distributed transactions

Why TiKV?

About 4 years ago, I started to solve the problems of Redis. To keep data persistent, the intuitive way was to save the data in the disk, not in memory. So I developed LedisDB, a database which uses RocksDB to save data, supports the Redis Protocol and provides the Redis data structure. LedisDB works well but it is not fully Redis-compatible. So later, I created RebornDB, a fully redis-compatible database.

Both LedisDB and RebornDB save data in the disk and overcome the memory limitation. But they can’t support ACID transaction. Although we can use codis to extend them to support the cluster, it is still not convenient and can’t support the distributed transactions.

So we need a new way. Fortunately, it is not hard for us now, because we have TiKV.

TiKV is a high performance, distributed transactional key-value store. Although it just provides a simple key-value API, you can build your own logic on top of it. For example, we have already built TiDB — a distributed relational database based on TiKV. TiDB supports the MySQL protocol, and it maps the database schema to the key-value internally. So for Redis, we can also build a service that supports the Redis protocol and maps the Redis data structure to the key-value format.

How

The whole architecture is simple. What we only need to do is to build a Redis Proxy, which parses the Redis Protocol, and translates the Redis data structure to the TiKV key-value structure.

Redis Protocol

The Redis Protocol is called RESP(Redis Serialization Protocol), it is text-based, human readable, and easy to parse. It uses “\r

” for the termination and uses different prefix for the different types. For example, for simple strings, the first byte is “+”, so the strings may like “+OK\r

”.

Mostly, the client uses Request-Response mode for the communication. The client sends a request (with the Array type consisting of Bulk strings) to the server and the server returns any RESP type as a reply. Redis also has two extension modes:

Pipeline: in which the client sends multi requests to the server and receives a reply Push: After the client subscribes a channel, the client will only receive the replies from the server.

Here is a simple example when the client sends the LLEN mylist command to the server:

C: *2\r



C: $4\r



C: LLEN\r



C: $6\r



C: mylist\r





S: :48293\r



The client sends an array with two bulk strings, the first bulk string length is 4 and second is 6, to the server. The server returns an integer 48293. As you can see, RESP is simple, and writing an RESP parser is very easy.

I just created a Go library goredis, using which we can easily parse RESP from the connection. For example,

// Create a buffer IO from the connection.

br := bufio.NewReaderSize(conn, 4096)

// Create a RESP reader.

r := goredis.NewRespReader(br)

// Parse the Request

req := r.ParseRequest()

The ParseRequest returns a parsed request, which is a [][]byte in Go, the first item is the command name like “LLEN”, and the rest items are the arguments for this command.

TiKV Transaction API

Before we start, I will show a simple example of how to use TiKV transactional API.

For every transaction, we will start with a Begin function, like:

txn, err := db.Begin()

Begin creates a function. If something goes wrong, we need to check the error here.

After we create the transaction, we can do some operations, like:

value, err := txn.Get([]byte(“key”))

// Do something with value and then update the newValue to the key.

txn.Put([]byte(“key”), newValue)

TiKV uses an optimistic commit model, so all the changes are cached in the client and then submitted to the server at the commit.

// Commit the transaction

txn.Commit(context.TODO())

Like common transaction, we can also cancel the transaction:

txn.Rollback()

If two transactions operate the same key, they meet key conflict. In this case, only one transaction can commit successfully, and the other one will return a conflict error and abort.

Mapping Data structure to TiKV

Now we know how to parse the Redis protocol, do the key-value operations in a transaction. The next thing is to support Redis data structure. Redis has four main data structures: String, Hash, Set and Sorted Set. But for TiKV, it only supports key-value, so we need to map these different data structures to key-value.

At first, we need to distinguish different data structures in TiKV. An easy way is to append a Type flag at the end of the key. For example, we can add ‘s’ for String, so the String “abc” will be saved as “abcs” in TiKV.

For other types, we need to consider more and we can’t use only one Type flag. E.g, for the Hash type, we need to support following operations:

HSET key field1 value1

HSET key field2 value2

HLEN key

A Hash has many fields and we also want to get the length of the Hash easily. So in TiKV, we will combine the Hash key and field together as a TiKV key, and also save the length using another key. The layout may look:

key + ‘h’ -> length

key + ‘f’ + field1 -> value

key + ‘f’ + field2 -> value

If we don’t save the length as an another key, every time if we want to get the length of Hash, we need to scan all the data with prefix “key”, which is not effective. But if we use another key for length, every time we add a new field for Hash, we must update the length value. This is a design trade-off, and here I prefer using another length key because HLEN is a frequent operation.

Demo

I build a very simple example which only supports String and Hash and a few operations. You can run with:

git clone https://github.com/siddontang/redis-tikv-example.git $GOPATH/src/github.com/siddontang/redis-tikv-example



cd $GOPATH/src/github.com/siddontang/redis-tikv-example

go build

Before running the example, you need to start TiKV, you can follow the instruction here, then run:

./redis-tikv-example

The example server will listen port 6380, you can use any redis client like redis-cli to communicate with it:

redis-cli -p 6380

127.0.0.1:6380> set k1 a

OK

127.0.0.1:6380> get k1

"a"

127.0.0.1:6380> hset k2 f1 a

(integer) 1

127.0.0.1:6380> hget k2 f1

"a"

Epilogue

Now some companies have already built their Redis Protocol Server based on TiKV and there is a cool open source project named tidis which does the same thing too. If you want to find a solution to replace your Redis server, you can have a try.

As you can see, TiKV is a building block, we can build many other systems easily based on top of it. If you want to join us to develop this wonderful project, drop me a note at tl@pingcap.com!