Want a simple, persistent, key-value store in Go? Something handy to have in your toolbox that’s easy to use and performant? Let’s write one!

Here’s what we’d like it to do:

// everything is persisted to disk store , err := skv . Open ( "/path/to/store.db" ) // store a complex object without making a fuss var info session . Info store . Put ( "sess-341356" , info ) // get it back later, identifying the object with a string key store . Get ( "sess-341356" , & info ) // delete it when we no longer need it store . Delete ( "sess-341356" ) // bye store . Close ()

Note that we want to directly store and retrieve Go values, much like the fmt.Print and fmt.Scan functions. That should keep things simple for the caller, by not requiring (de)serialization code for the types.

We also want the Put, Get and Delete methods to be goroutine-safe, so that the calling code has one less thing to worry about.

Of course, we also expect the store to be fast and scalable too!

Step 1: Encoders

Encoders convert in-memory objects to byte streams.

Go has a bunch of built-in encoders under the encoding package – like json and xml. There are also other popular formats like msgpack, protocol buffers and flatbuffers.

They all have their strengths, but for our needs we’ll choose gob. Gob is a built-in package ( encoding/gob ) that is performant and easy-to-use. You can read more about gob here, and see some of the other contenders here.

Using the gob encoding, you can encode a Go value into a byte slice like so:

// the result goes into this buffer var buf bytes . Buffer // make an encoder that will write into the buffer encoder := gob . NewEncoder ( & buf ) // actually encode var obj SomeComplexType encoder . Encode ( obj ) // get the result out as a []byte result := buf . Bytes ()

Decoding is also equally easy:

// make a reader for the input (which is a []byte) reader := bytes . NewReader ( input ) // make a decoder decoder := gob . NewDecoder ( reader ) // decode it int var obj SomeComplexType decoder . Decode ( & obj )

Step 2: Persistent Storage

Probably the most widely used embedded storage library is SQLite. It does have database/sql bindings for Go, but being a C library it needs cgo and the compile times are frustratingly high.

LevelDB and RocksDB are two other candidates, but perhaps a bit of an overkill for our use case.

We’ll settle on a pure-Go, dependency-free, memory-mapped B+tree implementation called BoltDB. It can store key-value pairs (with both keys and values being []byte ), segregated into “buckets”.

Here’s how you can open the database and create a bucket:

// open the db db , err := bolt . Open ( "/path/to/store" , 0640 , nil ) // create a bucket db . Update ( func ( tx * bolt . Tx ) error { _ , err := tx . CreateBucketIfNotExists ([] byte ( "this.is.a.bucket.name" )) return err })

And within a bucket, you can store key-value pairs like this:

db . Update ( func ( tx * bolt . Tx ) error { err := tx . Bucket ( bucketName ) . Put ( key , value ) return err })

To fetch a particular record, you can seek using a cursor:

db . View ( func ( tx * bolt . Tx ) error { c := tx . Bucket ( bucketName ) . Cursor () if k , v := c . Seek ( key ); k == nil { // db is empty, key not found } else if bytes . Equal ( k , key ) { // found, use 'v' } else { // key not found } return nil })

As you can see, BoltDB is fairly simple to use. You can read more about it here.

Step 3: Putting the pieces together

Putting together BoltDB to manage the persistent store, and using the Gob encoder to (de)serialize Go values, we get skv - the Simple Key-Value store! It works exactly like we described above:

import "github.com/rapidloop/svk" // open the store store , err := svk . Open ( "/var/lib/mywebapp/sessions.db" ) // put: encodes value with gob and updates the boltdb err := svk . Put ( sessionId , info ) // get: fetches from boltdb and does gob decode err := svk . Get ( sessionId , & info ) // delete: seeks in boltdb and deletes the record err := svk . Delete ( sessionId ) // close the store store . Close ()

To import it into your project, use:

go get -u github.com/rapidloop/skv

which will import and build both skv and boltdb.

The whole code is only one file, has tests, lives in GitHub, and is MIT licensed. It even has documentation! Feel free to poke around, fork the code, and use it as you will. Happy coding!