My new book guides you through the start-to-finish build of a real world web application in Go — covering topics like how to structure your code, manage dependencies, create dynamic database-driven pages, and how to authenticate and authorize users securely.

Working with Redis in Go

In this post I'm going to be looking at using Redis as a data persistence layer for a Go application. We'll start by explaining a few of the essential concepts, and then build a working web application which highlights some techniques for using Redis in a concurrency-safe way.

This post assumes a basic knowledge of Redis itself (and a working installation, if you want to follow along). If you haven't used Redis before, I highly recommend reading the Little Book of Redis by Karl Seguin or running through the Try Redis interactive tutorial.

Installing a driver

First up we need to install a Go driver (or client) for Redis. A list of available drivers is located at http://redis.io/clients#go.

The two drivers that I would recommend are gomodule/redigo and mediocregopher/radix. They are both well designed and actively maintained.

The key differences are that Redigo is completely self-contained (with no external dependencies) and it has a smaller, simpler API than Radix. Radix, on the other hand, provides support for Redis sentinel and cluster implementations.

Throughout this post we'll be using the Redigo driver.

Getting started with Redis and Go

As an example, let's say that we have an online record shop and want to store information about the albums for sale in Redis.

There's many different ways we could model this data in Redis, but we'll keep things simple and store each album as a hash – with fields for title, artist, price and the number of 'likes' that it has. As the key for each album hash we'll use the pattern album:{id} , where id is a unique integer value.

So if we wanted to store a new album using the Redis CLI, we could execute a HMSET command along the lines of:

127.0.0.1:6379> HMSET album:1 title "Electric Ladyland" artist "Jimi Hendrix" price 4.95 likes 8 OK

To do the same thing from a Go application, we need to combine a couple of functions from the gomodule/redigo/redis package.

The first is the Dial() function, which returns a new connection to our Redis server.

function, which returns a new connection to our Redis server. The second is the Do() method, which sends a command to our Redis server across the connection. This returns the reply from Redis as an interface{} type, along with any error if applicable.

Using them is quite straightforward in practice:

File: main.go package main import ( "fmt" "log" // Import the redigo/redis package. "github.com/gomodule/redigo/redis" ) func main() { // Establish a connection to the Redis server listening on port // 6379 of the local machine. 6379 is the default port, so unless // you've already changed the Redis configuration file this should // work. conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } // Importantly, use defer to ensure the connection is always // properly closed before exiting the main() function. defer conn.Close() // Send our command across the connection. The first parameter to // Do() is always the name of the Redis command (in this example // HMSET), optionally followed by any necessary arguments (in this // example the key, followed by the various hash fields and values). _, err = conn.Do("HMSET", "album:2", "title", "Electric Ladyland", "artist", "Jimi Hendrix", "price", 4.95, "likes", 8) if err != nil { log.Fatal(err) } fmt.Println("Electric Ladyland added!") }

In this example we're not really interested in the reply from Redis (all successful HMSET commands just reply with the string "OK") so we don't do anything except check the return value from Do() for any errors.

Working with replies

When we are interested in the reply from Redis, the gomodule/redigo/redis package contains some useful helper functions for converting the reply (which has the type interface{} ) into a Go type we can easily work with. These are:

redis.Bool() – converts a single reply to a bool

redis.Bytes() – converts a single reply to a byte slice ( []byte )

) redis.Float64() – converts a single reply to a float64

redis.Int() – converts a single reply to a int

redis.String() – converts a single reply to a string

redis.Values() – converts an array reply to an slice of individual replies

redis.Strings() – converts an array reply to an slice of strings ([]string)

redis.ByteSlices() – converts an array reply to an slice of byte slices ([][]byte)

redis.StringMap() – converts an array of strings (alternating key, value) into a map[string]string . Useful for HGETALL etc

Let's use some of these in conjunction with the HGET command to retrieve information from one of the album hashes:

File: main.go package main import ( "fmt" "log" "github.com/gomodule/redigo/redis" ) func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Issue a HGET command to retrieve the title for a specific album, // and use the Str() helper method to convert the reply to a string. title, err := redis.String(conn.Do("HGET", "album:1", "title")) if err != nil { log.Fatal(err) } // Similarly, get the artist and convert it to a string. artist, err := redis.String(conn.Do("HGET", "album:1", "artist")) if err != nil { log.Fatal(err) } // And the price as a float64... price, err := redis.Float64(conn.Do("HGET", "album:1", "price")) if err != nil { log.Fatal(err) } // And the number of likes as an integer. likes, err := redis.Int(conn.Do("HGET", "album:1", "likes")) if err != nil { log.Fatal(err) } fmt.Printf("%s by %s: £%.2f [%d likes]

", title, artist, price, likes) }

It's worth pointing out that, when we use these helper methods, the error they return could relate to one of two things: either the failed execution of the command, or the conversion of the reply data to the desired type (for example, we'd get an error if we tried to convert the reply "Jimi Hendrix" to a float64 ). There's no way of knowing which kind of error it is unless we examine the error message.

If you run the code above you should get output which looks like:

$ go run main.go Electric Ladyland by Jimi Hendrix: £4.95 [8 likes]

Let's now look at a more complete example, where we use the HGETALL command to retrieve all fields from an album hash in one go and store the information in a custom Album struct.

File: main.go package main import ( "fmt" "log" "strconv" "github.com/gomodule/redigo/redis" ) // Define a custom struct to hold Album data. type Album struct { Title string Artist string Price float64 Likes int } func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Fetch all album fields with the HGETALL command. Because HGETALL // returns an array reply, and because the underlying data structure // in Redis is a hash, it makes sense to use the Map() helper // function to convert the reply to a map[string]string. reply, err := redis.StringMap(conn.Do("HGETALL", "album:1")) if err != nil { log.Fatal(err) } // Use the populateAlbum helper function to create a new Album // object from the map[string]string. album, err := populateAlbum(reply) if err != nil { log.Fatal(err) } fmt.Printf("%+v", album) } // Create, populate and return a pointer to a new Album struct, based // on data from a map[string]string. func populateAlbum(reply map[string]string) (*Album, error) { var err error album := new(Album) album.Title = reply["title"] album.Artist = reply["artist"] // We need to use the strconv package to convert the 'price' value // from a string to a float64 before assigning it. album.Price, err = strconv.ParseFloat(reply["price"], 64) if err != nil { return nil, err } // Similarly, we need to convert the 'likes' value from a string to // an integer. album.Likes, err = strconv.Atoi(reply["likes"]) if err != nil { return nil, err } return album, nil }

Running this code should give an output like:

$ go run main.go &{Title:Electric Ladyland Artist:Jimi Hendrix Price:4.95 Likes:8}

Or an alternative, and arguably neater, approach is to use the redis.Values() and redis.ScanStruct() functions to automatically unpack the data to the Album struct, like so:

File: main.go package main import ( "fmt" "log" "github.com/gomodule/redigo/redis" ) // Define a custom struct to hold Album data. Notice the struct tags? // These indicate to redigo how to assign the data from the reply into // the struct. type Album struct { Title string `redis:"title"` Artist string `redis:"artist"` Price float64 `redis:"price"` Likes int `redis:"likes"` } func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Fetch all album fields with the HGETALL command. Wrapping this // in the redis.Values() function transforms the response into type // []interface{}, which is the format we need to pass to // redis.ScanStruct() in the next step. values, err := redis.Values(conn.Do("HGETALL", "album:1")) if err != nil { log.Fatal(err) } // Create an instance of an Album struct and use redis.ScanStruct() // to automatically unpack the data to the struct fields. This uses // the struct tags to determine which data is mapped to which // struct fields. var album Album err = redis.ScanStruct(values, &album) if err != nil { log.Fatal(err) } fmt.Printf("%+v", album) }

Note: Behind the scenes the redis.ScanStruct() function uses Go's strconv package to convert the values returned from Redis in to the appropriate Go type for the struct field — similar in principle to what we did in the previous example. By default this supports integer, float, boolean, string and []byte fields. If you need to automatically scan into a custom type, you can do so by implemeting the redis.Scanner() interface on your custom type.

Using in a web application

One important thing to know about gomodule/redigo/redis is that the Conn object (which is returned by the Dial() function we've been using so far) is not safe for concurrent use.

If we want to access a single Redis server from multiple goroutines, as we would in a web application, we must use establish a pool of Redis connections, and each time we want to use a connection we fetch it from the pool, execute our command on it, and return it too the pool.

We'll illustrate this in a simple web application, building on the online record store example we've already used. Our finished app will support 3 functions:

Method Path Function GET /album?id=1 Show details of a specific album (using the id provided

in the query string) POST /like Add a new like for a specific album (using the id

provided in the request body) GET /popular List the top 3 most liked albums in order

To avoid detracting from the main purpose of this blog post (which is talking about Redis) we'll use a deliberately over-simplified pattern for our web application. If you'd like to follow along, create a basic application scaffold like so…

$ mkdir recordstore && cd recordstore $ go mod init example.com/recordstore go: creating new go.mod: module example.com/recordstore $ touch main.go albums.go $ tree . ├── albums.go ├── go.mod └── main.go

…And use the Redis CLI to add a few additional albums, along with a new likes sorted set. This sorted set will be used within the GET /popular route to help us quickly and efficiently retrieve the ids of albums with the most likes. Here's the commands to run:

HMSET album:1 title "Electric Ladyland" artist "Jimi Hendrix" price 4.95 likes 8 HMSET album:2 title "Back in Black" artist "AC/DC" price 5.95 likes 3 HMSET album:3 title "Rumours" artist "Fleetwood Mac" price 7.95 likes 12 HMSET album:4 title "Nevermind" artist "Nirvana" price 5.95 likes 8 ZADD likes 8 1 3 2 12 3 8 4

In the albums.go file we'll define a global variable to hold a Redis connection pool, and we'll re-purpose the code we wrote earlier into a FindAlbum() function that we can use from our HTTP handlers.

File: albums.go package main import ( "errors" "github.com/gomodule/redigo/redis" ) // Declare a pool variable to hold the pool of Redis connections. var pool *redis.Pool var ErrNoAlbum = errors.New("no album found") // Define a custom struct to hold Album data. type Album struct { Title string `redis:"title"` Artist string `redis:"artist"` Price float64 `redis:"price"` Likes int `redis:"likes"` } func FindAlbum(id string) (*Album, error) { // Use the connection pool's Get() method to fetch a single Redis // connection from the pool. conn := pool.Get() // Importantly, use defer and the connection's Close() method to // ensure that the connection is always returned to the pool before // FindAlbum() exits. defer conn.Close() // Fetch the details of a specific album. If no album is found // the given id, the []interface{} slice returned by redis.Values // will have a length of zero. So check for this and return an // ErrNoAlbum error as necessary. values, err := redis.Values(conn.Do("HGETALL", "album:"+id)) if err != nil { return nil, err } else if len(values) == 0 { return nil, ErrNoAlbum } var album Album err = redis.ScanStruct(values, &album) if err != nil { return nil, err } return &album, nil }

Alright, let's head over to the main.go file. In this we will initialize the connection pool and set up a simple web server and HTTP handler for the GET /album route.

File: main.go package main import ( "fmt" "log" "net/http" "strconv" "time" "github.com/gomodule/redigo/redis" ) func main() { // Initialize a connection pool and assign it to the pool global // variable. pool = &redis.Pool{ MaxIdle: 10, IdleTimeout: 240 * time.Second, Dial: func() (redis.Conn, error) { return redis.Dial("tcp", "localhost:6379") }, } mux := http.NewServeMux() mux.HandleFunc("/album", showAlbum) log.Println("Listening on :4000...") http.ListenAndServe(":4000", mux) } func showAlbum(w http.ResponseWriter, r *http.Request) { // Unless the request is using the GET method, return a 405 'Method // Not Allowed' response. if r.Method != http.MethodGet { w.Header().Set("Allow", http.MethodGet) http.Error(w, http.StatusText(405), 405) return } // Retrieve the id from the request URL query string. If there is // no id key in the query string then Get() will return an empty // string. We check for this, returning a 400 Bad Request response // if it's missing. id := r.URL.Query().Get("id") if id == "" { http.Error(w, http.StatusText(400), 400) return } // Validate that the id is a valid integer by trying to convert it, // returning a 400 Bad Request response if the conversion fails. if _, err := strconv.Atoi(id); err != nil { http.Error(w, http.StatusText(400), 400) return } // Call the FindAlbum() function passing in the user-provided id. // If there's no matching album found, return a 404 Not Found // response. In the event of any other errors, return a 500 // Internal Server Error response. bk, err := FindAlbum(id) if err == ErrNoAlbum { http.NotFound(w, r) return } else if err != nil { http.Error(w, http.StatusText(500), 500) return } // Write the album details as plain text to the client. fmt.Fprintf(w, "%s by %s: £%.2f [%d likes]

", bk.Title, bk.Artist, bk.Price, bk.Likes) }

It's worth elaborating on the redis.Pool settings. In the above code we specify a MaxIdle size of 10, which simply limits the number of idle connections waiting in the pool to 10 at any one time. If all 10 connections are in use when an additional pool.Get() call is made a new connection will be created on the fly. The IdleTimeout setting is set to 240 seconds, which means that any connections that are idle for longer than that will be removed from the pool.

If you run the application: $ go run . 2019/08/17 11:01:41 Listening on :4000...

And make a request for one of the albums using cURL you should get a response like this:

$ curl -i localhost:4000/album?id=2 HTTP/1.1 200 OK Content-Length: 42 Content-Type: text/plain; charset=utf-8 Back in Black by AC/DC: £5.95 [3 likes]

Using transactions

The second route, POST /likes , is quite interesting.

When a user likes an album we need to issue two distinct commands: a HINCRBY to increment the likes field in the album hash, and a ZINCRBY to increment the relevant score in our likes sorted set.

This creates a problem. Ideally we would want both keys to be incremented at exactly the same time as a single atomic action. Having one key updated after the other opens up the potential for race conditions to occur.

The solution to this is to use Redis transactions, which let us run multiple commands together as an atomic group. To do this we use the MULTI command to start a transaction, followed by the commands (in our case a HINCRBY and ZINCRBY ), and finally the EXEC command (which then executes our both our commands together as an atomic group).

Let's create a new IncrementLikes() function in the albums.go file which uses this technique.

File: albums.go ... func IncrementLikes(id string) error { conn := pool.Get() defer conn.Close() // Before we do anything else, check that an album with the given // id exists. The EXISTS command returns 1 if a specific key exists // in the database, and 0 if it doesn't. exists, err := redis.Int(conn.Do("EXISTS", "album:"+id)) if err != nil { return err } else if exists == 0 { return ErrNoAlbum } // Use the MULTI command to inform Redis that we are starting a new // transaction. The conn.Send() method writes the command to the // connection's output buffer -- it doesn't actually send it to the // Redis server... despite it's name! err = conn.Send("MULTI") if err != nil { return err } // Increment the number of likes in the album hash by 1. Because it // follows a MULTI command, this HINCRBY command is NOT executed but // it is QUEUED as part of the transaction. We still need to check // the reply's Err field at this point in case there was a problem // queueing the command. err = conn.Send("HINCRBY", "album:"+id, "likes", 1) if err != nil { return err } // And we do the same with the increment on our sorted set. err = conn.Send("ZINCRBY", "likes", 1, id) if err != nil { return err } // Execute both commands in our transaction together as an atomic // group. EXEC returns the replies from both commands but, because // we're not interested in either reply in this example, it // suffices to simply check for any errors. Note that calling the // conn.Do() method flushes the previous commands from the // connection output buffer and sends them to the Redis server. _, err = conn.Do("EXEC") if err != nil { return err } return nil }

We'll also update the main.go file to add an addLike() handler for the route:

File: main.go package main import ( "fmt" "log" "net/http" "strconv" "time" "github.com/gomodule/redigo/redis" ) func main() { pool = &redis.Pool{ MaxIdle: 10, IdleTimeout: 240 * time.Second, Dial: func() (redis.Conn, error) { return redis.Dial("tcp", "localhost:6379") }, } mux := http.NewServeMux() mux.HandleFunc("/album", showAlbum) mux.HandleFunc("/like", addLike) log.Println("Listening on :4000...") http.ListenAndServe(":4000", mux) } ... func addLike(w http.ResponseWriter, r *http.Request) { // Unless the request is using the POST method, return a 405 // Method Not Allowed response. if r.Method != http.MethodPost { w.Header().Set("Allow", http.MethodPost) http.Error(w, http.StatusText(405), 405) return } // Retrieve the id from the POST request body. If there is no // parameter named "id" in the request body then PostFormValue() // will return an empty string. We check for this, returning a 400 // Bad Request response if it's missing. id := r.PostFormValue("id") if id == "" { http.Error(w, http.StatusText(400), 400) return } // Validate that the id is a valid integer by trying to convert it, // returning a 400 Bad Request response if the conversion fails. if _, err := strconv.Atoi(id); err != nil { http.Error(w, http.StatusText(400), 400) return } // Call the IncrementLikes() function passing in the user-provided // id. If there's no album found with that id, return a 404 Not // Found response. In the event of any other errors, return a 500 // Internal Server Error response. err := IncrementLikes(id) if err == ErrNoAlbum { http.NotFound(w, r) return } else if err != nil { http.Error(w, http.StatusText(500), 500) return } // Redirect the client to the GET /album route, so they can see the // impact their like has had. http.Redirect(w, r, "/album?id="+id, 303) }

If you make a POST request to like one of the albums you should now get a response like:

$ curl -i -L -d "id=2" localhost:4000/like HTTP/1.1 303 See Other Location: /album?id=2 Date: Sat, 17 Aug 2019 16:50:49 GMT Content-Length: 0 HTTP/1.1 200 OK Date: Sat, 17 Aug 2019 16:50:49 GMT Content-Length: 42 Content-Type: text/plain; charset=utf-8 Back in Black by AC/DC: £5.95 [4 likes]

Using the Watch command

OK, on to our final route: GET /popular . This route will display the details of the top 3 albums with the most likes, so to facilitate this we'll create a FindTopThree() function in the albums.go file. In this function we need to:

Use the ZREVRANGE command to fetch the 3 album ids with the highest score (i.e. most likes) from our likes sorted set. Loop through the returned ids, using the HGETALL command to retrieve the details of each album and add them to a []*Album slice.

Again, it's possible to imagine a race condition occurring here. If a second client happens to like an album at the exact moment between our ZREVRANGE command and the HGETALL s for all 3 albums being completed, our user could end up being sent wrong or mis-ordered data.

The solution here is to use the Redis WATCH command in conjunction with a transaction. WATCH instructs Redis to monitor a specific key for any changes. If another client or connection modifies our watched key between our WATCH instruction and our subsequent transaction's EXEC , the transaction will fail and return a nil reply. If no client changes the value before our EXEC , the transaction will complete as normal. We can execute our code in a loop until the transaction is successful.

File: albums.go package main ... func FindTopThree() ([]*Album, error) { conn := pool.Get() defer conn.Close() // Begin an infinite loop. In a real application, you might want to // limit this to a set number of attempts, and return an error if // the transaction doesn't successfully complete within those // attempts. for { // Instruct Redis to watch the likes sorted set for any changes. _, err := conn.Do("WATCH", "likes") if err != nil { return nil, err } // Use the ZREVRANGE command to fetch the album ids with the // highest score (i.e. most likes) from our 'likes' sorted set. // The ZREVRANGE start and stop values are zero-based indexes, // so we use 0 and 2 respectively to limit the reply to the top // three. Because ZREVRANGE returns an array response, we use // the Strings() helper function to convert the reply into a // []string. ids, err := redis.Strings(conn.Do("ZREVRANGE", "likes", 0, 2)) if err != nil { return nil, err } // Use the MULTI command to inform Redis that we are starting // a new transaction. err = conn.Send("MULTI") if err != nil { return nil, err } // Loop through the ids returned by ZREVRANGE, queuing HGETALL // commands to fetch the individual album details. for _, id := range ids { err := conn.Send("HGETALL", "album:"+id) if err != nil { return nil, err } } // Execute the transaction. Importantly, use the redis.ErrNil // type to check whether the reply from EXEC was nil or not. If // it is nil it means that another client changed the WATCHed // likes sorted set, so we use the continue command to re-run // the loop. replies, err := redis.Values(conn.Do("EXEC")) if err == redis.ErrNil { log.Println("trying again") continue } else if err != nil { return nil, err } // Create a new slice to store the album details. albums := make([]*Album, 3) // Iterate through the array of response objects, using the // ScanStruct() function to assign the data to Album structs. for i, reply := range replies { var album Album err = redis.ScanStruct(reply.([]interface{}), &album) if err != nil { return nil, err } albums[i] = &album } return albums, nil } }

Using this from our web application is nice and straightforward:

File: main.go package main import ( "fmt" "log" "net/http" "strconv" "time" "github.com/gomodule/redigo/redis" ) func main() { pool = &redis.Pool{ MaxIdle: 10, IdleTimeout: 240 * time.Second, Dial: func() (redis.Conn, error) { return redis.Dial("tcp", "localhost:6379") }, } mux := http.NewServeMux() mux.HandleFunc("/album", showAlbum) mux.HandleFunc("/like", addLike) mux.HandleFunc("/popular", listPopular) log.Println("Listening on :4000...") http.ListenAndServe(":4000", mux) } ... func listPopular(w http.ResponseWriter, r *http.Request) { // Unless the request is using the GET method, return a 405 'Method Not // Allowed' response. if r.Method != http.MethodGet { w.Header().Set("Allow", http.MethodGet) http.Error(w, http.StatusText(405), 405) return } // Call the FindTopThree() function, returning a return a 500 Internal // Server Error response if there's any error. albums, err := FindTopThree() if err != nil { http.Error(w, http.StatusText(500), 500) return } // Loop through the 3 albums, writing the details as a plain text list // to the client. for i, ab := range albums { fmt.Fprintf(w, "%d) %s by %s: £%.2f [%d likes]

", i+1, ab.Title, ab.Artist, ab.Price, ab.Likes) } }

One note about WATCH : a key will remain WATCH ed until either we either EXEC (or DISCARD ) our transaction, or we manually call UNWATCH on the key. So calling EXEC , as we do in the above example, is sufficient and the likes sorted set will be automatically UNWATCH ed.

Making a request to the GET /popular route should now yield a response similar to: