Service discovery and remote procedure calls (RPC) are big subjects. There are many existing solutions that require varying degrees of infrastructure (e.g. message queues, Consul) or architectural decisions (e.g. Erlang, Akka, etc.). In this post, I'll introduce a minimalistic Go library called sleuth that creates an ad hoc peer-to-peer network requiring almost no configuration and only a couple lines of code to existing services and clients.

sleuth is a library that lets web services announce themselves to each other and to other clients on a LAN. It allows peers to send requests to each other without having to coordinate where each one lives and what port it uses. It works without an external process because under the hood, it is powered by magic, *a.k.a.* ØMQ using the Gyre port of Zyre.

TL;DR If you already feel comfortable with these topics and just want to jump straight to usage, skip directly to the sleuth example. Or, if you just want to see how to convert a server and a client that use HTTP requests to communicate with each other into peers on a sleuth network, check out this pull request.

If you're not familiar with what ØMQ is or why I'm calling it magic, here's the top-level Wikipedia definition:

ZeroMQ (also spelled ØMQ, 0MQ or ZMQ) is a high-performance asynchronous messaging library, aimed at use in distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker. The library's API is designed to resemble that of Berkeley sockets.

Introducing the problem

Microservices are self-contained services that expose only a small, focused API. In REST services, this usually means that a particular microservice will answer some of the endpoints of an API, delegated to it by a reverse proxy. The idea is that the full API is composed of multiple microservices residing behind the proxy and each one only deals with a well-defined set of concerns and its own data source.

I wrote sleuth because I needed a group of services on the same LAN to be able to find and to make requests to each other within a larger application. Since the problem is a general one, sleuth is completely agnostic about what web framework, if any, that peers are using. As long as a service can expose an http.Handler , then it can be a sleuth service. Client-only peers that only consume services are similarly trivial; sleuth 's API accepts native http.Request objects to make requests to peers on the network.

To understand the use case for sleuth , I'll propose two simple services that need to communicate.

Sample service 1: article-service

The first service is an article-service and it answers requests to paths that begin with /articles . Its purpose is to serve the meta information for articles.

Here is a representative request:

GET /articles/049cd8fc-a66b-4a3d-956b-7c2ab5fb9c5d HTTP / 1.1

It produces this response:

HTTP / 1.1 200 OK Content-Length : 304 Content-Type : application/json

The second service is a comment-service and its purpose is to return the comments for an article. It answers requests to paths that begin with /comments .

For example, here is a request for the comments related to the article above:

GET /comments/06500da3-f9b0-4731-b0fa-fbc6cbe8c155 HTTP / 1.1

And the comment-service responds:

HTTP / 1.1 200 OK Content-Length : 232 Content-Type : application/json

So what if the article-service accepts a query string parameter includecomments which denotes whether an article's metadata should include its comments?

GET /articles/049cd8fc-a66b-4a3d-956b-7c2ab5fb9c5d?includecomments=true HTTP / 1.1

Here is the response that would result if the article-service had a way to communicate with the comment-service :

HTTP / 1.1 200 OK Content-Length : 533 Content-Type : application/json

Naive implementation using HTTP requests

Since sleuth can work with pretty much every Go web framework because all it needs is an http.Handler to share services, I'll use one of the more popular web packages, Gorilla, for URL routing.

The list of articles comes from my saved articles on Hacker News. The comments are the top-level comments for each of those articles. I used the Hacker News API on Firebase, now defunct, to save them.

Note that these examples are simplified and I'll hold all of the data in memory in a map . Obviously, in the real world, the data would reside a separate database of some sort for each service. I'll use the init function to populate the data from a local file for each service.

Both because API responses should be uniform and because services need to know how to read the data output by one another, I'll create a shared types package to define API responses.

package types // Article holds the metadata for an article. type Article struct { GUID string ` json:"guid" ` Byline string ` json:"byline" ` Comments []*Comment ` json:"comments,omitempty" ` Headline string ` json:"headline" ` URL string ` json:"url" ` Timestamp int64 ` json:"time" ` } // ArticleResponse is the format of article-service responses. type ArticleResponse struct { Success bool ` json:"success" ` Message string ` json:"message,omitempty" ` Data *Article ` json:"data,omitempty" ` } // Comment holds the text and metadata for an article comment. type Comment struct { GUID string ` json:"guid" ` Article string ` json:"article" ` Text string ` json:"text" ` Timestamp int64 ` json:"time" ` } // CommentResponse is the format of comment-service responses. type CommentResponse struct { Success bool ` json:"success" ` Message string ` json:"message,omitempty" ` Data []*Comment ` json:"data,omitempty" ` }

Since it does not depend on being able to access any other service, I'll start with the comment-service . To simplify things, it will only support GET requests like the one shown above and all the data will be stored in a flat file.

Here is a first pass at its implementation.

package main import ( // Standard library imports are omitted for readability. // See GitHub for complete source code. "github.com/afshin/sleuth-example/types" "github.com/gorilla/mux" ) var data = make ( map [ string ][]*types.Comment) // Key is article GUID. func init () { // Data loading code is omitted for readability. // See GitHub for complete source code. } func handler (res http.ResponseWriter, req *http.Request) { log. Println ( "GET " + req.URL. String ()) response := new (types.CommentResponse) guid := mux. Vars (req)[ "guid" ] if comments, ok := data[guid]; ok { response.Data = comments response.Success = true res. WriteHeader (http.StatusOK) } else { response.Success = false response.Message = guid + " not found" res. WriteHeader (http.StatusNotFound) } output, _ := json. Marshal (response) res. Header (). Set ( "Content-Type" , "application/json" ) res. Write (output) } func main () { router := mux. NewRouter () router. HandleFunc ( "/comments/{guid}" , handler). Methods ( "GET" ) fmt. Println ( "ready..." ) http. ListenAndServe ( ":9871" , router) }

The comment-service implementation is straightforward: it reads the contents of data.json , unmarshals the data and stores it in a local map in memory which is grouped by article GUID. When GET requests come in, it tries to locate the comments for a given GUID. If they are found, it returns a successful response, if they are not found it returns a 404.

Implementing article-service

Like the comment-service , the article-service also has its data stored in a flat file. If the includecomments query string parameter is set to true , its handler makes an HTTP request to find the comments for a given article.

package main import ( // Standard library imports are omitted for readability. // See GitHub for complete source code. "github.com/afshin/sleuth-example/types" "github.com/gorilla/mux" ) const commentsURL = "http://localhost:9871/comments/%s" var ( client = new (http.Client) data = make ( map [ string ]*types.Article) // Key is article GUID. ) func init () { // Data loading code is omitted for readability. // See GitHub for complete source code. } func getData (guid string , includeComments bool ) (article *types.Article) { datum, ok := data[guid] if !ok { return } // Data source is immutable, so copy the data. article = &types.Article{ GUID: datum.GUID, Byline: datum.Byline, Headline: datum.Headline, URL: datum.URL, Timestamp: datum.Timestamp} if !includeComments { return } url := fmt. Sprintf (commentsURL, guid) req, _ := http. NewRequest ( "GET" , url, nil ) if res, err := client. Do (req); err == nil { response := new (types.CommentResponse) if err := json. NewDecoder (res.Body). Decode (response); err == nil { article.Comments = response.Data } } return } func handler (res http.ResponseWriter, req *http.Request) { log. Println ( "GET " + req.URL. String ()) response := new (types.ArticleResponse) guid := mux. Vars (req)[ "guid" ] include := strings. ToLower (req.URL. Query (). Get ( "includecomments" )) if article := getData (guid, include == "true" ); article != nil { response.Data = article response.Success = true res. WriteHeader (http.StatusOK) } else { response.Success = false response.Message = guid + " not found" res. WriteHeader (http.StatusNotFound) } output, _ := json. Marshal (response) res. Header (). Set ( "Content-Type" , "application/json" ) res. Write (output) } func main () { router := mux. NewRouter () router. HandleFunc ( "/articles/{guid}" , handler). Methods ( "GET" ) fmt. Println ( "ready..." ) http. ListenAndServe ( ":9872" , router) }

Solving the problem with sleuth

The solution above presents a few big problems that sleuth can solve:

What happens if the comment-service port or location changes? There either needs to be additional complexity in deployment configuration or an external service that manages communication. Relatedly, what if there are multiple instances of comment-service running? Either an external service (like a proxy) needs to reside between the two services or I need to write some custom logic. How can I guarantee that article-service won't even start until comment-service is available?

Before continuing, I assume that you've already installed libzmq on your system or are using a Docker container that comes with Go and ØMQ. See sleuth installation for more information.

The pull request that switches from HTTP to sleuth is straightforward and only a few lines of code for each service. I'll discuss these changes below.

Again, I'll start with the comment-service because it has no dependencies. First, I import sleuth and create a global client variable:

package main import ( // Imports are omitted for readability. "github.com/ursiform/sleuth" ) var ( client *sleuth.Client data = make ( map [ string ][]*types.Comment) // Key is article GUID. )

Next, I update the main function to instantiate client and join the sleuth network as a service called comment-service , which uses the Gorilla router as its handler, because it conforms to the http.Handler interface:

func main () { router := mux. NewRouter () router. HandleFunc ( "/comments/{guid}" , handler). Methods ( "GET" ) // In the real world, the Interface field of the sleuth.Config object // should be set so that all services are on the same subnet. config := &sleuth.Config{Service: "comment-service" , Handler: router} client, _ = sleuth. New (config) fmt. Println ( "ready..." ) http. ListenAndServe ( ":9871" , router) }

And that's it, the comment-service can now be accessed by the article-service .

To update the article-service , I make some similar changes. First, I import the library, change the commentsURL to be a sleuth:// URL, and change the client variable to be a *sleuth.Client instead of an *http.Client .

package main import ( // Imports are omitted for readability. "github.com/ursiform/sleuth" ) const commentsURL = "sleuth://comment-service/comments/%s" var ( client *sleuth.Client data = make ( map [ string ]*types.Article) // Key is article GUID. )

And finally, I instantiate the client variable, and as a nice-to-have, I'll tell the article-service to wait until it has found at least one instance of a comment-service before it starts up:

func main () { router := mux. NewRouter () router. HandleFunc ( "/articles/{guid}" , handler). Methods ( "GET" ) // In the real world, the Interface field of the sleuth.Config object // should be set so that all services are on the same subnet. client, _ = sleuth. New (&sleuth.Config{}) client. WaitFor ( "comment-service" ) fmt. Println ( "ready..." ) http. ListenAndServe ( ":9872" , router) }

Crucially, the HTTP request logic does not need to change at all because sleuth uses the same semantics for making HTTP requests. It accepts a native http.Request object in a function called Do just like the native http.Client so it functions as a drop-in replacement for http.Client requests.

Conclusion

If your system is already large enough that you have a custom message-queue based solution or are using an external service to manage microservices, then sleuth may not be right for you. However, if you're building up a set of services and want the flexibility to add them organically and have them find each other in an ad hoc network, then sleuth will simplify building services that operate in a distributed environment.

sleuth is a library, not a framework, and it's compatible with any Go web framework that uses the native http.Handler interface; so it is unobtrusive when it comes to your architecture decisions.

The example code for this tutorial is separated into two branches:

The http branch is the original implementation that makes HTTP requests between the two services and assumes that the comment service is on the same machine and its port never changes.

The master branch branch solves this dependency by creating a sleuth network between the two services.

Again, here is the pull request that switches from HTTP to sleuth .

The sleuth API documentation is available on GoDoc.