I had to capture metrics from a service I couldn’t directly access. The go standard library ships with a reverse proxy implementation 🤯. By proxying client requests to the service I was able to capture metrics without modifying the service itself.

What is a Reverse Proxy?

“In computer networks , a reverse proxy is a type of proxy server that retrieves resources on behalf of a client from one or more servers. These resources are then returned to the client, appearing as if they originated from the proxy server itself. “ From Wikipedia

Essentially, a reverse proxy forwards traffic from a client to a set of servers behind the proxy. There are many applications for reverse proxies. Load balancing, TLS termination, and A/B testing are just a few. Reverse proxies are also useful for inserting instrumentation around an HTTP service without having to modify the service itself.

reverse proxy network diagram

If you’d like to learn more about proxying I recommend checking out Introduction to modern network load balancing and proxying by Matt Klein. Matt is the creator of Envoy Proxy, a robust proxy server that powers service mesh tools like Istio. His post does a great job of outlining the approaches used by modern load balancers and proxies.

Simple Go Reverse Proxy

Go is one of my favorite programming languages for many reasons. The designers of the language focused on Simplicity, practicality, and performance. These considerations make Go a joy to use. The language shines with networking tasks. Part of the reason for this is the incredibly comprehensive standard library, which among other common implementations includes a reverse proxy.

Rolling your own proxy in go is as simple as

Yep, that’s it. Let’s dig in here. The httputil.NewSingleHostReverseProxy method returns a ReverseProxy struct containing the following method.

All we need to do is configure the proxy and wire it up to a standard go HTTP server to have a working reverse proxy as shown below.

That’s it! This server can proxy HTTP requests and web socket connections. You’ll notice that I’ve configured the proxy.Director field. The ReverseProxy.Director is a function that modifies the incoming request before it is forwarded. The signature is as follows:

A common use case for the director function is modifying request headers. One of the principles of the Go programming language is that types should have sane defaults and be immediately usable. Following this principle, the default director implementation returned by httputil.NewSingleHostReverseProxy takes care of setting the request Scheme, Host, and Path. I didn’t want to duplicate the code, so I wrapped that implementation. Note: I had to reset the req.Host field to handle HTTPS endpoints. I’ve also included an example of setting a request header via req.Header.Set which will override the header value with the value passed into the method.

Capturing Metrics

Let’s extend our simple proxy to read and report metrics about the downstream service responses. To do this we’ll return to the httputil.ReverseProxy struct once more. It exposes a struct field ReverseProxy.ModifyResponse which gives us access to the HTTP response before it goes back to the client.

Go implements HTTP bodies as io.Reader’s and, therefore, you may only read them once. If you would like to parse a request or response before forwarding it you will need to copy the body into a byte buffer and reset the body. An obvious drawback is that we buffer the entire response in memory without limit. This could lead to memory issues in production if you received a large response but for my use case this wasn’t an issue. Here’s a quick implementation to parse and reset the response body.

With the request body problem solved, capturing metrics is simple.

And that’s it! The capture metrics function was pretty specific to my use case so I’ll leave it up to you to implement. I was proxying a deep learning model that was available via an HTTP endpoint. I ended up using the Prometheus client library to export metrics about predicted class labels from the model.

The full code for the metrics capturing proxy is as follows

Go’s standard library did all the heavy lifting. From here you could extend the proxy to any number of use cases.

If you found this at all useful leave a comment below.