For a specific high-performance workloads, I wanted to include a new and highly optimized endpoint onto Roundtrip.

If you don't know what Roundtrip is yet, feel free to quickly check out the previous Roundtrip post and come back once you got the idea of what it does.

I had to select both a wire protocol and an actual transport that will be very efficient. To gain an even higher margin over HTTP, I knew I wanted it to be at least binary and not very chatty.

A good option for this would be Thrift, for example. However I wanted to go as low as I could, because I didn't really need anything more than the bare simplest RPC mechanism.

However, going with straight up TCP wouldn't gain me much because I typically hold development ease and maintainability as an additional value. There was only one thing I felt offering an awesome development model and being as close to (or even better than, on some occasions) TCP...

ZeroMQ

If you haven't yet heard of ZeroMQ I urge you to check it out. The programming model is great, and the documentation is very accessible and interesting.

Installing it should be a breeze given that you pick the right combination of library version and gem. It may vary based on your platform, but the end goal is to install a binary library that ffi-rzmq will bind to.

On Linux in this case, I chose to install the older (time wise, it isn't THAT old) 2.2 series ZeroMQ variant.

The 2.2.0 ZeroMQ variant, from my experience, offered the smoothest experience for me with ffi-rzmq and everything just worked.

Before building from source, let's make sure we have the proper dependencies:

sudo apt-get install uuid-dev build-essential

Get the 2.2.0 sources, and build using the typical ./configure && make && sudo make install .

ffi-rzmq

Next, you should gem install ffi-rzmq whether on MRI or JRuby. To interoperate with my existing Ruby code, I chose ffi-rzmq because it offered the best compatibility with MRI and JRuby.

You should be aware though, that some Ruby examples in the ZeroMQ examples repository are not using ffi-rzmq and may be incompatible with it to varying degrees. Despite of that, I still think ffi-rzmq is the way to go.

Building a ZeroMQ Server

First, ZeroMQ is not a queue system, nor a queue broker.

ZeroMQ provides you with building blocks and messaging patterns that you can compose and form your own network topologies. The ZeroMQ Guide is the best place to learn about best-practices and networking patterns that you can use for your own projects.

REQuest / REPly

With ZeroMQ, REQ/REP stands for the Request / Reply messaging pattern. The simplest client-server topology you can probably think of. That is, the client makes a request, and the server replies in a synchronous, lockstep fashion.

Within Roundtrip's ZeroMQ endpoint, the communication model would be very simple. The Client sends a Command, the Server performs it synchronously (well, Redis is quite fast) and replies with a JSON-serialized response.

The protocol is simple. A string describing a one-letter action and method parameters.

<action> <*params.join(' ')>

For example:

S metric.name.foo i-have-an-id-optional

Is replied with

{"id":"id-xyz","route":"invoicing","started_at":"2012-11-30T18:23:23.814014+02:00"}

To understand how simple and powerful the ZeroMQ programming model is in the case of REQ/REP, here is the core of the request handling in the server code:

request = '' rc = socket . recv_string ( request ) # poor man's RPC unless request && request . length > 2 socket . send_string ({ :error => "bad protocol: [ #{ request } ]" } . to_json ) return end action , params = ACTIONS [ request [ 0 ]] , request [ 1 . . - 1 ]. strip . split ( /\s+/ ) begin resp = @core . send ( action , * params ) socket . send_string ( resp . to_json ) rescue puts "error: #{ $! } " if @debug socket . send_string ({ :error => $! } . to_json ) end

Things to note here are the symmetrical communication: after any recv there's a send . In addition, ZeroMQ handles all of TCP connections and juggling for you; you can take the leap-of-faith and all you do is talk to a socket sending a stream of messages.

At the other end, a client is responsible to send a request and handle the reply.

The trick with REQ/REP, or with any ZeroMQ messaging patterns for that matter, is to be able to address reliability concerns.

Even though the server code is simple, the client takes much of the complexity through handling reliability in the form of retries and timeouts:

def send ( message ) raise ( "Send: #{ message } failed" ) unless @socket . send_string ( message ) @retries . times do if @poller . poll ( @timeout ) > 0 s = '' @socket . recv_string s yield s return else close reconnect end end raise 'Server down' end end

Conclusion

Was it worth it?

In short, yes: using ZeroMQ to implement an RPC in this case was as close as using the code itself without a transport overhead at all.

Running 1000 transactions:

ZeroMQ: 4.876783s Code: 4.284264s

You can checkout the benchmarks here, and of course - take note that ZeroMQ benchmarking was done on a local machine, it doesn't take into account network overhead (just the bare bones nuts and bolts of ZeroMQ as a transport).

HTTP over ZeroMQ

You should also check out technoweenie/faraday-zeromq where Github's @technoweenie created a middleware over Faraday that transcodes HTTP over ZeroMQ. His benchmarks reveal similar advantages.