04 Jan 2014

As capnproto-rust approaches full feature support for Cap’n Proto serialization, now is an apt time to validate its usefulness on a slightly more involved example.

Thus I present zmq-explorers, a toy data pipeline which uses ZeroMQ as a transport layer.

The pipeline looks like this:

At the input end are any number of “explorer” nodes gathering data. In the middle is a “collector” node aggregating and processing the data. At the end is a “viewer” node consuming the processed data. The explorers communicate to the collector via a publish/subscribe pattern, and the viewer communicates with the collector via a request/reply pattern. ZeroMQ makes this communication quite simple to express, and Cap’n Proto operates directly on the bytes moved by ZeroMQ — no copying or parsing is necessary.

Concretely, the observations of the explorers are color values from an image:

struct Observation { timestamp @0 : Int64; x @1 : Float32; y @2 : Float32; red @3 : UInt8; green @4 : UInt8; blue @5 : UInt8; diagnostic : union { ok @6 : Void; warning @7 : Text; } }

Each explorer randomly moves within the two-dimensional space of the image, reporting the colors it observes, fudged by some noise.

The collector maintains a grid that aggregates the reported observations:

struct Grid { cells @0 : List(List(Cell)); numberOfUpdates @1 : UInt32; latestTimestamp @2 : Int64; struct Cell { latestTimestamp @0 : Int64; numberOfUpdates @1 : UInt32; meanRed @2 : Float32; meanGreen @3 : Float32; meanBlue @4 : Float32; } }

The collector sends its current grid to the viewer whenever it is requested.

Here is what the viewer might see if there are three explorers:

The image on left shows the current best estimate of the data. The image on the right shows a crude measure of confidence in the estimate; green represents the value of the numberOfUpdates field, and blue represents the value of the latestTimestamp field relative to the current time.

-- posted by dwrensha