In this article we introduce the big picture of CoralSequencer, a full-fledged, ultra-low-latency, high-reliability, software-based middleware for the development of distributed systems based on asynchronous messages. We discuss CoralSequencer’s main parts and how it uses a sophisticated and low-latency protocol to distribute messages across nodes through reliable UDP multicast.





You should also check the YouTube video below presented by Brian Nigito from Jane Street describing the philosophy behind the CoralSequencer architecture.

Fundamentals

What is a distributed system?

It is an integrated system that spans multiple machines, clients or nodes which execute their tasks in parallel in a non-disruptive way. A distributed system should be robust enough to continue to operate when one or more nodes fail, stop working, lag or are taken down for maintenance. Each machine can run multiple nodes, not just one. Each node communicates with each other by sending messages to the sequencer (i.e. broker). Nodes can produce and consume messages.

It is an integrated system that spans multiple machines, clients or which execute their tasks in parallel in a non-disruptive way. A distributed system should be robust enough to continue to operate when one or more nodes fail, stop working, lag or are taken down for maintenance. Each machine can run multiple nodes, not just one. Each node communicates with each other by sending messages to the sequencer (i.e. broker). Nodes can produce and consume messages. What is a Messaging Queue or MQ?

It is a message-oriented middleware that enables the development of a distributed system by using asynchronous messages for inter-node communication . A traditional approach for message distribution is the publish-subscribe pattern. CoralSequencer uses a different all-messages-to-all-nodes approach, through multicast, by implementing a reliable UDP protocol.

It is a message-oriented middleware that enables the development of a distributed system by using asynchronous messages for . A traditional approach for message distribution is the publish-subscribe pattern. CoralSequencer uses a different approach, through multicast, by implementing a reliable UDP protocol. What are distributed systems built on top of a MQ good at?

They are great at implementing applications with a lot of moving parts (i.e. nodes) which need real-time monitoring and real-time reaction capabilities. To summarize they provide: parallelism (nodes can truly run in parallel), tight integration (all nodes see the same messages in the same order), decoupling (nodes can evolve independently), failover/redundancy (when a node fails, another one can be running and building state to take over immediately), scalability/load balancing (just add more nodes), elasticity (nodes can lag during activity peaks without affecting the system as a whole) and resiliency (nodes can fail / stop working without taking the whole system down).

They are great at implementing applications with a lot of moving parts (i.e. nodes) which need real-time monitoring and real-time reaction capabilities. To summarize they provide: parallelism (nodes can truly run in parallel), tight integration (all nodes see the same messages in the same order), decoupling (nodes can evolve independently), failover/redundancy (when a node fails, another one can be running and building state to take over immediately), scalability/load balancing (just add more nodes), elasticity (nodes can lag during activity peaks without affecting the system as a whole) and resiliency (nodes can fail / stop working without taking the whole system down). What is the advantage of CoralSequencer?

CoralSequencer is a state-of-the-art implementation of a messaging queue for ultra-low-latency distributed systems that require a high degree of reliability. It has all the Coral Blocks’ advantages: high performance, low latency, low variance, zero garbage and simplicity. As you will see in subsequent articles, CoralSequencer is extremely easy to develop, configure, deploy, monitor and maintain.

CoralSequencer is a state-of-the-art implementation of a messaging queue for ultra-low-latency distributed systems that require a high degree of reliability. It has all the Coral Blocks’ advantages: high performance, low latency, low variance, zero garbage and simplicity. As you will see in subsequent articles, CoralSequencer is extremely easy to develop, configure, deploy, monitor and maintain. What are some examples of systems that can be built on top of CoralSequencer?

Electronic exchanges (ECNs), trading platforms, banking systems, automated online advertising systems, defense/military systems, credit card systems or any distributed system with a large number of transactions that requires all the advantages listed above plus ultra low latency and high reliability.

Quick Facts

All nodes read all messages in the exact same order, dropping messages they are not interested in

All messages are persisted so late-joining nodes can catch up and build the exact same state as other nodes

Message broadcasting is done through a reliable UDP protocol: no message is ever lost

No single point of failure, from the software down to the hardware infrastructure

Each session is automatically archived with all its messages so it can be replayed later for testing, simulation, analysis and auditing

High-level, straightforward API makes it easy to write nodes that publish/consume messages

As low-latency as it can be through UDP multicast

Zero garbage created per message – no gc overhead

Quick Features

Message agnostic – send and receive anything you want

Automatic node discovery through multicast, making it easy to move your nodes across machines

Message fragmentation at the application level, so you can send messages of any size transparently

Comprehensive test framework for deterministic single-threaded automated tests

Support for TCP and UDP rewind

Transparent batching and in-flight messages

Full duplex bridges (UDP and TCP)

Sequencer-generated messages

Hot-hot nodes

Multiple sequencers in parallel with cross-connect nodes

TCP, UDP and Memory transport support

Remote administration

Logger node

Backup node

Admin Node

And many others

Node Example





package com.coralblocks.coralsequencer.node; import com.coralblocks.coralsequencer.message.Message; import com.coralblocks.coralsequencer.mq.Node; import com.coralblocks.coralreactor.nio.NioReactor; import com.coralblocks.coralreactor.util.Configuration; public class SampleNode extends Node { private final static int PERIOD = 3000; // 3 seconds... private long time; public SampleNode(NioReactor nio, String name, Configuration config) { super(nio, name, config); } @Override protected void handleActivated() { // this method is called when the node becomes active sendCommand(); } @Override protected void handleEventTimeout(long now, int period) { // this method is triggered by the event timeout you are setting in the handleMessage method sendCommand(); } private void sendCommand() { time = System.nanoTime(); sendCommand("TIME"); } @Override protected void handleMessage(boolean isMine, Message msg) { if (!isMine || isRewinding()) return; // not interested, drop it System.out.println("Took " + (System.nanoTime() - time) + " nanoseconds to see my message in the bus"); setEventTimeout(PERIOD); // set a trigger to send the command again after 3 seconds (goes like that forever) } }