Onyx Design & Dashboard Introduction

Onyx is a fault tolerant distributed computation system in the mold of Storm or Spark, which can be used for batch and stream processing. As of 0.5.0, Onyx is implemented using a masterless design, i.e. there is no single process or entity that coordinates the peers. Rather, peers in Onyx coordinate via a shared, immutable, log that is written to ZooKeeper. In addition, where possible Onyx’s design favours data over code, which leads to some interesting run time and monitoring behaviour.

When Onyx peers start up, they replay this shared log until they are up to date with the current state (replica) of the cluster. They read the log in order, one entry at a time, and continue to update their local replica of cluster state by applying the changes. These log messages may result in job management (starting, killing jobs), job scheduling (peers allocated to a job), and task scheduling (scheduling between tasks within a job), and any other cluster coordination that may be required.

A side benefit of this design is the resulting replayable view of changes within the cluster over the time, in addition to view the overall state of the cluster at a given time, even if the cluster has been shutdown. We have leveraged this information model in order to create a dashboard that gives a high level overview of the state of an Onyx cluster.

Operation

On loading the dashboard a dropdown is presented, containing deployment IDs, each of which corresponds to a log for an Onyx cluster/deployment. When a deployment ID is selected, the dashboard server replays the log. As it reads the log messages, it applies the log entry to its current replica of the cluster state.

The dashboard interprets these lower level log messages, in addition to the current replica, in order to send higher level messages to the dashboard e.g. a job was started, a task is running, a job is killed. These higher level messages and the raw log entry are sent to the dashboard client via a websocket. As the log is replayed the dashboard view updates until it is up to date with approximately the last log entry (i.e. a Zookeeper read was made and no log entries were found).

Dashboard Selection & Log Replay

The animation above shows the selection of an Onyx deployment, and the view as the log for the deployment is replayed. The raw log entries for the deployment can be seen in the “Raw Cluster Activity” section of this view.

Dashboard Status

The Dashboard Status section shows the time of the last read log entry. As log messages are replayed, the status of the dashboard will advance until it is up to date with the cluster. When the dashboard is up to date with the cluster replica, the panel turns grey to signal that it has fully replayed the log.

Deployment Peers

Displays the cluster peers that are running for the deployment. An additional heartbeat check is performed by the dashboard in order to ensure that the peers described by the log are still alive.

Jobs Selector

This section section displays the jobs that have been started for a replica, sorted by their start date. Clicking on a job will enable additional job related panels that show information about the job in question, and also filters the raw log entries to only show log entries relating to that job.

Jobs Management

The Job Management section provides functionality to manage jobs. This allows users to kill jobs by writing a “kill job” entry to the log which will be read by the peers. It also allows for jobs to restarted by killing the job, assuming it is still running, and then start a new job using the same workflow and catalog data as the job that was killed.

Job Status

The Job Status section shows the current status of the job (pending, running, incomplete, finished, and killed), and the task scheduler used (currently round robin and percentage are available).

Running Tasks and Peers

The Running Tasks section shows the workflow tasks that are being processed by the cluster, as well as the peers that are processing them. In the above example, :read-rows is being processed by multiple peers.

Workflow

The Workflow section shows the workflow for the selected job. Workflows describe the DAG dataflow graph for the job’s computation. One of the key benefits to the design of Onyx is that the workflow design is flexible and is pure data, submitted when running a job, resulting in the data being serialised to the log where it is then read by the peers. The dashboard reads the workflow from the log to display to users.

Catalog

Jobs under Onyx must also be submitted with a catalog, which describes the tasks included in the corresponding workflow for the job. Information contained in the catalog might include the function run for a task, some parameters supplied to the function that is run, or possibly some information required to initialise the task e.g. the sql provider that should be used by an SQL input plugin.

Log Entries Filtered by Selected Job

The Log Entries section displays the raw log entries for the deployment, or for the job that is selected. These allow users to see the raw activity for the cluster

Future Work

We’ve created a basic dashboard that will be useful for many monitoring use cases, and some debugging use cases.

There is more that to be done. As Onyx jobs are described as data, there is the potential to allow for client side editing, manipulation, and submission of jobs. For example, debug mode jobs could be run, which would place additional nodes in between each workflow node, which could stream the intra-workflow segment results to the dashboard client.

We are also looking into best practices for monitoring cluster metrics. Although this data would probably not be written to the cluster log, as it could reduce performance, we may provide integration between dashboard and metrics visualisation tools. The combination of log interpretation with cluster metrics would substantially increase the power of these metrics, as information about the cluster (peer joining or leaving, scheduler used, etc) can be combined with throughput and latency metrics.

If you are interested in Onyx, take a look and try out the Onyx starter tutorial, which includes instructions for dashboard use.

You can also sign up for an Onyx Newsletter by yours truly. You will receive infrequent news about Onyx releases and tooling improvements as they happen.