Our last post provided a technical overview of how you can deploy an IPython-driven Spark cluster using Docker containers for each component. That setup works well to deploy Spark in standalone mode, but what if you want to run other big data frameworks on the same cluster? With infrastructure tools like Apache Mesos, you can gain fine-grained resource utilization and collapse multiple frameworks — such as Hadoop, Spark, and ElasticSearch — into one cluster. To that end, we’ve been examining how a Dockerized Spark setup could use Mesos as its task scheduler. What follows is our current thinking on three possible ways that IPython, Spark, Docker, and Mesos can be made to work together.

Figure 1: Mesos resource sharing increases throughput and utilization, via Apache Mesos at Twitter

Three’s Company

Before diving into ways to combine Spark and Mesos with Docker, it will help to give a little background on how Spark typically integrates with Mesos. The interplay between the two systems is important, so bear with me for this quick overview.

Out of the box, it is relatively straightforward for Mesos to distribute Spark tasks to slave nodes. Following the guidance of several overviews and tutorials, the integration begins by first building the Spark binary. Next, you place that binary in an HDFS location each Mesos slave can reach (or alternatively, within the same local directory on each slave). From that point on, when Mesos slaves accept Spark jobs, they retrieve the binary from HDFS (or point to their local path install) to do Spark magic. The following configuration snippet illustrates how to configure Mesos to pull the Spark binary from HDFS:

export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so export SPARK_EXECUTOR_URI=hdfs://hadoop-namenode/tmp/spark-0.8.0–2.0.0-mr1-cdh4.4.0.tgz export MASTER=zk://hadoop-namenode:2181/mesos

Figure 2. Mesosphere Spark configuration: First, specify the location of the libmesos.so library. Second, define the URI for the Spark executor, which the Mesos slaves will run. Then define the URIs for the Zookeeper masters.

Once configured, the Spark client can use Mesos as its master, as the Spark Python programming guide explicitly states:

spark-submit supports launching Python applications on standalone, Mesos or YARN clusters, through its — master argument. However, it currently requires the Python driver program to run on the local machine, not the cluster

This setup should work great for vanilla Spark, but what about our interest in using PySpark with nonstandard Python modules like Pandas or Numpy? As was the impetus for our ipython-spark-docker project, that would require getting all the right, compatible Python packages to each slave. It looks like using Mesos gets right back to the starting point, where we’d need to do one of the following to make sure each slave has all the right Python packages:

Install Python and all required packages on each slave. Luckily, to manage versions and environments, you can use Anaconda on each machine if you didn’t want to do system-wide installs.

Use the Spark — py-files option to distribute Python packages via egg/zip files. When I attempted this before with Spark standalone mode, I encountered a few difficulties: 1) finding the right packaged module; 2) distributing the modules to all slaves when starting the Spark cluster. I’m sure that has a lot to do with my relative Spark inexperience and interest in having so many Python modules available. Either way, this isn’t a path I plan to revisit.

Making a long story short, things aren’t straightforward for getting our desired IPython-driven Spark setup working with Mesos out of the box. So what to do?

Option 1: Bare-metal with a dash of container

Just like before, this desire for portability and repeatability leads to Docker. Since there are several ways to combine these systems, I’ll next walk through three potential architectures for deployment.

Our first thought for Mesosizing a Dockerized Spark setup was to just install Mesos on bare-metal. Given a vanilla Mesos installation, the Mesos master should be able to accept Spark jobs, send them to the slave nodes, and hosts would then run containerized Spark workers. Voila, distributed analysis! Easy, right?

Not so fast. As near as I can tell, it might not be that straightforward to run Mesos on bare metal as the master for our containerized Spark in cluster mode. Our current Spark worker containers are configured for Spark standalone mode, where the workers register with a Spark master node when launched. Using Mesos as the master would require Mesos to launch the Spark worker containers as task executors. Luckily, Mesos v0.20.1 added the following magic:

Spark can make use of a Mesos Docker containerizer by setting the property spark.mesos.executor.docker.image in your SparkConf. The Docker image used must have an appropriate version of Spark already part of the image, or you can have Mesos download Spark via the usual methods.

That suggests it might be possible to extend our Spark worker image to include Mesos libraries, point our IPython-Spark client container at Mesos, and configure Spark to launch worker containers to execute Spark tasks. If this option works, it could be a solid way to benefit from Mesos and use our desired Python ecosystem inside Spark worker containers. It could also avoid all the network routing we needed for standalone mode.

Just in case this option has unforeseen issues, or in case we want to consider an alternate architecture in the future, there seem to be two additional options for combining Spark, Mesos, and Docker.

Option 2: One happy container

As outlined above, putting Spark inside Docker provides the ability to quickly spin up and teardown a Spark cluster. To build on that key benefit, it seems like the next step would be to consider a Dockerized Mesos setup similar to our standalone version of Spark-in-Docker containers. But those Mesos containers, of course, also need to include all the requisite Spark and Python libraries, which makes for a pretty beefy container.

Although overloading one Docker image makes me uneasy, I’m wondering if that would deliver a highly portable option for using Spark with Mesos on top of Docker. After all, it should be technically feasible to build a set of master and slave/worker containers that include both Spark and Mesos. Each container would then have all the necessary packages, configurations, and versions. The last step — shuttling communication between hosts and containers — would repeat the network routing work outlined in our last post, leading to a similar situation where each host runs one container and “transparently” routes traffic among the cluster hosts and containers.

If we did pursue the Path to Mordor (i.e. “one container to rule them all”), we could build on top of the heavy lifting others have already done to Dockerize Mesos. For example, this article and related GitHub repo looks like a solid way to “launch a fault tolerant multi node cluster with about seven Docker commands.” Merging this with our existing repo would take some effort, but we should be able to leverage prior work by adding libraries and configuring host-container routing.

That path is not without its perils, however. Cramming several frameworks into one container could become a slippery slope. As our containerized Spark project demonstrated, network routing (i.e. container1->host1->host2->container2) also isn’t the most straightforward undertaking. Adding Mesos into the mix only complicates matters.

Figure 3: Concept architecture for incorporating Mesos into the existing Docker image

Option 3: The (sort of) standalone deployment

Faced with the previous two options (everything in one container or just using Spark worker containers), our team brainstormed a possible third choice. As outlined above, Mesos can launch tasks that contain Docker images. Further, Marathon is a Mesos framework designed specifically to launch long-running applications using containers. So instead of putting Mesos+Spark into one container, and instead of deploying things on bare metal, could we try Running Docker Containers on Marathon? Therefore, instead of using the Mesos master to distribute Spark jobs to Mesos slaves, Marathon might be able to run the standalone ipython-spark-docker cluster as a service inside of Mesos. I haven’t seen anyone try this specific setup with Spark (and maybe for good reason), but it should be possible for Mesos to spawn Spark containers that would look, feel, and act like a standalone Spark cluster.

One downside of this approach is that we would probably lose some of the efficiencies gained by using Mesos as Spark’s master. Second, it would require the Mesos slaves to redirect a large portion of host ports to the Spark containers, which could break Mesos communication patterns or initiate dominos of errors that could be hard to debug. On that latter point, the standard ports for Mesos (5050/5051), Zookeeper (2181,2888,3888,…) and Marathon (customizable) do not appear to overlap with Spark, giving me hope that network routing might actually be possible. At this point, the only way to know for sure might be to try and see what works and where things break.

Figure 4: Concept architecture for running Dockerized Spark in standalone mode within a Mesos cluster

The path forward

The best path forward probably depends on specific needs for using Mesos with Spark with Docker. If Mesos can launch our Spark worker containers, keeping Mesos on metal would position it squarely as a piece of infrastructure and launch Spark jobs as an application (as intended). For those interested in maximizing benefits from Docker, the idea of containerizing Spark, Mesos, and all associated libraries should make it possible to quickly deploy (and/or rebuild) a cluster. On the other hand, despite the efficiencies gained through Mesos, adding several frameworks to one Docker container feels a bit messy. If that is too much, using Marathon to run the Standalone Spark containers as a service might be the option to consider.

Overall, it seems worthwhile to experiment and see where things fall over. We’d be interested in knowing whether anyone else has figured out a way to containerize both Mesos and Spark in a multi-node cluster. As of now, we plan to do the following:

Get the multi-node Mesos-in-Docker up and running on a few of our OpenStack nodes. We could just as easily try to install Mesos on metal, but I want to give that project a test drive. Test how easy it is for Mesos to launch our Spark worker containers to execute Spark jobs. See if we can use Mesos to spin up the ipython-spark-docker master and worker images to create a standalone Spark cluster within Mesos. Poke around to see what it would take to run an IPython-on-Spark-on-Mesos set of containers.

Stay tuned for a follow-up post after I finish the above steps. Until then, thanks for reading!