Purpose

Livy is an open source component to Apache Spark that allows you to submit REST calls to your Apache Spark Cluster. You can view the source code here: https://github.com/cloudera/livy

In this post I will be going over the steps you would need to follow to get Livy installed on a Hadoop Cluster. The steps were derived from the above source code link, however, this post provides more information on how to test it in a more simple manner.

Install Steps

Determine which node in your cluster will act as the Livy server Note: the server will need to have Hadoop and Spark libraries and configurations deployed on them. Login to the machine as Root Download the Livy source code cd /opt wget https://github.com/cloudera/livy/archive/v0.2.0.zip unzip v0.2.0.zip cd livy-0.2.0 Get the version of spark that is currently installed on your cluster Run the following command spark-submit --version Example: 1.6.0 Use this value in downstream commands as {SPARK_VERSION} Build the Livy source code with Maven /usr/local/apache-maven/apache-maven-3.0.4/bin/mvn -DskipTests=true -Dspark.version={SPARK_VERSION} clean package Your done!

Steps to Control Livy

Get Status

ps -eaf | grep livy

It will be listed like the following:

root 9379 1 14 18:28 pts/0 00:00:01 java -cp /opt/livy-0.2.0/server/target/jars/*:/opt/livy-0.2.0/conf:/etc/hadoop/conf: com.cloudera.livy.server.LivyServer

Start

Note: Run as Root

cd /opt/livy-0.2.0/ export SPARK_HOME=/usr/lib/spark export HADOOP_CONF_DIR=/etc/hadoop/conf ./bin/livy-server start

Once started, the Livy Server can be called with the following host and port:

http://localhost:8998

If you’re calling it from another machine, then you will need to update “localhost” to the Public IP or Hostname of the Livy server.

Stop

Note: Run as Root

cd /opt/livy-0.2.0/ ./bin/livy-server stop

Testing Livy

This assumes you are running it from the machine where Livy was installed. Hence why we’re using localhost. If you would like to test it from another machine, then you just need to change “localhost” to the Public IP or Hostname of the Livy server.