NOTE: you can also download this post in PDF format if you’re into that.

NOTE2: this post was originally written on July 2014. Was moved here after some online revamping.

Introduction

I’ve been using SmartOS for one year as my main development OS, and I want to share my experiences because SmartOS is a very atypical system and as such it requires an atypical style of usage. Well, atypical for me. Unlike all of the generations of engineers that came before me, I’ve been using Linux, OpenSolaris, and then OpenIndiana on stand-alone consumer hardware — desktops and laptops. This translates into me being the sole user of a system who’s usefulness is more or less independent of network connectivity. Having this kind of monopoly leads to very bad habits such as messing with global configs, installing custom software directly into /bin , storing personal files in /rpool/docs , not using Zones whenever possible, and so on. All in all, a very messy situation, that doesn’t bother anyone but should bother me. Naturally, I’ve been slow to change my heathen ways. In fact I didn’t change these habits until I was forced to by deciding to use SmartOS as my main development OS (yeah, it’s that compelling).

Hardware

First thing’s first: you probably won’t be able to run it on your desktop or laptop. It’s not that it can’t theoretically run on those systems — it’s very lean — it just has some problems with firmware and drivers and the like. As such it requires a very specific brand of motherboard: supermicro. Supermicro boards are server boards, which means you can only install SmartOS on a server. As I said, I never had the need for a server, so I had to part with some cash to get one from eBay. My current hardware is at the bottom of this post. Another thing to keep in mind is that the usability of your SmartOS server is directly correlated to the network connectivity. That’s kind of a downer, as it won’t always be available to you, but it will be available to you from potentially all devices that are connected to the internet — yes, I’ve used SmartOS from my phone.

SmartOS Basics

The other thing is that SmartOS is designed to be a hypervisor — a host for virtual machines. Most OSes can be such hosts in addition to being “regular” operating systems — like running VMWare or Parallels on top of Mac OS X. SmartOS can only be used to host VM’s and nothing more. It’s like being able to run VMWare on the Mac, but not being able to use the Mac for anything else — everything you do happens in one virtual machine or another.

The SmartOS host (also known as the global zone) runs entirely in RAM and is read-only. Areas like /bin , /etc , and /usr can never be changed. They can only be changed if you compile a custom SmartOS live USB. The only areas that can be changed are /opt and /usbkey . The latter holds the configuration for SmartOS while the former can be used to install optional software. If you need to use /opt you really need to use it. It’s typically used to start custom services during boot, though I’ve also seen people use it to install Xorg.

The main consequence of this kind of design is that the host is completely insulated from the VMs. The host can be upgraded without breaking anything in the VM itself. It also means that there is no root ZFS pool. To OpenSolaris and Solaris veterans this is huge, because it allows you to use RAID-Z on your system, where before you would have been constrained to mirroring — unless you were forward-looking enough to use separate boot and data drives.

The main challenge with SmartOS is figuring out how to separate your data from the host system, and how to make this data available to a multitude of virtual machines — many of which will need to be deleted and re-created. Furthermore, one has to figure out how to efficiently restore those re-created virtual machines to the usable state they were in before they were destroyed — this mostly refers to installed packages, config files, and the home directory. Also, one has to get used to hopping between various purpose-specific virtual machines, instead of just sticking to a single machine.

Separating Your Static Data

As I said before, I used to store my data — which consists of a vast collection of (legally obtained) PDF books, music files, personal software projects, and historical performance data — as a bunch of subdirectories in /rpool . This simply will not do in the new world order. The default ZFS pool in SmartOS is the zones pool. To start off, in the global zone, I created a non-conflicting directory heirarchy to hold all of my data:

zfs create zones/depot zfs set mountpoint=/depot zones/depot chmod a+rwx /depot

I broadly divide my software activities into two categories: analysis and synthesis. The analysis category consists of various software analysis data and notes that are persistent, and allows me to resume a line inquiry after a long break or distraction. Most of my source repositories have infrastructure for collecting and logging performance (and debug) data. However, this data will be overwritten the next time the framework is run, or the next time make clean is executed. In order to have historical performance data, the logs had to be dated and stored somewhere and thus the analysis data set was born. The synthesis data set contains everything that I create which is mostly software, blog posts, and essays. My collection of ebooks and music was located in /rpool/dlib and /rpool/music . This is what the transfer looked like from an OpenIndiana machine to a SmartOS machine:

zfs send -R rpool/synthesis@snap-67| ssh root@$smartos zfs recv -d zones zfs send -R rpool/analysis@snap-23 | ssh root@$smartos zfs recv -d zones zfs send -R rpool/dlib@snap-12 | ssh root@$smartos zfs recv -d zones zfs send -R rpool/music@snap-44 | ssh root@$smartos zfs recv -d zones zfs set mountpoint=/depot/synthesis zones/synthesis zfs set mountpoint=/depot/analysis zones/analysis zfs set mountpoint=/depot/dlib zones/dlib zfs set mountpoint=/depot/music zones/music chmod -R a+rwx /depot

There was also the neccessity to separate my home-files from the system. So I created a new data set just for this purpose:

zfs create zones/synthesis/home_files mkdir /depot/synthesis/home_file/skel

The skel directory contains all of the dot-files from the OpenIndiana install. Any changes to those dot-files has to happen in that directory. Then a script is used to deliver the files to a home directory. Keeping your files in a VM is a bad idea since (a) destroying that VM will destroy those files too (as well as the user that owns those files — in my case the user nickziv ) — and (b) sharing those files between VMs becomes less convenient.

You will want to automate the creation of your user for new VMs as well as the delivery of those files. Thankfully the useradd utility seems to have had this use-case in mind. Here is a script that creates my user, delivers the files, and gives me the administrative privileges (local to the guest VM only). Note that this script only works on SmartOS VMs — more on those later.

#!/bin/ksh #This script copies all of the files and folders to the current user's home #directory. It ignores .audioctl and .mozilla. useradd -s /bin/ksh -b /home -m -k /depot/synthesis/home_files/skel\ -P "Service Management","Software Installation","Primary Administrator" nickziv

The -k switch copies the files in skel to $HOME for that user. Most administrators like to use NFS for this sort of thing, but I think that it is overkill for my needs. In later sections we will see how we can save our installed packages between VMs.

Virtual Machines

Virtual machines are the unit of currency in SmartOS. You can create two kinds of VMs: OS and KVM. In short, OS VMs are faster in every respect to KVM VMs, but are less flexible: they can only run applications that run on Illumos, and their system directories can’t be permanently changed (specifically /etc , /bin , /lib , and /usr ) — this is because they share these directories with the global zone, which means all of your OS VMs get upgraded at no cost when you upgrade your release of SmartOS. This makes them ideal as sandboxes for work you would have previously done in an Illumos machine’s global zone. KVM VMs are slower, and use more RAM and disk, but they make up for it by allowing you to run any OSes (like Linux, Windows, Haiku, FreeBSD, and Plan9) that you may need. This makes them ideal for running legacy applications and applications that aren’t packaged for SmartOS (but are packaged for, say, Debian or FreeBSD).

Both OS and KVM VMs are created from _images}. Images are base templates. The two commands SmartOS provides for managing images and VMs are, respectively, imgadm and vmadm . Generally, you use imgadm to get images, while you use vmadm to turn them into working VMs.

By default images are provided by Joyent, but you can specify other publishers. To see which ones are available for your use just do the following:

[root@78-e7-d1-8c-9b-9e ~]# imgadm avail ... e65e0c4c-dc4b-11e3-b843-93672a0b57d8 cassandra 14.1.0 smartos 2014-05-15T16:13:49Z bb41287c-e02a-11e3-aa6b-db678b82f5fc java 14.1.0 smartos 2014-05-20T14:26:27Z 4b348706-e122-11e3-9cb4-9b2c5062255c nginx 14.1.0 smartos 2014-05-21T19:58:35Z 1daa8e14-e1ba-11e3-bd59-a3971b58fa36 elasticsearch 14.1.0 smartos 2014-05-22T14:05:21Z 3ff5a554-e4ed-11e3-820d-33b5e236480b qemu-kvm-1.1.2 1.2.2 QEMU 2014-05-26T15:49:02Z e9b67338-e4ee-11e3-8668-c35a00c0e8dd qemu-kvm-1.1.2 1.3.2 QEMU 2014-05-26T16:00:56Z 4b46e5e4-f17c-11e3-9da1-17ebb4120b97 standard64 14.1.1 smartos 2014-06-11T15:23:07Z 39a95440-f193-11e3-9ce4-332a0d637d3a standard 14.1.1 smartos 2014-06-11T18:07:16Z 1eed39b6-f7cc-11e3-aacb-7bffef5bf8b6 mysql-cluster 14.1.0 smartos 2014-06-19T16:09:40Z 4db4df60-f880-11e3-a1c2-873cdb86240f nodejs 14.1.1 smartos 2014-06-20T13:39:27Z

The list shows available images, which are uniquely identified by a UUID. You can grab an image by executing the following:

imgadm import <uuid>

You can create a VM based off of an image using vmadm , which takes JSON manifests of the VM you want. Below is one such manifest that I use to create my development VM. You’ll notice that at the end of the file are properties which tell vmadm to create a LOFS mount of the /depot directory (which, keep in mind, is the root of ALL of my other personal ZFS data sets). Using this config, you have to manage these datasets using the zfs command from the global zone — trust me, this is the simplest way.

If you’re like me and like having lots of VMs, you’ll want to have a place where you can store all of your manifests. I created a ZFS dataset for this purpose under zones/depot/manifests . I’m not sure why I didn’t put this under the synthesis dataset, but whatever, this is what I ended up with.

Also, if you use vim, you’ll want to save the manifests using a .js extension instead of a .json extension, because vim only uses highlighting for the former.

{ "brand": "joyent", "image_uuid": "dc0688b2-c677-11e3-90ac-13373101c543", "hostname": "dev", "alias": "dev", "max_physical_memory": 8192, "quota": 5, "resolvers": [ "8.8.8.8", "8.8.4.4" ], "nics": [ { "nic_tag": "stub0", "ip": "10.0.2.0", "netmask": "255.0.0.0" }, { "nic_tag": "admin", "ip": "172.16.60.10", "netmask": "255.240.0.0", "gateway": "172.16.0.1", "primary": "1" } ], "filesystems": [ { "options": "rw", "type": "lofs", "source": "/depot", "target": "/depot" } ] }

You’ll notice that you can give your VMs names (aliases). All of the tools in SmartOS identify VMs using UUIDS, which are a pain to type. So, if you choose to use unique aliases for each VM, you can write wrapper-scripts that expand the aliases into UUIDS. I use the following script to login to OS VMs, instead of zlogin.

#!/bin/ksh # This command logs into vm with alias $2 using user $1. # zvmlogin nickziv dev vmuuid=$(vmadm list -H -o uuid alias=$2) zlogin -l $1 $vmuuid It is stored in `/opt/local/bin` in the global zone, and is run like so:

zvmlogin nickziv dev

You’ll probably want to make such scripts for common VM-related things you do in the global zone. The bash shell in the global zone can do tab-completion for those UUIDs if you start typing them, but I find it unsatisfactory — requires listing the VMs and eyeballing the first 4 to 6 letters of the UUID.

Package Management In OS VMs

In an OS VM you install packages through the pkgin utility which is a wrapper over the cross-platform pkgsrc packaging infrastructure from NetBSD. You search of a package using the search sub-command:

] pkgin search cairo ruby200-rcairo-1.12.6nb1 Ruby bindings for cairo ruby200-gnome2-cairo-gobject-2.0.2nb1 Ruby binding of cairo-gobject ruby193-rcairo-1.12.6nb1 Ruby bindings for cairo ruby193-gnome2-cairo-gobject-2.0.2nb1 Ruby binding of cairo-gobject ruby18-rcairo-1.12.6nb1 Ruby bindings for cairo ruby18-gnome2-cairo-gobject-2.0.2nb1 Ruby binding of cairo-gobject py27-cairo-1.10.0nb1 = Python bindings for cairo py26-cairo-1.10.0nb1 Python bindings for cairo p5-cairo-1.103nb1 Perl bindings to the cairo graphics library guile-cairo-1.4.0nb12 Guile wrapper for cairo gst-plugins1-cairo-1.0.10 Open source multimedia framework - cairo plugin gst-plugins0.10-cairo-0.10.31nb6 Open source multimedia framework - cairo plugin goocanvas2-2.0.1nb2 Cairo-based canvas widget for GTK+3.0 goocanvas-1.0.0nb15 Cairo-based canvas widget for GTK+ glitz-0.5.6nb2 OpenGL 2D graphics library and a backend for gl output in cairo cairomm-1.10.0nb9 C++ API for cairo cairo-gobject-1.12.16 = Vector graphics library with cross-device output support cairo-clock-0.3.3nb27 Analog clock drawn with vector-graphics cairo-1.12.16 = Vector graphics library with cross-device output support =: package is installed and up-to-date <: package is installed but newer version is available >: installed package has a greater version than available package ]

You install a package using the install subcommand, which can be abbreviated to in .

] pfexec pkgin -y in p5-cairo calculating dependencies... done nothing to upgrade. 4 packages to be installed: pkg-config-0.28 p5-ExtUtils-PkgConfig-1.13nb2 p5-ExtUtils-Depends-0.304nb2 p5-cairo-1.103nb1 (377K to download, 1096K to install) downloading packages... downloading pkg-config-0.28.tgz: 0% downloading p5-ExtUtils-PkgConfig-1.13nb2.tgz: 0% downloading p5-ExtUtils-Depends-0.304nb2.tgz: 0% downloading p5-cairo-1.103nb1.tgz: 0% installing packages... installing pkg-config-0.28... installing p5-ExtUtils-PkgConfig-1.13nb2... installing p5-ExtUtils-Depends-0.304nb2... installing p5-cairo-1.103nb1... pkg_install warnings: 0, errors: 0 reading local summary... processing local summary... updating database: 100% marking p5-cairo-1.103nb1 as non auto-removable

The -y flag tells the utility to assume “yes” for all questions.

I maintain a KSH script that installs all of the packages that I need for development, in the home_files directory I mentioned above. It looks like this:

#!/bin/ksh # devpkgs.ksh # This script installs the neccessary dev packages. pfexec pkgin -y in gcc47 pfexec pkgin -y in postgresql93 pfexec pkgin -y in gnuplot pfexec pkgin -y in R pfexec pkgin -y in graphviz pfexec pkgin -y in git pfexec pkgin -y in python33 pfexec pkgin -y in py27-cairo pfexec pkgin -y in xpdf ...

Whenever a I create an OS VM for development, usually to replace an old one, I run the create\_nickziv.ksh script to create my user, and then I run ` devpkgs.ksh` to install all packages. Sometimes packages for certain environments are managed outside of pkgsrc in an environment-specific packaging system (like that used by R). You’ll have to maintain separate scripts for these, as I do for R (called install.R ). Which can run like so: pfexec Rscript install.R .

# install.R install.packages("ggplot2") install.packages("tm") install.packages("wordcloud") install.packages("RColorBrewer")

This is a far-cry from a full-blown Chef or SaltStack deployment, but it’s worked for me so far, and I see no need at the moment to go with something more complicated.

End

I hope that my experience is helpful to you. There are other SmartOS topics that I’d like to discuss, but time is limited. Good night, and good luck.