FiFo 0.9.3 has been in the works for a while, and it comes with quite a few new features. With our last release, we started experimenting with FreeBSD support. Since then much work has gone into improving this. We also did something rather exciting with the mystery box! However, more on that in a later post.

The stable release of 0.9.3 will land within a few days with only packaging and documentation tasks left to do. Part of this means that we’ll have packages for all major components that work natively on BSD. There is no more need for a SmartOS box to run the components!

FreeBSD

When we introduced FreeBSD support last version we marked it as an experimental feature. We needed to try out and experiment what works and what does not. Understand the way FreeBSD does things, what tools exist, and how those align with our workflow. Bottomline we were not even sure BSD support was a thing in the future.

We are happy to announce that with 0.9.3 we are now sure BSD support is a thing, and it is here to remain. That said it was good that we experimented in the last release, we did some significant changes to what we have now. When first looking at FreeBSD we went ahead and used existing tooling, namely iocage, to manage jails. It turns out the tooling around jails is not on par with what exists on illumos and especially SmartOS. The goodness of vmadm as a CLI for managing zones is just unparalleled. So we do what every (in)sane person would do!

vmadm

So with 0.9.3, we did what every (in)sane person would do! We implemented a version of vmadm that would work with FreeBSD and jails and keep the same CLI. Our clone works completely stand alone; vmadm is a compiled binary, written in rust which is blazing fast! The design takes up lessons learned from both zoneadm and vmadm in illumos/SmartOS for how things work instead of trying to reinvent the wheel. Moreover, while we love giving the FreeBSD community a tool we learned to love on SmartOS this also makes things a lot easier for us. FiFo now can use the same logic on SmartOS and FreeBSD as the differences are abstracted away inside of vmadm. That said there are a few notable differences.

First of all, vmadm uses datasets the same way it does on SmartOS. However, there is no separate imgadm tool. Instead, we encapsulate the commands under vmadm images. To make this work we also provide a dataset server with base images for FreeBSD that used the same API as SmartOS dataset servers. Second, we needed to work around some limitations in VNET to make jails capable of being fully utilized in multi-tenancy environments.

nested jails

While on illumos a virtual nic can be bound to an IP that can not be changed from inside the zone, VNET does not support this. Preventing tenants from messing with IP settings is crucial from a security standpoint!

To work around that each jail created by vmadm are two jails: a minimal outer jail with nothing but a VNET interface, no IP or anything and an internal one that runs the user code. This outer jail then creates an inner jail with an inherited NIC that gets a fixed IP, combining both the security of a VNET jail as well as the security of a fixed IP interface.

The nested jail layout resembles the way that SmartOS handles KVM machines, running KVM inside a zone. So in addition to working around VNET limitations, this already paves the way for bhyve nested in jails that might come in a future release. We hope to leverage the same two-step with just a different executable started in the outer jail instead of the jail command itself.