Qubes OS is a single-user GNU/Linux-based operating system that has its focus on isolation between different virtual machines on a personal computer. You get different environments to work in which can not affect the others.

Virtual machines could be Linux-based (currently Fedora 23, Debian 8, Whonix) or even Microsoft Windows. «Qubes» is short for «Qubes OS» and is also used as name for the VMs within Qubes OS.

Multiple experts do promote Qubes OS. Most prominent: Edward Snowden.

If you're serious about security, @QubesOS is the best OS available today. It's what I use, and free. Nobody does VM isolation better. https://t.co/FyX5NX47cS — Edward Snowden (@Snowden) September 29, 2016

I watched a 32C3 talk by the awesome Joanna Rutkowska who is part of the inner Qubes OS team.

Further more, the German IT magazin iX features in its current issue and interesting article on nine months of experience with Qubes OS within a team. They were using Qubes for email, web, LaTeX, LibreOffice, password management, chat, printing, system administration, and VPN networks.

So I gave it a try. This long blog article summarizes my journey and the things I currently do miss with Qubes OS.

Hardware Requirements

There is a list of system requirements and a hardware compatibility list. It is also possible to certify hardware for Qubes OS although this is not very common yet.

Because Qubes OS is using virtualization for multiple VMs in parallel, it is probably recommended to start with eight Gigabytes of RAM minimum. My desktop as well as my notebook do provide 16 Gigabytes of RAM. So I am on the safe side here.

For trying out Qubes OS you don't have to overwrite your precious operating system. I am testing Qubes OS on a (fast) 32GB USB 3.0 flash drive on my lenovo x260 notebook without touching the internal SSD once. Your flash drive should not be smaller than 32GB if you want to add/test/update VMs by yourself. Testing the default might also work with 16 Gigabytes I guess. The project teams does state it has to be 32 Gigabytes minimum.

Installing On Flash Drive

Installing Qubes OS on a USB flash drive requires you to have a second one for the installation process:

Download latest Qubes OS I was using Qubes OS 3.2 Create the bootable installation drive I guess 4GB are OK but I had a larger one

Via something like sudo dd if=Qubes-R3-x86_64.iso of=/dev/sdX Boot the installation drive and install Qubes OS on the second flash drive This took me an hour or so. This depends on the effective performance of your USB drives. Installation drive could be re-formatted and used for different purposes again.

Now, the target USB flash drive is a bootable Qubes OS. Plug it into your hardware and boot from it. This does not affect your built-in HDD or SSD.

Qubes OS is finishing the installation by setting up basic VMs. This takes another half an hour or so.

Running @CubesOS on my 64GB USB flash drive really heats up the poor USB drive — Karl Voit (@n0v0id) July 22, 2017

Terms and Definitions

Before we continue, you have to know a few definitions in order to be able to follow me here.

Qubes OS consists of multiple virtual machines (VMs) separated via the hypervisor Xen. Some of them are for the operating system (dom0, sys-net, sys-firewall, sys-whonix) and some are template VMs for the App VMs (debian-8, fedora-23, whonix-gw).

«dom0» is a special VM since it is the core of the system. It is only for displaying on your screen (Window Manager, Desktop Environment), managing keyboard and mouse input, managing the other VMs via «Qubes VM Manager» and system network and got no network connection itself. It must not be used for anything else like web surfing or working with files. I dared to add two applications though via: sudo qubes-dom0-update redhsift-gtk htop

The other VMs are for the user to work in: «App VMs». By default, Qubes OS sets up the App VMs «work», «personal», «vault», and «untrusted» which are derived from «fedora-23» by default. Additionally, there is always a «disposable VM» at hand which is started for a single task and destroyed afterwards.

App VMs are derived from Template VMs. Their system is a «copy» of the Template VM system without requiring the extra space. This means that you can persistent data within the $HOME folder but not within the rest of the system. Therefore, installing an app within an App VM is possible but this app is not around any more after a restart of the App VM is done. If you want to install an application to be used normally, you have to install the application in the corresponding Template VM.

Once again:

Installing an application within a Template VM makes it possible to use the application in the Template VM itself (not recommended) and all of its derived App VMs after they have been restarted. Data within $HOME is also available in the derived and re-started App VMs.

Installing an application within an App VM provides you this application for this App VM only as long as you don't shut down the App VM. You don't get this application in other App VMs.

Data in $HOME of an App VM is persisted only for this App VM. You can remove files that were created within the Template VM but this also affects the current App VM only.

All VMs are separated. This is the main purpose for the whole concept. App VMs like «work» might be compromised during daily business according to the security of the used operating system of the Template VM. This can not affect anything in other VMs such as «vault» or even the most precious one: dom0.

Default Software

The thing I don't like: dom0/fedora is RPM-based. I switched from RPM-based Linux distribution to DEB-based a long time ago. Therefore, I am a bit of a stranger in the VMs which are derived from the fedora-23 template. Lucky me, there is the possibility to use Debian templates for the App VMs I am using mostly. After setting up Qubes OS, you hardly need to deep dive in any fedora VM.

The default desktop is Xfce which is nice to me. It is my preferred desktop environment for the recent years. You can use KDE as well but I don't want to go there again I guess.

Copy and Paste

Copy and paste between VMs is complicated but secure. In addition to the copy command of the current application, you have to press Ctrl-Shift-c to copy the content of the clipboard to the Qubes OS clipboard. Then you switch to the target VM, press Ctrl-Shift-v to move the Qubes OS clipboard content to the current VM and the you are able to paste as usual, typically Ctrl-v . This prevents any other VM from spying on the shared clipboard.

Unfortunately, the default Terminal application is the GNOME Terminal (v3.22.2) which has already mapped Ctrl-Shift-v . Therefore, if you want to paste into a Terminal, you have to start at least a second application in the target VM. Before pasting, you have to switch to this second application, invoke the non-mapped Ctrl-Shift-v and then switch back to Terminal and paste with Ctrl-Shift-v . This a really silly usability bug.

System Crash and Fresh Start

While the system was under some load (see upgrading Debian below), the X.org server crashed. Unfortunately, I was not able to recover from that: could not switch to a text console, nor did the system requests for Linux work: Alt Print s =u= b to safely write to disk and reboot.

While checking out @CubesOS, its Xorg crashed. Had to hard turn off power. Now it won't boot any more. Not very convincing experience ☹ — Karl Voit (@n0v0id) July 22, 2017

The only thing left was to turn off the power. This resulted in a non-bootable Qubes OS. I had to start from scratch which was a bummer since I invested half a day of work for coming this far.

In my first installation (before the crash), I had no Indication plugin in my Xfce panel. Therefore, I could not select or switch a WiFi network. The Ethernet LAN did work out of the box. Also, the scrolling did not work with two fingers on my touch panel. Both issues did vanish with my second installation. This is good but still alarming.

By the way: if you want to enable scrolling with your Thinkpad trackpoint, you can follow the instructions here.

Updating dom0

The Qubes VM Manager shows an indicator to every VM which got pending updates. It also offers an easy to use GUI upgrade feature. This is quite handy, especially for the dom0 since it does not have direct network connection. Qubes OS uses a trick to update dom0 from secure sources anyway.

USB Drives

USB media such as flash drives are not added to the system when being inserted. This is a security feature since flash drives are a popular source of troubles. Some contain malware, some are not passive flash drives but small computers built to hack your system.

Qubes OS provides a method to access flash drives within VMs. When I tried it, I failed to make it work: «Huston, we do have a problem»-dialog without any content. Using the command line method, I got a libvirtError 'virDomainAttachDevice() failed' .

Suspend and Resume

Suspend to RAM did work out of the box. It does not seem to be feasible with a plugged-in USB flash drive though. Hibernate does not work: the screen goes black, the system works but never goes to any sleep mode.

Minor Things

I faced some display refresh issues in Firefox browser when the system was under load.

There is basically no full screen. You can't watch movies in full screen, Firefox just maximizes its window. Some things can be changed.

The start menu shows sub-menus per VM. So if you want to start Firefox, you have to click in the application menu button in the upper left corner, choose the desired VM such as «work», and then select the shortcut to Firefox. You can add as many apps to this VM menu as you want. By default, there is «Files», «Firefox», and «Terminal» for ordinary App VMs. If a VM is not running yet, it gets started automatically with the first app you start for it. Each VM gets its color and all windows gets window borders with this color. This way, you can differ a Firefox windows from «work» from a Firefox window from «personal».

I miss the possibility to quickly start apps via «Application Finder» when everything is one system.

«Files» is a file manager I did not know. The about-window states «Files 3.18.5». I guess this is GNOME Files which I never used. I prefer the text console or Thunar from Xfce.

Sound does work: YouTube in Firefox works out of the box. Even the Thinkpad keys for sound control and screen brightness are working as expected.

Updating Template VMs is nothing special: you start up the Terminal of the Template VM (not a derived App VM), use the OS-specific update mechanism, shut down the Template VM and re-start all its derived App VMs.

Since Debian in my preferred operating system, I used Debian for my App VMs. Qubes OS 3.2 comes with Debian 8. Since may, Debian 9 is the most current stable release. Upgrading the Template VM from Debian 8 to Debian 9 is well documented. Be prepared to a couple hours of run-time again.

Get used to it that the root user is not that priviledged as it used to be. The root passwords are disabled by default. Anybody within a VM is able to do sudo commands. Privilege is enforced by the VM borders and not via user permissions.

No Shared Filesystem

Separation is the key. Therefore, I do miss my single file hierarchy to maintain and synchronize with my other hosts. The user of Qubes OS has to maintain one separate $HOME data hierarchy per VM.

I already mentioned that copying files (and folders) is possible within VMs. But this does not scale.

I installed Syncthing on my «Debian 9» Template VM and activated it in my «work» App VM. It gets my Org-mode files (and more) synced between my hosts. This works as expected.

However, Qubes OS does not provide a solution for my use-case out of the box. I planned to use a «work-online» VM and a «work-offline» VM. Both from the same Template VM, both with the same file system hierarchy which is in-sync with my other hosts.

Using two separate App VMs with the same set of user data in $HOME (or below), I could decide to use my Emacs with /org/work.org in my «work-online» environment and open a derived /org/work.pdf export file with a PDF viewer in my «work-offline» environment. Applications such as PDF viewers, video players, password safes, local python scripts, and so forth do not need Internet access in my use-cases. Therefore, this makes much sense to me.

This can only be implemented with some kind of file server/client architecture within the network of VMs. I have to set up NFS/SMB or similar. The files gets synced within an App VM with network access and all connected VMs are able to read/write data to the share. What server would you choose for this Qubes/Linux-only use-case? Samba seems to be too much Windows-related (is it?), NFS seems to be too old or «naïve» when I take a look at /etc/exports «authentication» (without doing the cumbersome Kerberos overhead), sshfs is easy to set-up but has severe performance drawbacks. I guess I need to provide the authentication via iptables firewall. And then there is still the drawback on having poor file system performance when the whole network/NFS stack gets added as well.

I guess this could be improved by the Qubes OS using an easy to use default setup or at least a technological recommendation for this use-case.

Usability Improvement: VM Separation via Virtual Desktops

Qubes OS provides a seamless GUI. This means within the same (virtual) desktop, there is one window from App VM «work» and a different window from App VM «vault» at the same time. You can differ them with their mandatory window decoration color.

So far, so good.

But how about following different and optional(!) approach: You set up a number of virtual desktop per VM. Let's say four of them. With each running VM, you get this number of additional virtual desktops. They are stacked on-top so that with three running VMs, the first row of four virtual desktops are from the first VM. Each row of virtual desktop belong to a VM.

With this kind of setup, you are able to simplify the start menu. Instead of having the same menu structure with all VMs on all virtual desktops, you can hide most of them and display only the ones related to the current VM.

Yes, this might be interpreted as some kind of setback compared to the nicely done seamless GUI. But for aunt Martha and me, it might be the preferred way of having things arranged in virtual desktops without the possibility to have windows of separate VMs on one desktop as a deliberate restriction.

The simplification of the application menu is also a good improvements in my opinion.

Whatever.

Restrict Network to Personal LAN Only

I still have to figure out how to configure VMs so that they don't get access to the Internet but to my local LAN only.

Of course, this has to be done so that being in a different LAN is detected so that I don't get access to the foreign, potentially dangerous LAN at all even if they have similar IP ranges. Think about Internet Cafés or student dormitories: you really don't want to expose your computer to those LAN.

So: when my computer is in my personal LAN, I am able to define App VMs that are basically offline from the Internet but get access to my other devices within my personal LAN.

Do you have an idea how to accomplish this? Drop me a comment below!

System Overview

Qubes VM Manager shows a neat overview of the VMs of the system. However, you don't see any network traffic statistics. In dom0, top does only show dom0 and not the other VMs. So you never get a better overview on, e.g., RAM usage of the whole computer. Maybe there is a workaround but I did not found any so far.

I do miss this overview I usually get with a few Xfce panel plugins: network bandwidth history graph, CPU history graph per core, disk usage.

This does not seem to be possible in Qubes OS.

Summary

All in all, the Qubes OS team did an awesome job on integrating all this things so far. The security of the App VMs is not better than the security of the corresponding Template operating systems. However, if a App VM gets an issue, it does not affect the others. If you plan to do weird things, you can use a disposable VM where all changes gets discarded afterwards. It is very easy for anybody to create App VMs without network for example. And this is something I would like to use ever since I stumbled over Firejail which provides app-specific sandboxing.

In general, the documentation of Qubes OS is awesome. I learned a lot and I had to. Qubes OS is nothing to set-up by aunt Martha. You have to have deeper technical understanding to set-up the system. Afterwards, anyone is able to use it with a short introduction to the basic guidelines.

Of course, I found some usability issues and some bugs here or there. But overall, Qubes OS is a valid option for a security purist or a privacy-aware person.

When Qubes OS meets my personal requirements, it complicates things though. For example the file server/client architecture adds complexity you don't have to maintain within a personal computer.

Accessing USB devices, network printers and so forth is cumbersome as well.

You have to set your priorities. ;-)

Will I install it on my notebook or desktop and use it on a daily basis? My notebook would be cool since it is in potential harmful environments such as WiFi networks I don't control. On the other side, I do need USB flash drive access with good usability and need to connect to projectors.

My home server/computer runs 24/7 and could profit from Qubes OS since I got many different things running on this machine. I could separate those domains. Working in offline VMs with applications that don't need network is also a very nice thing to have. The USB flash drive thing is also a thing here from time to time. Restricting to LAN access only would be fine. System crash resulting in an encrypted system that does not boot any more is a no go. Providing services for devices in my LAN is a must. Syncthing discovery and direct sync connection between my notebook and my server within my LAN probably won't work without manual modifications.

Well, I am not convinced yet. Probably I stick to Debian 9 or I do find the urge to come around the issues and find a Qubes OS setup which serves me well.

Where is the «edit» functionality of @Twitter? I mistyped @QubesOS as CubesOS in several tweets over the last days :-( — Karl Voit (@n0v0id) July 23, 2017

Here is an article that covers Qubes OS 3.0 (from 2015).