Introduction

This document compares Upstart and systemd with a view to aiding in the transition to the latter.

Support status

First, it is important to note that systemd is only fully supported in Ubuntu 15.04 and later releases. While systemd is available in prior releases through the Ubuntu repositories, there is a deemphasis of support for these releases as noted here. Hence, it is advised to use the default upstart on prior releases.

System Init Daemon

This has changed as part of the Ubuntu 15.04 devel cycle.

Ubuntu 15.04 (using Systemd by default):

Systemd runs with PID 1 as /sbin/init .

Upstart runs with PID 1 as /sbin/upstart .

Prior versions (using Upstart by default):

Upstart runs with PID 1 as /sbin/init .

Systemd runs with PID 1 as /lib/systemd/systemd .

Switching init systems

If you are running Ubuntu vivid (15.04), you can easily switch between upstart and systemd at will since both packages are installed at present. As of March 9 2015, vivid was changed to use systemd by default, before that upstart was the default.

Switch to upstart for a single boot

In grub, select "Advanced options for Ubuntu", where you will find an "Ubuntu, with Linux ... (upstart)" entry. This will boot with init=/sbin/upstart .

If you have upstart-sysv installed and thus boot with upstart by default, there will be an "Ubuntu, with Linux ... (systemd)" entry, which will boot with init=/lib/systemd/systemd .

Permanent switch back to upstart

Install the upstart-sysv package, which will remove ubuntu-standard and systemd-sysv (but should not remove anything else -- if it does, yell!), and run sudo update-initramfs -u . After that, grub's "Advanced options" menu will have a corresponding "Ubuntu, with Linux ... (systemd)" entry where you can do an one-time boot with systemd.

If you want to switch back to systemd, install the systemd-sysv and ubuntu-standard packages.

High-level startup concept

Upstart's model for starting processes (jobs) is "greedy event-based", i. e. all available jobs whose startup events happen are started as early as possible. During boot, upstart synthesizes some initial events like startup or rcS as the "tree root", the early services start on those, and later services start when the former are running. A new job merely needs to install its configuration file into /etc/init/ to become active.

systemd's model for starting processes (units) is "lazy dependency-based", i. e. a unit will only start if and when some other starting unit depends on it. During boot, systemd starts a "root unit" ( default.target , can be overridden in grub), which then transitively expands and starts its dependencies. A new unit needs to add itself as a dependency of a unit of the boot sequence (commonly multi-user.target ) in order to become active.

Job vs. unit keywords

This maps the keywords that can occur in an upstart job to the corresponding ones in a systemd unit. Keywords which don't have a direct equivalent are marked with "-".

Upstart stanza systemd unit file directive systemd unit file section Notes apparmor load AppArmorProfile Available in systemd version 210 and later apparmor switch - author - chdir WorkingDirectory Service chroot RootDirectory console output StandardOutput=tty , StandardError=tty console owner StandardOutput=tty , StandardError=tty No real equivalent? console none StandardOutput=null , StandardError=null description Description Unit env Environment , EnvironmentFile Service exec ExecStart Service expect fork Type=forking Unit expect daemon Type=forking Unit expect stop Type=notify Unit Similar, not equivalent. Requires daemon to link to libsystemd-daemon and call sd_notify(). instance Use " %I " and " %i " in ExecStart , etc to specify an instance See /lib/systemd/system/getty@.service for an example kill signal KillSignal kill timeout TimeoutStopSec limit as LimitAS limit core LimitCORE limit cpu LimitCPU limit data LimitDATA limit fsize LimitFSIZE limit memlock LimitMEMLOCK limit msgqueue LimitMSGQUEUE limit nice LimitNICE limit nofile LimitNOFILE limit nproc LimitNPROC limit rss LimitRSS limit rtprio LimitRTPRIO limit sigpending LimitSIGPENDING limit stack LimitSTACK manual No directive(?) - use systemctl disable foo.service nice Nice Unit normal exit SuccessExitStatus oom score OOMScoreAdjust post-start exec/script ExecStartPost Service post-stop exec/script ExecStopPost Service pre-start exec/script ExecStartPre Service pre-stop - reload signal ExecReload=/bin/kill -SIGFOO $MAINPID Service respawn Restart=on-failure Service respawn limit RestartSec script / end script See shell scripts below setgid Group Service setuid User Service start on Wants , Requires , Before , After Unit stop on Conflicts , BindsTo (but not commonly used) Unit task Type=oneshot Unit umask UMask Unit usage Documentation Unit no direct equivalent version -

Shell scripts

systemd does not provide special support for shell scripts (by design). For short shell commands you can use something like

ExecStart=/bin/sh -ec 'echo hello'

Longer scripts are usually program logic and should not be directly in a conffile and duplicated between upstart and systemd; factor it out in a proper script in e. g. /usr/share/myapp/ and call it from both the upstart job and the systemd unit.

Automatic starting

As described above, services which want to start during boot (i. e. are not activated through sockets, D-BUS, or similar) need to become a dependency of an existing boot target. Those need an [Install] section with a WantedBy= that specifies the unit which that new service wants to become a dependency of (see man systemd.unit ). Very commonly this is multi-user.target , which is roughly equivalent to start on runlevel [2345] in upstart; see man systemd.special for other common targets.

Invalid configurations

upstart Refuses to start a job if it contains invalid syntax.

systemd Ignores invalid directives and starts the service. Note that you must test your configuration carefully since a typo will not be detected! Use systemd-analyze verify <unit> file to get warnings on typos and badly formatted options.



Override Files

upstart Upstart allows part or all of a Job Configuration (" /etc/init/$job.conf ") file to be overridden. To create an override file: Create file " /etc/init/$job.override " containing one or more stanzas which take priority over their counterparts from the original " $job.conf " file.

systemd Systemd allows a similar facility via "drop-ins": a drop-in allows a directive to be modified without changing the original unit file. To create a drop-in: Create a subdirectory below either " /etc/systemd/system/ " or " /lib/systemd/system/ " called " ${unit}.d/ ". Create files called <something>.conf in the " ${unit}.d/ " directory containing the directives that you wish to override.



Behavioral differences (aka Gotchas)

systemd will refuse to run a binary (via ExecStart , ExecStartPre , etc) unless the full path to the binary is specified: BAD (systemd will not search for sleep in $PATH ): ExecStart=sleep 20 GOOD (absolute path specified): ExecStart=/bin/sleep 20



Commands

Note that these are commands for interactive human usage. Package maintainer scripts, ifupdown hooks and similar must always use the init system agnostic abstractions like invoke-rc.d .

Operation Upstart Command Systemd equivalent Notes Start service start $job systemctl start $unit Stop service stop $job systemctl stop $unit Restart service restart $job systemctl restart $unit See status of services initctl list systemctl status Check configuration is valid init-checkconf /tmp/foo.conf systemd-analyze verify <unit_file> Show job environment initctl list-env systemctl show-environment Set job environment variable initctl set-env foo=bar systemctl set-environment foo=bar Remove job environment variable initctl unset-env foo systemctl unset-environment foo View job log cat /var/log/upstart/$job.log sudo journalctl -u $unit tail -f job log tail -f /var/log/upstart/$job.log sudo journalctl -u $unit -f Show relationship between services initctl2dot systemctl list-dependencies --all Shows pstree -style output.

Example Services

Example Upstart Service

/etc/init/foo.conf :

description "Job that runs the foo daemon" # start in normal runlevels when disks are mounted and networking is available start on runlevel [2345] # stop on shutdown/halt, single-user mode and reboot stop on runlevel [016] env statedir=/var/cache/foo # create a directory needed by the daemon pre-start exec mkdir -p "$statedir" exec /usr/bin/foo-daemon --arg1 "hello world" --statedir "$statedir"

Example Systemd service

/lib/systemd/system/foo.service :

[Unit] Description=Job that runs the foo daemon Documentation=man:foo(1) [Service] Type=forking Environment=statedir=/var/cache/foo ExecStartPre=/usr/bin/mkdir -p ${statedir} ExecStart=/usr/bin/foo-daemon --arg1 "hello world" --statedir ${statedir} [Install] WantedBy=multi-user.target

Outstanding Work

If you'd like to help out with the migration, take a look at ...

Outstanding packages to convert: http://people.canonical.com/~jhunt/systemd/packages-to-convert/ Note that the priority are the packages in " main ".

blueprint: https://blueprints.launchpad.net/ubuntu/+spec/core-1411-systemd-migration

... and then come and chat to us on #ubuntu-devel .

Common Idioms

Service that specifies shell meta-characters

upstart Upstart automatically detects if a job requires a shell to expand meta characters. For example, the following job will automatically be run via " /bin/sh -e ": exec mydaemon --date $(date)

systemd systemd does no such detection so the equivalent unit file would need to be: [Service] ExecStart=/bin/sh -ec "exec /usr/bin/mydaemon --date $(date)"



Service that does not fork (runs in foreground)

upstart job description "blah, blah, blah" exec sleep 999

systemd unit [Unit] Description=blah, blah, blah [Service] Type=simple # (NOTE: "Type=simple" is the default) ExecStart=/usr/bin/sleep 999



Do not run service if no daemon configuration file created

upstart job # defines PORT variable env config=/etc/default/foo pre-start script [ -e "$config" ] || { stop; exit 0; } end script exec mydaemon --port="$PORT"

systemd unit [Service] Type=forking EnvironmentFile=/etc/default/foo ExecStart=/usr/bin/mydaemon --port=$PORT



Note: EnvironmentFile=/etc/default/foo is making /etc/default/foo mandatory. If the environment file (and so variables) are optional, you can use: EnvironmentFile=-/etc/default/foo (notice the - in front of the path).

/etc/default files which enable/disable jobs

enable=1|0 type settings in /etc/default files should generally be avoided. The canonical way to enable/disable a service in an init system agnostic way is update-rc.d <service> enable|disable , which will translate to init system specific actions such as adding/removing symlinks (SysV and systemd) or creating/removing job override files (upstart). For systemd in particular, admins also often call systemctl enable|disable <service> directly. Thus these settings are redundant in /etc/default .

There is no clean way to evaluate these in a systemd unit. You can check them in ExecStartPre= , but that would mean that the unit will be in "failed" state if the service gets disabled that way, and so, is not desirable.

For these reasons (confusing/duplication/cannot be modelled in systemd), these settings should be removed. This was done in whoopsie 0.2.42, you can check its diff for a transition which respects the old default setting and removes it on upgrade.

Start a service when a file is created

upstart job start on file FILE=/var/crash/*.crash EVENT=create exec crash-handler-daemon

systemd Two files are required: /lib/systemd/system/foo.service : [Service] Type=forking ExecStart=/usr/bin/crash-handler-daemon /lib/systemd/system/foo.path : [Path] PathExistsGlob=/var/crash/*.crash Unit=foo.service # (NOTE: the default is "Unit=foo.service" if this is called "foo.path")



Start a service when a D-Bus name is acquired

upstart job start on dbus SIGNAL=NameAcquired INTERFACE=... OBJPATH=... SENDER=... DESTINATION=... exec mydaemon

systemd /lib/systemd/system/foo.service : [Unit] Description=Service that acquires a D-Bus name [Service] Type=dbus # (NOTE: systemd will consider this service ready when it claims the 'foo.bar.baz' name on dbus) BusName=foo.bar.baz ExecStart=/usr/bin/mydaemon /lib/systemd/system/bar.service : [Unit] Description=Service that needs the foo service Requires=foo.service After=foo.service [Service] ExecStart=/usr/bin/mydaemon2



Well-known Sequence Points

Upstart provides a set of "well-known" events that jobs can make use of (such as " runlevel " and " started "). These are summarised in the upstart-events(7) manual page.

Systemd has a similar concept in the form of "targets". See the systemd.special(7) manual page for details and bootup(7) for the boot sequence.

For the particular case of upstart's static-network-up : Our ifupdown integrates that into network-online.target (see man systemd.special(7)). Thus, translate start on static-network-up to Requires/After=network-online.target .

Environment Differences

Both Upstart and systemd provide a very limited environment to the services they run. The environment can be modified by specifying additional stanzas (for Upstart) or directives (for systemd).

However, there are some subtle differences between the default service environment provided by both init systems. The table below shows the major differences:

Description Upstart systemd Environment variables set PATH , TERM PATH , all from /etc/default/locale Standard out and Standard error terminal (pty pseudo-terminal) pipes (to journal)

Note that this information is subject to change; to determine a precise difference on your system:

Install the procenv tool: $ sudo apt-get -y install procenv Create an Upstart job to run procenv : $ cat <<EOT | sudo tee /etc/init/procenv.conf description "Display Upstart environment" exec procenv --file=/tmp/procenv-upstart.log EOT Run procenv under upstart: $ sudo start procenv Create a systemd unit file to run procenv : $ cat <<EOT | sudo tee /lib/systemd/system/procenv.service [Unit] Description=Display systemd environment [Service] Type=oneshot ExecStart=/usr/bin/procenv --file=/tmp/procenv-systemd.log EOT Run procenv under systemd: $ sudo systemctl start procenv Compare the environments: $ diff /tmp/procenv-upstart.log /tmp/procenv-systemd.log

Common Problems

How to identify which init system you are currently booting with

At the time of writing, it is possible to boot an Ubuntu system with either Upstart or systemd since both are installed.

To determine which init daemon you are currently booting with, run:

$ ps -p1 | grep systemd && echo systemd || echo upstart

Why don't some commands (like grep) work in /lib/systemd/system ?

If you hit this problem...

$ cd /lib/systemd/system $ grep foo * grep: invalid option -- '.' Usage: grep [OPTION]... PATTERN [FILE]... Try 'grep --help' for more information.

... you need to add " -- " to the grep(1) call:

$ cd /lib/systemd/system $ grep foo -- *

This is required since systemd provides a file called " -.slice ". After the shell has expanded the asterisk (" * "), that file gets passed to the command in question ( grep , cat , etc) and that command will then attempt to interpret " -.slice " as a command-line option rather than a filename.

Alternative work-around:

$ cd /lib/systemd/system $ POSIXLY_CORRECT=1 grep foo *

To view the pesky file:

$ cd /lib/systemd/system $ cat -- -.slice

Note that this issue only occurs when your current working directory is the directory containing " -.slice ". The simplest work-around is to make the grep or cat call from a different directory:

$ grep foo /lib/systemd/system/* $ cat /lib/systemd/system/-.slice

Debugging

Since this implies the service works under Upstart but is problematic under systemd, details of both systems are provided to allow for some comparison.

Boot Time

Common Setup

Remove the following from the kernel command-line via the grub menu:

" quiet "

" splash "

Upstart

Add " --debug to the kernel command-line via the grub menu.

Optionally add " console=ttyS0 to the kernel command-line via the grub menu if you have a serial console.

systemd

Add " systemd.log_level=debug to the kernel command-line via the grub menu.

Optionally add one of the following too: " systemd.log_target=kmsg " " systemd.log_target=console "



Starting a rescue shell

Run: $ sudo systemctl enable debug-shell.service

Reboot.

If the system fails to boot, you can now switch to tty9 (CTRL+ALT+F9) for a getty console login.

From a running system

Upstart

To switch to debug mode for the system init (PID 1):

$ sudo initctl log-priority debug

To switch to debug mode for a session init (init PID != 1):

$ initctl log-priority debug $ tail -f ~/.xsession-errors

Debian Packaging

Packages installing systemd services should build-depend on dh-systemd and either call dh --with systemd (if they use dh ) or call dh_systemd_enable and dh_systemd_start before/after dh_installinit respectively. Files under debian called *.service will be installed analogously to *.upstart files. See dh_systemd_enable(1p) for how to customize the installation.

Further Information