As many of you know, systemd is the new Fedora init system, starting with F14, and it is also on its way to being adopted in a number of other distributions as well (for example, OpenSUSE). For administrators systemd provides a variety of new features and changes and enhances the administrative process substantially. This blog story is the first part of a series of articles I plan to post roughly every week for the next months. In every post I will try to explain one new feature of systemd. Many of these features are small and simple, so these stories should be interesting to a broader audience. However, from time to time we'll dive a little bit deeper into the great new features systemd provides you with.

Verifying Bootup

Traditionally, when booting up a Linux system, you see a lot of little messages passing by on your screen. As we work on speeding up and parallelizing the boot process these messages are becoming visible for a shorter and shorter time only and be less and less readable -- if they are shown at all, given we use graphical boot splash technology like Plymouth these days. Nonetheless the information of the boot screens was and still is very relevant, because it shows you for each service that is being started as part of bootup, wether it managed to start up successfully or failed (with those green or red [ OK ] or [ FAILED ] indicators). To improve the situation for machines that boot up fast and parallelized and to make this information more nicely available during runtime, we added a feature to systemd that tracks and remembers for each service whether it started up successfully, whether it exited with a non-zero exit code, whether it timed out, or whether it terminated abnormally (by segfaulting or similar), both during start-up and runtime. By simply typing systemctl in your shell you can query the state of all services, both systemd native and SysV/LSB services:

[root@lambda] ~# systemctl UNIT LOAD ACTIVE SUB JOB DESCRIPTION dev-hugepages.automount loaded active running Huge Pages File System Automount Point dev-mqueue.automount loaded active running POSIX Message Queue File System Automount Point proc-sys-fs-binfmt_misc.automount loaded active waiting Arbitrary Executable File Formats File System Automount Point sys-kernel-debug.automount loaded active waiting Debug File System Automount Point sys-kernel-security.automount loaded active waiting Security File System Automount Point sys-devices-pc...0000:02:00.0-net-eth0.device loaded active plugged 82573L Gigabit Ethernet Controller [...] sys-devices-virtual-tty-tty9.device loaded active plugged /sys/devices/virtual/tty/tty9 -.mount loaded active mounted / boot.mount loaded active mounted /boot dev-hugepages.mount loaded active mounted Huge Pages File System dev-mqueue.mount loaded active mounted POSIX Message Queue File System home.mount loaded active mounted /home proc-sys-fs-binfmt_misc.mount loaded active mounted Arbitrary Executable File Formats File System abrtd.service loaded active running ABRT Automated Bug Reporting Tool accounts-daemon.service loaded active running Accounts Service acpid.service loaded active running ACPI Event Daemon atd.service loaded active running Execution Queue Daemon auditd.service loaded active running Security Auditing Service avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack bluetooth.service loaded active running Bluetooth Manager console-kit-daemon.service loaded active running Console Manager cpuspeed.service loaded active exited LSB: processor frequency scaling support crond.service loaded active running Command Scheduler cups.service loaded active running CUPS Printing Service dbus.service loaded active running D-Bus System Message Bus getty@tty2.service loaded active running Getty on tty2 getty@tty3.service loaded active running Getty on tty3 getty@tty4.service loaded active running Getty on tty4 getty@tty5.service loaded active running Getty on tty5 getty@tty6.service loaded active running Getty on tty6 haldaemon.service loaded active running Hardware Manager hdapsd@sda.service loaded active running sda shock protection daemon irqbalance.service loaded active running LSB: start and stop irqbalance daemon iscsi.service loaded active exited LSB: Starts and stops login and scanning of iSCSI devices. iscsid.service loaded active exited LSB: Starts and stops login iSCSI daemon. livesys-late.service loaded active exited LSB: Late init script for live image. livesys.service loaded active exited LSB: Init script for live image. lvm2-monitor.service loaded active exited LSB: Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling mdmonitor.service loaded active running LSB: Start and stop the MD software RAID monitor modem-manager.service loaded active running Modem Manager netfs.service loaded active exited LSB: Mount and unmount network filesystems. NetworkManager.service loaded active running Network Manager ntpd.service loaded maintenance maintenance Network Time Service polkitd.service loaded active running Policy Manager prefdm.service loaded active running Display Manager rc-local.service loaded active exited /etc/rc.local Compatibility rpcbind.service loaded active running RPC Portmapper Service rsyslog.service loaded active running System Logging Service rtkit-daemon.service loaded active running RealtimeKit Scheduling Policy Service sendmail.service loaded active running LSB: start and stop sendmail sshd@172.31.0.53:22-172.31.0.4:36368.service loaded active running SSH Per-Connection Server sysinit.service loaded active running System Initialization systemd-logger.service loaded active running systemd Logging Daemon udev-post.service loaded active exited LSB: Moves the generated persistent udev rules to /etc/udev/rules.d udisks.service loaded active running Disk Manager upowerd.service loaded active running Power Manager wpa_supplicant.service loaded active running Wi-Fi Security Service avahi-daemon.socket loaded active listening Avahi mDNS/DNS-SD Stack Activation Socket cups.socket loaded active listening CUPS Printing Service Sockets dbus.socket loaded active running dbus.socket rpcbind.socket loaded active listening RPC Portmapper Socket sshd.socket loaded active listening sshd.socket systemd-initctl.socket loaded active listening systemd /dev/initctl Compatibility Socket systemd-logger.socket loaded active running systemd Logging Socket systemd-shutdownd.socket loaded active listening systemd Delayed Shutdown Socket dev-disk-by\x1...x1db22a\x1d870f1adf2732.swap loaded active active /dev/disk/by-uuid/fd626ef7-34a4-4958-b22a-870f1adf2732 basic.target loaded active active Basic System bluetooth.target loaded active active Bluetooth dbus.target loaded active active D-Bus getty.target loaded active active Login Prompts graphical.target loaded active active Graphical Interface local-fs.target loaded active active Local File Systems multi-user.target loaded active active Multi-User network.target loaded active active Network remote-fs.target loaded active active Remote File Systems sockets.target loaded active active Sockets swap.target loaded active active Swap sysinit.target loaded active active System Initialization LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. JOB = Pending job for the unit. 221 units listed. Pass --all to see inactive units, too. [root@lambda] ~#

(I have shortened the output above a little, and removed a few lines not relevant for this blog post.)

Look at the ACTIVE column, which shows you the high-level state of a service (or in fact of any kind of unit systemd maintains, which can be more than just services, but we'll have a look on this in a later blog posting), whether it is active (i.e. running), inactive (i.e. not running) or in any other state. If you look closely you'll see one item in the list that is marked maintenance and highlighted in red. This informs you about a service that failed to run or otherwise encountered a problem. In this case this is ntpd. Now, let's find out what actually happened to ntpd, with the systemctl status command:

[root@lambda] ~# systemctl status ntpd.service ntpd.service - Network Time Service Loaded: loaded (/etc/systemd/system/ntpd.service) Active: maintenance Main: 953 (code=exited, status=255) CGroup: name=systemd:/systemd-1/ntpd.service [root@lambda] ~#

This shows us that NTP terminated during runtime (when it ran as PID 953), and tells us exactly the error condition: the process exited with an exit status of 255.

In a later systemd version, we plan to hook this up to ABRT, as soon as this enhancement request is fixed. Then, if systemctl status shows you information about a service that crashed it will direct you right-away to the appropriate crash dump in ABRT.

Summary: use systemctl and systemctl status as modern, more complete replacements for the traditional boot-up status messages of SysV services. systemctl status not only captures in more detail the error condition but also shows runtime errors in addition to start-up errors.

That's it for this week, make sure to come back next week, for the next posting about systemd for administrators!