Troubleshooters.Com®, Linux Library

and Init System Choices Present:

The Manjaro Experiments

Copyright © 2014 by Steve Litt

See the Troubleshooters.Com Bookstore.

CONTENTS:

December 17, 2014

As I begin this document, it's December 17, 2014. Here's my current situation:

My Manjaro 8.11 64bit OpenRC edition experimental computer boots with a combination runit/OpenRC init system. It boots Grub to command prompt in 10 seconds, and shuts down LXDE to poweroff in 5 seconds. This is a circa 2008 commodity box, with an AMD Athlon II X2 250 Processor. According to cpubenchmark.net, this processor has about one third the computing power of a current Intel I5. This computer has 4GB of RAM, and an ordinary 3.5" hard drive. Equipped with its runit->OpenRC init and its dbus-less LXDE, it's a reasonable desktop computer with sound, video, Internet.

[root@openrc ~]# ps ax | grep -v grep | grep dbus [root@openrc ~]#

PID 1 is runit. Runit has three "stages", each represented by a shellscript. Stage 1 does one-time bootup tasks. Stage 3 does shutdown tasks. Stage 2 manages processes, and is a heck of a lot like daemontools. My stage 1 script, /etc/runit/1, follows:

#!/bin/sh # system one time tasks PATH=/command:/sbin:/bin:/usr/sbin:/usr/bin rm /tmp/sentinels/* #/usr/bin/openrc sysinit #/usr/bin/openrc boot #/usr/bin/openrc default ### START devfs AND sysfs, WRITE SENTINEL FILES /usr/bin/openrc minimum rm /tmp/sentinels/* touch /tmp/sentinels/devfs.sent touch /tmp/sentinels/sysfs.sent #runsv /service/udevd ### SET UP NETWORK WITHOUT systemd, NetworkManager, dbus or wicd ### /root/upnet.sh /etc/init.d/rcS /etc/init.d/rmnologin touch /etc/runit/stopit chmod 0 /etc/runit/stopit

The preceding is pretty simple, all things considered. The stuff about /tmp/sentinels is my attempt to record which tasks have already been done, because, as far as I know, runit has nothing like OpenRC's "provides" keyword, so I actually need to leave something that can be tested.

The three commented openrc calls (sysinit, boot and default), would perform almost all initialization using OpenRC, if they were uncommented. In OpenRC, sysinit, boot and default are called "runlevels", which is a different definition than used with the more familiar sysvinit init system, where you tend to run one specific runlevel, instead of consecutively running three.

The point is this: I wanted to transfer as much responsibility as possible from OpenRC to runit, so I made a new OpenRC runlevel, called minimum, that starts only these three things:

devfs sysfs udevd

With just those three underpinnings accomplished, I can do almost everything else with runit.

It would have been easy enough to run NetworkManager in order to set up networking, but NetworkManager depends on dbus, and I'd prefer to minimize my reliance on dbus. As a matter of fact, dbus doesn't run on my current Manjaro box, although that might change later. So, to start up networking, after instantiation of devfs, sysfs, and udevd, the runit stage 1 script runs a shellscript called /root/upnet.sh, which looks like this:

#!/bin/sh hostname -F /etc/hostname ifconfig lo up ifconfig eth1 add inet 192.168.100.88 broadcast 192.168.100.255 netmask 255.255.255.0 ip link set dev eth1 up route add default gw 192.168.100.96

NOTE: The preceding script uses the ifconfig command, which is now deprecated in favor of the ip command. I've made a better Network Start Script that uses only ip commands.

A couple notes about the preceding. First, the fact that this shellscript is in /root is just an experimental convenience. It can be anywhere. Second, this script hardcodes all the information. It would be easy enough to use dhclient, to connect the wired ethernet port via DHCP. Third, I've already created a script to connect to a specific Wifi ssid via wpa_supplicant, using DHCP, no NetworkManager or wicd (both of which now depend on dbus) required. For use on the road, it wouldn't be particularly hard for me to create a small front end to wpa_supplicant and iwlist to pick from a list of available Wifi networks and log into one. No NetworkManager or wicd required, and no consolekit (or whatever they're calling it these days), because I can manage that via groups and sudo.

This Document's Purpose

The purpose of this document is so others can follow in my footsteps, but take minutes or hours to accomplish what it took me days or weeks to do with research and experimentation. Therefore, this document's content must be explicit and unambiguous. If you run into any ambiguities in this doc, or if following its suggestions doesn't work for you, please email me so I can make it clearer. I have a dog in this fight: I want to make sure that anyone wanting to choose their own init system can do so, preferably on any distro they choose.

This doc will be an ongoing project, as I learn more and more about init systems via my Manjaro experimental computer.

Haters, Whingers, Demotivators and Trolls

If you say "I don't want systemd on my computer", the insults fly quick and fast. I don't just mean on the mailing list of a distro that's gone systemd, such as Debian. I mean on LUG lists and IRC channels, and even on lists and channels serving those wanting a systemd alternative. "Hater." "Whinger." "Demotivator." "Troll." "Offtopic." "Take it to the offtopic list." "Spammer." These characterizations are unavoidable to the person who chooses to kick systemd off their computer.

So let me state my position: I'm the one who will choose the init system for my computer!

Hey, I live in a free country. I use free software. I'm free to modify my software to fit my needs. Tell me one more time just so I understand: Why does wanting choice, including the choice to kick systemd off one's computer, make one a hater or a whinger?

You know, the Debian init system wars were never about systemd. They were about choice. When systemd started subsuming more and more parts of the operating system, our choices were foreclosed, or at least the choices that were easily done with the package manager. The fight was about that foreclosure.

So the next time you hear people characterizing alternative-init fans as "haters" and "whingers", the question you might want to ask the epithet hurler is this: "What do you have against choice?"

And understand that as you read this document, it's about choice.

Init System Primer

The Init System is the program run directly by the kernel. It's always PID (Process IDentifier) number 1. After Grub or LILO or whatever hands control to the kernel, the kernel initializes itself, runs any necessary kernel processes (the processes the ps command prints in square brackets), the kernel runs exactly one program, and then just hangs around waiting for requests. That one program is the Init System. The Init System first runs a bunch of setup (as in runit stage 1), then runs and manages a bunch of processes (as in runit stage 2), all the while looking out for interrupts like Ctrl+Alt+Del, or programmatic shutdown/reboot commands like halt, reboot, /sbin/shutdown -h now, init 0, init 6, openrc shutdown, runit-init 0, runit-init 6, and the like. When any of those interrupts or commands are heard, runit stage 3 is run. Runit stage 3 shuts down or reboots the machine in an orderly fashion.

Runit stage 2 also manages processes that it started. This means that it handles logging of those processes, restarts them if they crash, and gives you tools to start, stop, and restart individual processes, as well as temporarily disabling processes, even through reboots, without disrupting your configuration.

And, perhaps best of all, runit manages all these processes without need for the user or admin to worry about PIDs. If runit ran it, runit knows how to find it, without a burdening the user or admin with the need to track PIDs.

The init program filename can be anything. Traditionally, it's been called /sbin/init, but it can be called anything. Simply append the following to your Grub boot line:

init=/path/to/program/myinit_program

Or you can back up your existing /sbin/init, and then copy /path/to/program/myinit_program to /sbin/init.

A few pieces of information: The systemd init program is tucked several directories deep, so if you're switching over to a different init system, once everything's debugged with that system, you're probably better off changing the init program location permanently within Grub or LILO. Also, it might seem a good idea to have /sbin/init be a symlink to the real init program. That can fail. Specifically, it fails with runit, where a symlink brings up a jailed busybox emergency environment.

In summary, when the kernel is done initializing itself, it runs exactly one program, the init program. The init program sets up various necessities, and then runs and manages all necessary processes. If hit with specific interrupts or commands, it cleanly shuts down or reboots the computer.

Make a Damage Control Plan

Step 2 of the Universal Troubleshooting Process is Make a damage control plan. The purpose is to be able to back out and/or minimize any damage you cause during your troubleshooting (or experimentation). The Damage Control Plan must in place before work commences. In the case of init system experimentation, here is the Damage Control Plan I use:

Work on an experimental machine

Have a known good System Rescue CD at the ready

Keep track of the steps you've done

Work on an experimental machine

If you never fall off your skateboard, you're not really going for it. In exactly the same way, if you never make your box unbootable during init system experimentation, you're not really going for it. And of course, whenever you bork your bootability, the restoration presents the possibility of inconvenient unscheduled backup at the best, and data loss at the worst.

If you do your experimentation on an experimental machine, the worst that can happen to you is you reinstall the OS and redo the successful steps you've taken. It's worry free.

Doing init experimentation on a virtual machine is just as good, always assuming:

You don't accidentally perform a step on the host OS or the wrong VM. Your experimentation doesn't include VM hosting.

Have a known good System Rescue CD at the ready

With a known good, tested, bootable CD or DVD with the System Rescue CD system on it, you can bust back into almost any borked OS, as long as the filesystems are still good. So, when you boot to an init that won't even bring up a terminal, no problem: Boot System Rescue CD, mount the partition(s), back out your last change, and reboot to the hard drive. No muss, no fuss, no bother.

Obviously, this is time consuming, involving two boots, so what you might want to do is have a known good init at, let's say, /sbin/init.knowngood, so you can simply tack init=/sbin/init.knowngood on the end of your grub2 boot command. But if all else fails, if you have a CD formatted to System Rescue CD, you'll probably be able to bust back in and fix the problem without needing to reinstall.

Keep track of the steps you've done

You can't back out what you've forgotten, so be sure to record each step you took. And obviously, don't record it on the experimental computer, which could become unbootable. Pencil and paper, or another computer, or a networked drive, are best. And save often. Besides the benefit of being able to back out your last change or two, recording your steps with their results helps you document the correct process later, and helps you quickly reproduce it on another computer.

Make It Easy On Yourself

Experimentation is no place to load yourself down with every difficulty known to man. During early experimentation, you would no more use a multi-partition setup than you would learn to drive with a stick shift. Can it be done? Yes, in both cases. Will it add more than the sum of the challenges in terms of time to mastery? Yes, much more.

Init system experimentation is not the time to gratuitously use non-defaults. Where possible, use the defaults, and if possible use only a single partition, for /, and have your whole OS and /home on that partition. When you need to mount the OS partition from System Rescue CD, you'll be very glad you need to mount only one partition.

Make it easy on yourself. When experimenting, delay the exotic.

Lessons Learned As Of 12/17/2014

Here are some of the lessons I've learned about init systems as of 12/17/2014:

There are more than three init systems.

All init systems have grossly inadequate documentation.

Linux users' ignorance of init systems is astonishing.

I'm amazingly ignorant of init systems.

Much init system complexity is fluff and edge case CYA.

Manjaro's Pretty Cool

Debian's Seen Better Days

There are more than three init systems.

Most of those involved in the Debian CTTE Deliberations implied that the only init system choices were sysvinit, Upstart and systemd. Nothing could be farther from the truth. Here are the init systems I know about:

runit OpenRC sysvinit Upstart systemd S6 perp nosh Epoch Init System

If anyone tries to argue with you based on there being three possible init systems, smile and nod and try not to laugh.

All init systems have grossly inadequate documentation.

Systemd has been (rightly) attacked as having inadequate documentation. But so does upstart. And sysvinit, if you need to do anything the slightest bit creative. S6's docs were such that I installed it, but didn't know what to do next, and had to abandon it. I couldn't even get nosh compiled, and I think that's mostly because of bad docs. Runit's documentation is so bad that I had to learn it by experimentation, even though I know its ancestor daemontools inside and out. OpenRC has some good docs, mostly courtesy of Gentoo and Arch, but not really good enough to understand it.

Beneath all of that, very few have explained the high level architecture of an init system. I think Runit comes the closest to a good high level explanation, at least if you combine it with djb's and my docs on daemontools. But it's not good enough to develop a critical mass of Linux user understanding.

This universal lack of documentation produces several problems:

Very few technical users know how to replace/install an init system.

Most technical users view init systems as black boxes.

BS'ers can talk all sorts of trash about init systems without challenge.

Lesser known but excellent init systems wither on the vine.

If you know, really know, the nuts and bolts of any init system, please document it. And document it for someone not already familiar with it. Don't say "this function takes four arguments", but instead show the exact syntax and give examples. Too many great init systems have languished because nobody had the time to learn their details by experimentation.

Linux users' ignorance of init systems is astonishing.

I can't really say more than the title of this subsection. Now that I have a sliver of init system knowledge, I appreciate the black box most see it for. And of course there are those who proclaim knowledge they don't have.

I'm ignorant of init systems.

I'm no exception to init system ignorance. I've never coded an init system. I have little knowledge of how the init system handles interrupts and other inputs (from FIFOs and sockets, for instance). My sole claim to fame is that I've booted from sysvinit->OpenRC, runit->OpenRC, and that I've reduced OpenRC to running just three OpenRC scripts (devfs, sysfs and udevd). I'm not an authority, I'm just a guy five feet ahead on the trail, wielding a machete to cut through the brush, so that those who follow can do less machete work.

Much init system complexity is fluff and edge case CYA.

In case you didn't catch it when I said it earlier in this document, my experimental computer, which is a reasonable desktop computer running LXDE, starts these programs or processes:

Via /etc/runit/1 devfs sysfs udev My home grown, less than ten line network initialization shellscript

Via /etc/runit/2 (daemontools style) Virtual terminals 4, 5 and 6 sshd

(daemontools style)

With just the preceding, I've made a fairly good desktop computer. Is it "ready for prime time?" No. But it wouldn't take much to get it there:

An fsck daemon so it will check disks every so often. Somewhat tricky.

Start my local Dovecot server as a daemon: Trivial.

Start an http server. Trivial.

If I want, start xdm as a daemon: Trival, but I like booting to CLI and typing startx .

. Ntpd time support: Probably fairly easy runit stage 2 task.

Dbus support. I don't want it, but it's easy to run as a daemon early in stage 2.

Dmesg support. This occurs in /etc/runit/1 , and runit isn't as good at process dependencies as OpenRC or systemd, so this might be a little difficult. But not a big problem.

, and runit isn't as good at process dependencies as OpenRC or systemd, so this might be a little difficult. But not a big problem. Loopback support: Assuming you make the script Linux specific, instead of making one script fit both Linux and BSD, this is a fairly easy runit stage 1 task, and once again, the biggest problem will be calculating process dependencies.

Get cron running. I actually did this just now, in about five minutes, using runit. It appears to work, but the crond man page says that if you use -n to run crond in the foreground, which you need to do with all daemontools-like process managers such as runit, then /etc/pam.d/crond must not enable the pam_loginuid.so module. But that wouldn't be hard, and anyway, a few months ago I made my own cron service with Python, so if worst comes to worst...

A lot of things that were originally intended to be started by OpenRC, but are not started as part of my OpenRC setup, appear to "just work" without running them. Urandom, the clock, the /proc directory, and sound are just a few. Will I later find problems with these things? Maybe, but if so, most of these things are very easy to either run, or make substitutes for.

Which brings up this question: If I got this system working using only 10% of the normal processes, why all these processes? My theory is that all these processes are so that every conceivable use case will work, without programming, scripting, or anything outside the packaging system. Of course, this makes the packaging system so complex that, personally, I'd rather be writing scripts.

The purpose of a lot of this complexity is to make a user interface where all configuration is done with successions of mouse clicks rather than config files or shellscripts. Where the user never needs to become root, because something like consolekit keeps a granular list of what the user can configure. Where separate programs magically react to each other. For the right person, these are all nice benefits. But what's the cost? My experimentation tells me the cost is a tenfold increase in the number of daemons the init system needs to spawn, and therefore a much more complex and fragile software environment.

Is it worth it? Sure, for some. They use Gnome or Unity or Windows or OS-X to drag and drop and click to their heart's content, never dealing with a command prompt or editing a config file. They consider any other way of working unprofessional. These users need lots of daemons, each with a complex init script or unit file to take into account every use case, because the user can't or won't configure or write a simple shellscript.

But for others, starting with a simple system and personally addressing their own individual needs with a shellscript here, a config file there, and maybe a few Python programs, is much easier. And much less brain-taxing. This isn't everybody's cup of tea, but for me, it means a simple system designed specifically for me. And I know a lot of other people who think the same way.

I'd compare the "daemon for every conceivable use case" with prescription medicine. One or two pills a day, fine. Five, well, maybe. But some people take ten pills a day, and the pills interact with each other in strange and undiagnosable ways, and the next thing you know, those pills have brought the patient a whole new set of problems.

Manjaro's Pretty Cool

This lesson learned has its own section

Debian's Seen Better Days

This lesson learned has its own section

Manjaro Experiments, Pre-20141217

In a way, these experiments have gone on for months: Ever since I heard about systemd's architecture and decided that architecture wasn't something I wanted on my computer. My experiments took two paths:

Experiment with various systemd-free distros. Experiment with replacing systemd myself.

After trying Funtoo, (I'd tried Gentoo years before and didn't like it), Porteous, PCLinuxOS, OpenBSD, FreeBSD, PC-BSD, Arch, and a couple others, I determined that 2015 would be slim pickings for the guy who wanted fast, easy installation and a good package manager, and no systemd. That being said, these experiments proved to me that I could base my business on PC-BSD. This was a relief.

But the systemd wars on the Debian-User mailing list, combined with what I considered the breathtaking stupidity of the arguments I'd heard there, and the April Fools Joke architecture of systemd itself, got me curious enough to find and install alternate init systems, even after I'd decided on PC-BSD. After all, systemd proponents had stated quite firmly that no modern computer could be without the assistance of systemd. That got me curious, and more important, it got me ornary.

I began trying to replace init systems.

One of my first attempts, and this was probably a couple months ago, was to use the code near the bottom of this blog. I kinda-sorta got it to run bash, but that was about it. I lacked the knowledge to go further.

I also tried the yocto init shellscript, but failed to get it to put me into a command prompt. I compiled nosh and S6, but had insufficient documentation to get them to assist in booting the computer. At least without a crazy amount of trial and error experimentation.

The turning point came when a guy on the Freenode #debianfork IRC channel told me that systemd on the Manjaro distro could easily be replaced with OpenRC, and recommended I tune in to the #manjaro-openrc channel. I followed some instructions to replace systemd with OpenRC, with patchy success. Then somebody told me there was now a Manjaro 8.11 OpenRC ISO, so I downloaded and installed it.

NOTE: Over time, the link for the latest Manjaro OpenRC ISO will change. You can always find links to the latest ISO at https://forum.manjaro.org/index.php?board=50.0, in the "Sticky Topics" section.

Having a working OpenRC init system was experimental heaven. I could now make one change and see its results, instead of having to guess exact syntax and procedures for ambiguous documentation. I became conversant with the rc-update and rc-status commands. I installed a service that started daemontools, and started my Virtual Terminals from daemontools after commenting them out of /etc/inittab. It worked.

One of my first moves with Manjaro's sysvinit->OpenRC setup was to replace sysvinit with a shellscript:

#!/bin/bash if test "$1" = "0"; then /usr/bin/openrc shutdown exit 0 elif test "$1" = "6"; then /usr/bin/openrc reboot exit 0 fi echo TOP OF INIT.SH env | grep -v COLOR echo echo $@ env | grep -v COLOR > /etc/initargs.log echo >> /etc/initargs.log echo $@ >> /etc/initargs.log sleep 5 # System initialization, mount local filesystems, etc. /usr/bin/openrc sysinit # Further system initialization, brings up the boot runlevel. /usr/bin/openrc boot /usr/bin/openrc default echo IF YOU READ THIS, YOU'VE FALLEN THRU TO THE BOTTOM.

The preceding init script, which obviously contains lots of diagnostic prints and logs, kinda-sorta worked, booted up fine, but had problems on shutdown or reboot. Nonetheless, I'd proven the concept of using a non-sysvinit PID1 with OpenRC, although obviously OpenRC was doing all the work.

But with OpenRC, my init process still seemed like a black box. My ultimate goal was to init the box with a daemontools-like init system. At first I had OpenRC hand over control to daemontools, but that caused a lot of problems.

I managed to get runit to compile based on the install instructions on the runit website. Then I managed to deploy it using the PID 1 replacement instructions. I'll cover details and troubleshooting of this step later in this document. It wasn't easy for me, but now that I've been through it, I can explain how it can be easy for you, later in this document. Anyway, as per the instructions, the only thing runit did was expose Virtual Terminal 5 when I pressed Ctrl+Alt+5.

Actually, it wasn't that easy, because I had a wrong run command in my runit stage 2 getty-5 directory, but I finally troubleshot that.

My next step was to add the following to /etc/runit/1:

/usr/bin/openrc sysinit /usr/bin/openrc boot /usr/bin/openrc default

In other words, I was using OpenRC exactly how it was supposed to be used: To do everything except receive the handoff from the kernel. It worked perfectly.

Next, I experimented to see how many OpenRC actions I could remove and still successfully boot, initting from runit with less and less help from OpenRC. It turns out I needed to have OpenRC run and manage udevd, and udevd required sysfs and devfs. So I made a brand new OpenRC run level called "minimum", that contained links for sysfs, devfs and udev, and replaced the three openrc commands in /etc/runit/1 with the following single command:

/usr/bin/openrc minimum

After doing that, the computer booted up to the point where I could use ifconfig and the ip and route commands to set up the network. And of course I added a sshd service to stage 2 of runit. This gave me a desktop computer that, if you didn't delve too deeply into details like the fact that it never prophylactically runs fsck, worked almost identically to my Debian Wheezy daily driver.

Using OpenRC with Manjaro

I imagine that with enough knowledge you can install OpenRC on any Linux distro. I chose Manjaro because it's an excellent desktop distro, very similar to Debian, and it has a ready-made OpenRC edition available at https://forum.manjaro.org/index.php?topic=19215.0. From this point, I'll assume you have OpenRC installed, complete with the init scripts necessary to run your computer.

OpenRC is a cool init system. It has facilities very similar to what I've read systemd has for calculating process dependencies. On Manjaro, all its init scripts are kept in /etc/init.d. These init scripts are quite similar to those for sysvinit, except they use #!/usr/bin/openrc-run instead of #!/bin/sh. OpenRC's runlevels are used differently than those for sysvinit. With OpenRC, each runlevel runs a set of tasks, and in booting up, you'll typically run several runlevels, such as "sysinit", "boot" and "default". When shutting down, you typically run runlevel "reboot" or "shutdown".

These runlevels are implemented similarly to those in sysvinit: Through symlinks. All the init scripts are in /etc/init.d, and the init scripts run by a runlevel, let's call it myrunlevel, would be symlinks in /etc/runlevels/myrunlevel to the files in /etc/init.d. Here's an example for my runlevel called "minimum":

[slitt@openrc ~]$ ls -l /etc/runlevels/minimum total 0 lrwxrwxrwx 1 root root 17 25.11.2014 14:42 devfs -> /etc/init.d/devfs* lrwxrwxrwx 1 root root 17 25.11.2014 14:42 sysfs -> /etc/init.d/sysfs* lrwxrwxrwx 1 root root 16 16.12.2014 21:07 udev -> /etc/init.d/udev* [slitt@openrc ~]$

You can add or remove processes from levels by deleting or making symlinks. Also, the rc-update command is a more user friendly way of dealing with those symlinks, and is portable in that it works no matter where your distro keeps those symlinks. You can make a new runlevel just by making a new, appropriately named directory, below /etc/runlevels, and creating the desired symlinks inside of it.

Technically, OpenRC isn't a complete init system. It can't receive a handoff from the kernel, because it doesn't do certain things that a PID1 must do. So the way you use OpenRC is to use a "real" init system, such as sysvinit or runit, to make all the necessary OpenRC calls, after which OpenRC manages all the processes. I can't tell you offhand whether systemd can act as the PID1 and hand off to OpenRC because systemd tends to do a lot more than just init, and I don't know how separable the parts are.

Some day I'd like to use the code near the bottom of this blog to init and then hand off to OpenRC, perhaps by making /etc/rc look like this:

#!/bin/sh /usr/bin/openrc sysinit /usr/bin/openrc boot /usr/bin/openrc default

My impression is that if you get OpenRC installed, it's both powerful and trivial to administer if somebody has written the OpenRC init scripts for you. Those init scripts tend to be as complex as those for sysvinit.

When I hear people brag about systemd, those boasts usually involve fast boots, clean process dependency handling, and parallel initialization of processes. OpenRC fulfills all three of those criteria. And then OpenRC gets the hell out of the way.

This section contains some very basic facts about daemontools, a process management software written by Daniel J. Bernstein (djb)

Runit is a daemontools-like init system. So are nosh and S6. If you're truly interested in alternate init systems, it behooves you to learn about daemontools.

In my opinion, the best daemontools documentation is contained djb's daemontools documentation and Steve Litt's daemontools documentation. That being said, there are many other excellent sources.

Daemontools daemonizes and controls foreground processes, not background processes. When starting a normally background process from daemontools, find that process's command line argument to put it in the foreground. For instance, /usr/bin/sshd -d or /usr/bin/crond -n. Daemontools and runit have some kludgy workarounds to foregroundize processes that give no choice but to run in the background, but these kludges might not work, might be unstable, or might produce side effects. When you can, run it in the foreground. When you can't, think long and hard whether you need that daemon.

Adding a daemon (service in daemontools-speak) requires creating a directory tree, and then symlinking that directory tree to a special directory (/service by default). Here's an example:

ln -s /etc/sv/sshd /service/sshd

In the preceding, you did all the daemon configuration in tree /etc/sv/sshd, and then, to make it actually function, you symlinked it to /service/sshd

The default symlink target, /service, can easily be changed. This is important to those who don't like miscelleneous directories created straight from the root.

The tree you create to assemble a daemontools-managed daemon is built like this:

/etc/sv/sshd |-- log | |-- run | `-- supervise | |-- control | |-- lock | |-- ok | |-- pid | |-- stat | `-- status |-- run `-- supervise |-- control |-- lock |-- ok |-- pid |-- stat `-- status

The supervise directories in both sshd and sshd/log are created and managed by daemontools, after you symlink sshd into the /service directory. You needn't construct or modify the supervise directories, except in very unusual troubleshooting situations when the system gets into a bad state. Your work is to create the run scripts, which are usually pretty simple.

Undocumented Runit Rules of the Road

First, I highly suggest you obtain runit from what Debian would call "the upstream", in other words, runit author Gerrit Pape's website, rather than from a package. When you use Gerrit Pape's code with the instructions on that page, you're dealing with a known entity, and not something a package manager can screw up (or abandon). If you or your employer or customer really have a problem with /service directory being right off the root, this can be changed. But...

I'd highly suggest you follow Gerrit Pape's instructions exactly, even if they produce directory structures that are against your best practices. Once you have everything working, it will be trivial to adjust all directories to conform to your needs and principles.

The next thing to understand is that runit probably won't run the first time. That's OK, you can troubleshoot it.

Another thing: This section assumes you named the service for running tty5 "getty-5", and the directory in which you assemble all services is called /etc/sv, and your service directory, in which you create symlinks to the directories contained in /etc/sv, is called /service. If you diverge from any of these conventions, you'll need to change your diagnostic tests to reflect those divergences.

To maximize the chance of your computer displaying a reasonable interface for troubleshooting, I suggest you enable virtual terminals 2 and 3 within /etc/inittab. This enablement would look something like this:

c2:2345:respawn:/usr/bin/agetty 38400 tty2 linux c3:2345:respawn:/usr/bin/agetty 38400 tty3 linux

If /etc/inittab enables virtual terminals (tty) 1, 4, 5 and 6, I suggest you comment them out. Commenting out tty1 means the boot display remains visible on Ctrl+Alt+F1. Commenting out tty 4 through 6 gives you room to instantiate those ttys with runit. Be sure to reboot after making those changes to /etc/inittab.

Gerrit Pape's instructions yield a proof of concept, a "Hello World" if you will, that does nothing but instantiate tty5. No network, no other niceties, just tty5. Its a proof of concept. Assuming you commented tty5 out of /etc/inittab, if Ctrl+Alt+F5 brings you to a login prompt, runit is acting partially or completely correctly. If it brings you to an empty blank screen, runit is not working correctly: Time to troubleshoot.

Init systems are complex, and require a reboot for every troubleshooting iteration. Process managers are complex, and require a restart of the whole process manager for every troubleshooting iteration. You need to attack this with a troubleshooting ladder, going from the highest level, cheapest (quickest) diagnostic tests to the lowest level, difficult and time consuming tests:

Note that Gerrit Pape's example tty uses the getty executable, but your distro might not have getty command, but instead might have agetty or mingetty instead. Use the executable you have in order to enable your service run. For the purposes of this troubleshooting ladder, name your tty5 service "getty-5". Once everything's working, you can name it anything you want. The first few tests should happen before you create symlink /service/getty-5 for file /etc/sv/getty-5 . Also, you might use a different directory instead of /etc/sv/getty-5 , and that's perfectly OK: The concept's the same, just change directory names in your diagnostic tests. Run the exec line of the service's run script, whose filename is probably something like /etc/sv/getty-5/run , at the command prompt, minus the exec part of the command. Run it as root at the command prompt. If this doesn't work, meaning it doesn't produce a login prompt when you press Ctrl+Alt+F5, troubleshoot until it does. Use any error messages the command produces, as well as various ps commands, to narrow it down. [root@openrc ~]# /sbin/agetty 38400 tty5 linux

The preceding command should run without returning, and should throw up a login prompt on tty5 (Ctrl+Alt+F5). After you log in and out of tty5, the command should return to the command prompt.

Note that the preceding command runs tty5 only once, meaning that if you log in and log out, tty5 does not reprint a login prompt nor accept user input.

Note that the preceding command might act funny if runit is already supervising tty5, and may not accept a password. In that case, just keep trying till the screen erases and you have just one login prompt, at which time the preceding command terminates and returns to the command prompt. Run the run service's run script ( /etc/sv/getty-5/run , for instance), as root, at the command prompt. If it doesn't produce a login prompt when you press Ctrl+Alt+F5, troubleshoot until it does. [root@openrc ~]# /etc/sv/getty-5/run

The preceding command should run without returning, and should throw up a login prompt on tty5 (Ctrl+Alt+F5). After you log in and out of tty5, the command should return to the command prompt.

Note that the preceding command runs tty5 only once, meaning that if you log in and log out, tty5 does not reprint a login prompt nor accept user input.

Note that the preceding command might act funny if runit is already supervising tty5, and may not accept a password. In that case, just keep trying till the screen erases and you have just one login prompt, at which time the preceding command terminates and returns to the command prompt. runsv /etc/sv/getty-5 Note that the preceding command runs tty5 only once, meaning that if you log in and log out, tty5 does not reprint a login prompt nor accept user input.

Before continuing this troubleshooting ladder, make sure to terminate tty5 either by pressing Ctrl+C on the runsv command, or logging into and out of tty5.

command, or logging into and out of tty5. Note that the preceding command might act funny if runit is already supervising tty5, and may not accept a password. In that case, just keep trying till the screen erases and you have just one login prompt, at which time the preceding command terminates and returns to the command prompt.

This was the final step on the troubleshooting ladder that should run tty5 only once. All further steps, if everything is functioning correctly, rerun tty5 over and over again, which means you can log in and log out of tty5 as often as you want. ln -s /etc/sv/getty-5 /service/getty-5 The preceding command should run the service within five seconds, and should rerun it every time it ends. That rerunning means that you can log into tty5 and log out, and you'll get another login prompt. If you don't get another login prompt, it's not rerunning the service. svstat /service/getty-5 Rerun the preceding command several times. Output should look like: /service/getty-5: up (pid 1051) 12386 seconds, stopped Pay no attention to the "stopped" at the end. As you repeatedly run it, it should be "up" instead of "down" every time, and unless you logged into and out of tty5, it should display the same PID and an ever increasing time.

Pay no attention to the "stopped" at the end. As you repeatedly run it, it should be "up" instead of "down" every time, and unless you logged into and out of tty5, it should display the same PID and an ever increasing time. If, every time you run the command without logging into and out of tty5, it keeps having an uptime less than 5 seconds and a different PID, this indicates that this service's run script is failing. Use the sv down /service/getty-5 command to disable the service, then go back to running that run script on the command prompt in order to figure out why it runs at the command prompt but not from within runit.

command to disable the service, then go back to running that run script on the command prompt in order to figure out why it runs at the command prompt but not from within runit. Every time you log out and back in, what runit should do is rerun the service at a different PID, and start from up-time 0 again. Use the sv up /service/getty-5 and sv down /service/getty-5 commands to enable and disable the tty5 service. View the results with the ps command and the svstat /service/getty-5 command. Make sure that /etc/runit/1 has the proper commands in it. It should look something like the following: #!/bin/sh PATH=/command:/sbin:/bin:/usr/sbin:/usr/bin /etc/init.d/rcS /etc/init.d/rmnologin touch /etc/runit/stopit chmod 0 /etc/runit/stopit The rmnologin is not a part of and is not supplied by runit. Just make your own. This command deletes the /etc/nologin file. On my system, that's all it does. If your system doesn't have the rmnologin shellscript, just make a shellscript that deletes /etc/nologin . The rcS command is not a part of and is not supplied by runit. I simply made my own do-nothing shellscript to eliminate the error message. Make sure that /etc/runit/2 is as supplied by runit. It should look something like this: #!/bin/sh PATH=/command:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin exec env - PATH=$PATH \ runsvdir -P /service 'log: .............................................................. ......................................................................................... ......................................................................................... ......................................................................................... ..................................................................' WARNING: Do not copy and paste the preceding. I put linefeeds in the series of dots to accumulate a browser. In the actual /etc/runit/2 shellscript, there should be a series of 400 dots without any linefeeds. If you have an unusual path on your computer, you can accommodate it by changing the PATH= command in this shellscript. The runsvdir command polls every subdirectory in the directory named by its argument ( /service in this example), and runs each. It keeps polling every directory, and reruns any that crash. When you've gotten all the preceding to work, it's time to test it by rebooting. At the grub prompt's kernel line, to the end append a space and then /sbin/runit-init so that this boot boots runit. If the boot goes badly, you can always go back to the system's current default boot.

Using the preceding troubleshooting ladder enables the majority of your diagnostic tests to take very little time. Although I normally recommend Divide and Conquer (half splitting) as the most effective troubleshooting strategy, in this case the ladder takes maybe ten extra minutes if everything was already set to work, but can save hours if there was a problem.

Packaging System Limitations

Hey, I like packaging systems. Without them, I'd have never installed Linux. I typically use over a hundred user applications, and I wouldn't want to manually compile each one. If package managers weren't huge timesavers, I'd have been using FreeBSD years ago. But the last time I installed FreeBSD, it had several conflicting, surprising and error prone package managers. In fact, surprising or ineffective packaging systems are the #1 reason I reject distros and BSD versions. The reason I've stuck with Ubuntu and Debian for seven years was because their apt-get based package management just worked 95% of the time. In other words, a good package manager is worth its weight in gold.

But package managers aren't optimal for every situation.

The most obvious situation I ran across was when I was a developer for the VimOutliner project. We had great install instructions, and for those not wanting to read and follow instructions, we had a trivial install shellscript. It just worked. But various distros packaged VimOutliner, and some of those packages took liberties with the software, including changing some timesaving keystrokes to time-consuming keystrokes. And besides, the packages often didn't work, and when confused users emailed us for help, we couldn't help, because we didn't know what the distro was doing with our software.

Gnome needs a package. Who in their right mind will keep on ./configure;make;make install, time after time, while installing its numerous dependencies. It's nice to have a package for Vim: I'd rather install Vim in five minutes than spend hours tracking down all the dependencies and figuring the root cause of C language errors.

But there are occasional pieces of software so important that you'll hand compile to get the software's full feature set, get the latest version, or just get it to work at all. I don't think I can count on my fingers the number of times I've hand compiled LyX myself: LyX was mission-critical for my business, and when I needed a version with a needed feature, or a version not having the bug that the package-bestowed LyX had, I hand compiled. No biggy: It was important.

Another piece of software I always hand-compile is daemontools. Ubuntu and Debian just couldn't document its installation and use like the program's author, djb, did. So on Ubuntu and Debian, it just took too much trial and error and experimentation to do it right. By doing it djb's way, I could install it on Ubuntu, Debian, OpenSuSE, OpenBSD, or anything else.

I think recent events in the world of Linux have elevated init systems to "compile it yourself" importance. If you want a specific init system (or don't want one specific init system), your choice of init systems mustn't be limited by the packages offered by the distro. Packages get broken, packages get dropped. Runit is particularly suited to hand compilation and installation, because runit is so simple, and has so few compile dependencies, that it's probably easier to compile and install it via Gerrit Pape's instructions.

One place I always like to do an end-run around the packaging system is when I disable or replace processes. For instance, I'm more of a startx kinda guy than one who boots to GUI and types in his password. A few years ago I was a Ubuntu user, and if you know Ubuntu, you know that startx is contrary to their whole philosophy, and it's very difficult to eliminate the display manager via the package manager, in order to get the chance to run startx. No problem, I rigged a lightdm config file to exit immediately, so my computer booted to a command prompt. Another example: On Manjaro/OpenRC, when I wanted to boot to CLI, I didn't use the package manager to remove xdm, I just removed its symlink out of /etc/runlevels/default. I can't count the number of times I've renamed something to keep it from running, or swapped in a symlink to a different executable to run something else. Sure, people criticize me for this, but it gives me lots more time to do real work.

Bottom line: Sometimes it's better to go around the packaging system than to jump through all its hoops, and hope that everything won't change with the next release. Which brings up one last topic...

If your experience is anything like mine, the distro mailing list throws rotten tomatoes at you when they find out you've been doing end runs around the packaging system. When you ./configure;make;make install they become indignant, sometimes demanding you make a package out of your hand compile before installing it. Oh, and if they find out you renamed something to disable or replace it, they go on the warpath, telling you just how completely and utterly wrong you are.

Ignore them.

Here are the facts: It takes 5 seconds to rename something. It takes a couple minutes to do it with the package manager. But the operant fact is this: It takes hours to research exactly which packages you need to install or uninstall. They now have packages to uninstall other packages, for gosh sake.

So, if you install runit and don't want to mess with packaging or mess with the dangerous and confusing grub2, you can always do this:

[slitt@mydesk ~]$ cd /path/to/systemd [slitt@mydesk ~]$ mv systemd systemd.org [slitt@mydesk ~]$ cp -p /sbin/runit-init systemd

Is the preceding kludge a stain on my technical ability, or a stain on packaging systems and grub2? I'll leave that question as a challenge for the reader, and I'll simply add that the kludge takes less than a minute.

Another audience who throws rotten tomatoes are the anti-systemd purists. They'll tell you the only true solution is to exile all systemd-related software from your machine, and that requires package management. I completely understand their viewpoint: I've exiled every KDE library from my computers. But KDE doesn't have its tentacles in everything from window manager geegaws to PID1. What I would tell the anti-systemd purists is this: The perfect is the enemy of the good. If you start by package-manager removing everything systemd, as a practical matter you'll be patching things up for the next several days or weeks. If you simply leave the systemd stuff where it lays, but replace its PID1 duties with another init system, then later, one by one, you can replace other systemd components. Either way, it takes a long time to have a systemd free computer. But if you do it my way, you have an operational computer the whole time, plus you really know the internals of your computer, so the package management crowd can't keep making your life difficult with an ever-lengthening parade of init system "improvements".

Manjaro's Pretty Cool

Manjaro's my kind of desktop Linux. It installs easily, like Debian. It has a good package manager, called pacman, that keeps the OS and all but the most customized or rare applications running well. It has respect for those who want systemd, and those who want OpenRC, and makes either pretty easy to use. Because Manjaro is a derivative of Arch, most of the Arch documentation applies, and the Arch documentation is known far and wide to be outstanding.

Speaking of Arch, Manjaro's a descendent of Arch, with these differences:

Easy, automated install

Friendly to alternate init systems such as OpenRC

Out of the box, I found Manjaro 8.11 OpenRC edition to feel pretty much like Debian Wheezy. Easy install, "Just Works", great package manager. The one difference is that Manjaro is a rolling release, with the advantage that you never need to version-upgrade, and the disadvantage that a bad update can leave your computer a brick. When using the pacman packager, get very acquainted with the --ignore option. pacman refuses to upgrade anything if even one package is defective (for instance, bad signature), so when such things go wrong, use --ignore to get the majority upgraded, and then in your leisure address the bad package.

WARNING: When you use rolling releases, you need to back up all the way to the bare metal, because OS re-installation will be a nightmare.

I haven't seen too much of the overall Manjaro community, but I'm quite pleased with the OpenRC section of the Manjaro community.

A Few Manjaro Tips

Manjaro's a very special distro, because although it comes with systemd, it's relatively easy to replace the systemd with a sysvinit->OpenRC combination. And unlike many distros, its developers aren't hostile to any particular init system. In fact, some Manjaro people have put together a Manjaro/OpenRC DVD image, making it trivial to install an OpenRC equipped Manjaro. It's available here:

https://forum.manjaro.org/index.php?topic=19215.0

I suggest you pay particular attention to the following three things listed as missing on the ISO's installer:

Hostname set via installer probably wont work, can be set in /etc/conf.d/hostname Console keymap can be set in /etc/conf.d/keymaps /tmp can be added as tmpfs in /etc/fstab

As examples, here are the examples from my computer:

hostname="littmanj" keymap="us" Note: keymaps file contains some other fairly obvious config options tmpfs /tmp tmpfs nodev,nosuid 0 0

Download and burn the ISO and install the DVD, using "Use whole disk" so you have only one partition, fix up the three items listed above, and if your experience is anything like mine, you'll have a very nice working distro in which the kernel passes control to a sysvinit PID1, which promptly yields control to OpenRC.

Now, with a working computer, you can quickly experiment.

Debian's Seen Better Days

These are tough times for Debian. Just now, after all these months, the systemd wars on their user mailing list are winding down. I think the major reason for the relative calm is that most of us who objected to systemd have gone to other distros, or rolled our own. The systemd patriots on the Debian lists probably consider this a good thing, and for those who departed, it is. For Debian, I have a feeling that long term, this exodus of people with the desire and tech chops to control their own computers will dramatically reduce both Debian's mindshare and credibility.

The project really shouldn't have hidden behind their constitution, their CTTE, GR, mailing list rules, "upstreams", arcane package manager commands, and all that stuff. They shouldn't have made these things the user's problem. Because, in the end, a lot of us wanted a working system with interchangeable parts, not project baggage.

The systemd patriots on the list were forever characterizing those of us wanting choice (and obviously systemd is built from the ground up to limit choice) as "haters", "offtopic", "spammers" and worse. They told us that if we didn't like it, we were free to make our own.

So we did.

Some of us are de-systemding Debian, some of us have moved to other distros, and some of us are making software rendering systemd moot on any distro. All these alternatives are excellent, and I want to make it clear that any ill feeling I might (or might not) have toward Debian do not extend to the Devuan project (formerly called the "Debian fork") or the things being discussed on the modular-debian mailing list. To the extent possible, I'll work to help any and all de-systemd'd Debian forks or descendants. We're all in the same boat.

The future is a crystal ball. Time will tell. My best guess is that Debian's best days are behind it.

Wrapup, 12/17-12/19/2014

I installed runit on a native sysvinit->OpenRC Manjaro system (native because somebody made an OpenRC edition, normal Manjaro is systemd). At this point, my OpenRC setup runs only devfs, sysfs, and udev, with runit handling the remainder. I'm pretty sure that the only need for devfs and sysfs is because they're required by udev. Once I find runit init scripts for udev, and possibly the other two, I can pretty much have my way with the initialization of any distro. Perhaps it would be fun to initialize Fedora with runit: how's that for irony?

The system I described from 12/17 to 12/19 is nowhere near robust enough for daily work. There are no provisions for edge cases, or even common cases like multiple partitions. It doesn't prophylactically run fsck after X number of boots, and not doing those fscks is dangerous in production.

But that's not the point. The point is that I, a Troubleshooting Trainer with no admin experience, managed to get runit initting a system. Replacing an init system, even without help from the package manager, is not as prodigious a task as many people think, and perhaps not as prodigious a task as other people want you to think. It's going to be documented, by me or by others, very shortly. And then, those of us who don't like systemd's architecture can begin to replace the systemd weld-ons, like udev, with non-systemd replacements.

12/21/2014: RichFelker->OpenRC

A few days ago, Rich Felker licensed his 16 line init program with a free software license (see bottom of page for both license and code). That day I fulfilled a long ambition to boot off it, and it was cool.

First, to get gcc -Wall to work right, I needed to add #include<sys/wait.h> to the list of includes. Then it compiled no errors, no warnings:

[root@openrc ~]# gcc -Wall richfelker.c [root@openrc ~]#

Next, I copied a.out to /usr/bin/init.richfelker. I didn't have an /etc/rc file, so I made one that looks like this:

#!/bin/sh /usr/bin/openrc sysinit /usr/bin/openrc boot /usr/bin/openrc default

Then I booted to it, adding init=/usr/bin/init.richfelker to the end of the Grub2 kernel line.

It booted, with three caveats:

There were no virtual terminals It shut down badly, requiring a journal restore on every reboot. It starts without network facilities, so you need to run a shellscript to start the network.

Other than the preceding, it booted perfectly to the display manager xdm. The referenced network-enabling shellscript is the same one I used in the December 17, 2014 section.

Respawn and Virtual Terminals

It turns out that OpenRC has no facility for respawning processes it manages, so OpenRC itself cannot manage virtual terminals. I tried the following OpenRC init script:

#!/usr/bin/openrc-run # Copyright 1999-2007 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 depend() { need localmount after bootmisc } start() { ebegin "Starting getty2" /usr/bin/start-stop-daemon --start --background \ --exec /usr/bin/agetty --make-pidfile \ --pidfile /run/getty2.pid -- 38400 tty2 linux eend $? } reload() { ebegin "ReStarting getty2" /usr/bin/start-stop-daemon \ --exec /usr/bin/agetty \ --pidfile /run/getty2.pid -s1 eend $? } stop() { ebegin "Stopping getty2" start-stop-daemon --stop --exec /usr/bin/agetty --pidfile /run/getty2.pid eend $? }

Close, but no cigar. The preceding runs agetty once, because OpenRC has no native, idiomatic way of respawning a service the way sysvinit and runit and even daemontools can do. I tried putting the getty call in a looping shellscript, but no joy. Some bizarre property of the agetty command reaches back and kills the thing that started it. So if you want virtual terminals spawned in a RichFelker->OpenRC setup, you need to have OpenRC run daemontools, and let daemontools manage the virtual terminals.

Rich Felker's 16 line init program is potentially a great way to start OpenRC, or maybe even a tweaked daemontools or daemontools-encore, but more work needs to be done to make sure such a setup can respawn services to run virtual terminals, shut down cleanly, and automatically run networking. This could be an excellent alternative, because in 2014, I think many of us would prefer to keep sysvinit (and Upstart and systemd) off our computers. If you find new and better ways to use Rich Felker's 16 line init, either by managing what descends from /etc/rc, or by modifying Rich's code, please email me.

12/23/2014: ip Only Network Start Script

A few days ago I made a Network Start Script to start the network in a runit->OpenRC init with minimal use of OpenRC. I was quickly informed that ifconfig is deprecated, and everything should be done with the ip command. So I changed the script, replacing all ifconfig and route commands with their corresponding ip commands. The following is the new, ip only Network Start Script:

#!/bin/sh hostname -F /etc/hostname ip link set dev lo up ip link set dev eth1 down ip addr add 192.168.100.88/24 dev eth1 ip link set dev eth1 up ip route add default via 192.168.100.96

[ Training | Troubleshooters.Com | Email Steve Litt ]

12/23/2014: Pure Runit Init Proof of Concept

No systemd. No sysvinit. No OpenRC. No dbus. No package manager. Just pure runit, aided by a shellscript to enable devfs and a shellscript to enable udev. Boots Grub menu to command prompt in 8 seconds, shuts down from LXDE to poweroff in 5. Plays Youtube videos, including sound. Due to lack of dbus, it can't run a display manager (xdm, lxdm, gdm etc) but startx starts LXDE just fine. The /etc/runit/1 script follows:

#!/bin/sh # system one time tasks PATH=/command:/sbin:/bin:/usr/sbin:/usr/bin ### START devfs /root/run_devfs.sh ### START sysfs # TODO, but seems not to matter ### START udev /root/udev_start.sh ### ENABLE SOUND MIXING WITHOUT ALSA, PULSEAUDIO (USE rexima or aumix) modprobe snd_mixer_oss ### SET UP NETWORK WITHOUT systemd, NetworkManager, dbus or wicd ### ### SORRY LENNART ### /root/upnet.sh /etc/init.d/rcS /etc/init.d/rmnologin touch /etc/runit/stopit chmod 0 /etc/runit/stopit

In the preceding, the following shellscripts take the place of formerly OpenRC-started functionalities:

/root/run_devfs.sh : This starts the devfs filesystem, which is necessary for running terminals and ssh'ing into the computer. This is a nasty, convoluted script.

: This starts the devfs filesystem, which is necessary for running terminals and ssh'ing into the computer. This is a nasty, convoluted script. /root/udev_start.sh : This is a fairly simple script that starts udev, at least enough so you can load and configure your network devices.

: This is a fairly simple script that starts udev, at least enough so you can load and configure your network devices. modprobe snd_mixer_oss : This is necessary if you want to use a mixer on a distro without ALSA or Pulseaudio.

: This is necessary if you want to use a mixer on a distro without ALSA or Pulseaudio. /root/upnet.sh : This starts up the network, hard coding it to 192.168.100.88/24.

The remainder of this section discusses each of the shellscripts, and then discusses the true meaning of a pure runit init.

/root/run_devfs.sh

This is one of the ugliest shellscripts I've ever made. Actually, I didn't really make it: I re-molded Manjaro's devfs OpenRC init script into a shellscript. It's ugly, I have no doubt it's fragile, I'd be shocked if it were portable, but it works, and that's a good sign for a proof of concept. The shellscript follows:

#!/bin/sh # Copyright (c) 2007-2008 Roy Marples <roy@marples.name> # Released under the 2-clause BSD license. # Heavily kludged by Steve Litt in 2014 description="Set up the /dev directory" mount_dev() { local mountopts="exec,nosuid,mode=0755" if grep -q devtmpfs /proc/filesystems; then devfstype=devtmpfs mountopts="$mountopts,size=10M" elif grep -q tmpfs /proc/filesystems; then devfstype=tmpfs mountopts="$mountopts,size=10M" fi if [ -n "$devfstype" ]; then mount -n -t $devfstype -o $mountopts dev /dev else echo "WARNING This kernel does not have devtmpfs or tmpfs support, and there" echo "WARNING is no entry for /dev in fstab." echo "WARNING This means /dev will not be mounted." echo "WARNING To avoid this message, set CONFIG_DEVTMPFS or CONFIG_TMPFS to y" echo "WARNING in your kernel configuration or see /etc/conf.d/devfs" fi } seed_dev() { # Seed /dev with some things that we know we need # creating /dev/console, /dev/tty and /dev/tty1 to be able to write # to $CONSOLE with/without bootsplash before udevd creates it [ -c /dev/console ] || mknod -m 600 /dev/console c 5 1 [ -c /dev/tty1 ] || mknod -m 620 /dev/tty1 c 4 1 [ -c /dev/tty ] || mknod -m 666 /dev/tty c 5 0 # udevd will dup its stdin/stdout/stderr to /dev/null # and we do not want a file which gets buffered in ram [ -c /dev/null ] || mknod -m 666 /dev/null c 1 3 # so udev can add its start-message to dmesg [ -c /dev/kmsg ] || mknod -m 660 /dev/kmsg c 1 11 # extra symbolic links not provided by default [ -e /dev/fd ] || ln -snf /proc/self/fd /dev/fd [ -e /dev/stdin ] || ln -snf /proc/self/fd/0 /dev/stdin [ -e /dev/stdout ] || ln -snf /proc/self/fd/1 /dev/stdout [ -e /dev/stderr ] || ln -snf /proc/self/fd/2 /dev/stderr [ -e /proc/kcore ] && ln -snf /proc/kcore /dev/core # Mount required directories as user may not have them in /etc/fstab for x in \ "mqueue /dev/mqueue 1777 ,nodev mqueue" \ "devpts /dev/pts 0755 ,gid=5,mode=0620 devpts" \ "tmpfs /dev/shm 1777 ,nodev,mode=1777 shm" \ ; do set -- $x grep -Eq "[[:space:]]+$1$" /proc/filesystems || continue mount | grep $2 && continue if [ ! -d $2 ]; then mkdir -m $3 -p $2 >/dev/null >&1 || \ echo "WARNING Could not create $2!" fi if [ -d $2 ]; then echo "Mounting $2" if ! mount | grep $2; then mount -n -t $1 -o noexec,nosuid$4 $5 $2 fi fi done } startt() { mount_dev seed_dev } startt

The preceding is as ugly as it gets, and I bet over half of it is fluff for edge cases. I doubt it would work other places, but it works on my pure runit Manjaro machine. If my pure runit setup has an Achilles Heel, this is it.

My finding was what I need really needed from this script was /dev/pts so that I could ssh into the computer, and so LXDE could run terminals with bash. I left the rest of it in because I didn't know which things were a prerequisite of /dev/pts.

If you try this script and find that it doesn't work for you, or if you need to modify it, please email me so I can document more knowledge on devfs.

/root/udev_start.sh

Five lines and you're done:

#!/bin/sh /usr/bin/udevd --daemon /usr/bin/udevadm trigger --action=add --type=subsystems /usr/bin/udevadm trigger --action=add --type=devices /usr/bin/udevadm settle

I have absolutely no idea what any of the preceding means: It was given to me by user Subsentient on #epoch on Freenode. Subsentient is the main developer for the Epoch Init System, another one I want to try very soon.

What I know is that the preceding script works on Subsentient's machine, it works on my machine, so it's probably somewhat portable, and if not, it's easy for anyone familiar with the ins and outs of udev to tweak into working.

Thank you, Subsentient!

/root/upnet.sh

I've talked about this shellscript before. I made it, and as far as I know, it's constructed well and will work anywhere the ip command is available, which, as far as I know, is any Linux distro. The shellscript follows:

#!/bin/sh hostname -F /etc/hostname ip link set dev lo up ip link set dev eth1 down ip addr add 192.168.100.88/24 dev eth1 ip link set dev eth1 up ip route add default via 192.168.100.96

The preceding shellscript completely removes the need for playpen apps like NetworkManager and wicd, at least if you're dealing with wired Ethernet. As far as Wifi, some time in the not too distant future I'm going to make a NetworkManager workalike that relies on wpa_supplicant, iwlist, ip, and a little sudo fu. No dbus necessary.

Why This Is Important

The ability to lay down a hand compiled and configured runit right alongside a distro's normal init, and simply run runit instead of what the distro had in mind, is huge. It takes the distro's packagers right out of the equation. If your distro decides to reach in and replace your init system, you can simply lay down a pure runit init, back up their /usr/bin/init, and replace it with your own. Or better yet, make a tiny shellscript, so that every time an upgrade replaces /usr/bin/init, you run the shellscript to get runit back. The shellscript would look something like this:

#!/bin/sh datestamp=`date +"%Y%m%d_%H%M%S"` cd /usr/bin cp -p init init_thoseguys$datestamp cp -p runit-init init

NOTE: The reason I keep saying /usr/bin/init instead of /sbin/init is because Manjaro is one of those "forward thinking" distros whose /sbin is just a symlink to /usr/bin. Guess whose idea that was? Anyway, runit aborts to a busybox prompt if you init to it in a symlinked directory. So if you have one of those Freedesktop.org inspired distros, just init to /usr/bin/init and be happy. By the way, this is about the only downside I've found to Manjaro in my three weeks of testing it out.

Once the documentation for hand compiled, hand configured runit becomes accurate, complete with better shellscripts to start devfs, sysfs, udev, and even dbus, as well as /etc/runit/1, it becomes almost trivial to convert almost any distro to runit, always assuming you're not married to Gnome or Unity or KDE or some other "Desktop Environment" that does it all for you. And if you like such a desktop environment, you probably wouldn't mind systemd, so the point is probably moot.

NOTE: If you like a heavy, integrated desktop environment but don't want systemd, check out the Devuan project. They're not shipping yet, but their distro will have all of Debian's capabilities, without the systemd baggage.

Anyway, with the ability to lay down a clean and simple runit system, you control your computer with an iron hand.

But is this really ready for prime time?

Of course not. Like I said, I have serious doubts about the portability of my /root/run_devfs.sh script. And I'm sure many other problems will pop up moving this procedure from distro to distro. None of which matters.

The point is, sooner or later a critical mass of knowledge will develop about standalone runit, and accurate, easy documentation will follow. What you're reading now is a proof of concept, but soon you'll be able to do this on any distro you want.

12/24/2014: Pure Epoch Init System

The afternoon of 12/24 I finished my research on the Epoch Init System, and ran a "hello world" to boot with an Epoch init and make available a virtual terminal. By that night, I had a full blown, Epoch-initted, Linux desktop machine running on my Manjaro Experimental Computer. It worked, almost out of the box. One of the reasons I got it running so fast is that, of all the init systems, Epoch has some of the best documentation:

Main Page http://universe2.us/epoch.html Site map Bottom of http://universe2.us/epoch.html Config file howto http://universe2.us/epochconfig.html Epoch commands http://universe2.us/epochmanagement.html FAQ http://universe2.us/epochfaq.html Downloads Bottom of http://universe2.us/epoch.html Some misc resources http://universe2.us/collector/ Sample epoch.conf http://universe2.us/collector/epoch.conf

On most inits, the hardest thing to do is formulate the init scripts or the, for want of a better word, "unit files". I guess you could say that Epoch's config more resembles "unit files", but they're all in one file, /etc/epoch/epoch.conf. One huge benefit of Epoch is its author, a guy named Subsentient, has provided a sample epoch.conf. In fact, it's one he uses to init Fedora. Think about that for a second. Anyway, the sample epoch.conf is located at http://universe2.us/collector/epoch.conf. That sample file clears up a lot of questions you'll have at first, and contributes to quick init replacement.

Among all the init systems I've found so far, Epoch has the best documentation. It's also extremely simple, and it's a very easy replacement for your current init.

Perhaps I should define "replacement". Some folks, let's call them the Purists, define it as removing every vestige of the old init, and replacing it with the new init. It's like ripping up the old highway, and paving a new, better highway right over the site of the old one. That's not how I use Epoch (or runit).

The other definition of "replacement" is building a better highway in parallel with the first, perhaps even using portions of the former highway in the route, and just changing the signs so that people follow the new route. This is the sense in which I replaced my current init with Epoch (same's true for runit). My way's quicker and easier for a bunch of different reasons:

I don't have the bother of removing the old stuff

If things temporarily go wrong with the new one, I can still boot to the old one (via an init= on the Grub kernel line). This is great for recovering from mistakes, for A/B regression testing, and the like.

on the Grub kernel line). This is great for recovering from mistakes, for A/B regression testing, and the like. I can use parts of the old one. For the time being, I'm still using the systemd-provided Udev. Imagine how difficult things would be if I needed to make a replacement for every last part before beginning construction on the new init!

My way is much more conducive to construction without a package manager, and, after all, it wouldn't be too much of a stretch to say that what really got us into the systemd mess was package managers.

Bottom line, I'm a huge fan of laying down a parallel init system and using that. Much easier, much faster, no package manager interference.

Another advantage of Epoch is its reasonably simple service configurations. Epoch's config is declarative like systemd. Once you learn the ins and outs from the docs and sample epoch.conf, it's fairly easy. I'm not saying scripts are guaranteed to be difficult: runit scripts are simple. But one look at sysvinit scripts or OpenRC scripts is enough to make you run screaming, and I think the #1 reason for resistance to various init systems is the difficulty of configuring individual services. Bottom line, unlike sysvinit or OpenRC, you don't need to be a professional Admin to perform Epoch service config: A mere civilian can handle the job quite well.

So which is better, Epoch or runit? I don't know, which would you prefer, having your robot win the 220 pound combat title at Robogames, or driving a Tesla controlled by your hand programmed Raspberry Pi? Some choices are just too good to be contemplated. In making such a decision, I'd look at a few guidelines:

If you need to replace the init system in a hurry (a few hours), Epoch has the advantage.

If you're starting and controlling a large number of daemons (30 perhaps), runit has the advantage.

If you're a huge fan of daemontools, runit has the advantage. However, I am a huge fan of daemontools, and still like Epoch a lot.

If you prefer scripts rather than "declarative syntax", runit has the advantage, although of course Epoch can manage scripts.

If you prefer "declarative syntax" to scripts, Epoch has the advantage.

You can't go wrong with either one.

I'm stoked about the Epoch Init System because it's a replacement init that a fairly smart computer user can use to replace his current init in less than a day. Let others argue the pros and cos of systemd: Systemd won't darken your door again after that day of setting up Epoch.

All init systems are somewhat complex, some more than others. To maximize your likelihood of a straight forward route to an Epoch initted computer, I've written the next section, called Getting Epoch Running...

Getting Epoch Running

This section is long for one and only one reason: I've written it explicitly that someone who has never before installed and configured an init system can do so, with a minimum of ambiguity, confusion or surprise. Its length has nothing to do with it being complicated: In fact, it's the simplest and fastest to install and configure init system I've to date, and also the best documented. So let's get started...

First, let me repeat the official Epoch documentation pages.

Main Page http://universe2.us/epoch.html Site map Bottom of http://universe2.us/epoch.html Config file howto http://universe2.us/epochconfig.html Epoch commands http://universe2.us/epochmanagement.html FAQ http://universe2.us/epochfaq.html Downloads Bottom of http://universe2.us/epoch.html Some misc resources http://universe2.us/collector/ Sample epoch.conf http://universe2.us/collector/epoch.conf

The preceding official documentation pages are spectacularly helpful. Use them!

Starting on 12/24/2014, and continuing until there's a newer stable version, the version you want to download is 1.2.0, code name "Peroxide". Make yourself a blank directory (I used /root/build), and within that directory untar the tarball. The following is what you get, except that before you compile, the built and objects directories are empty:

build `-- epoch-1.2.0 |-- buildepoch.sh |-- built | |-- bin | | `-- wall -> ../sbin/epoch | `-- sbin | |-- epoch | |-- halt -> ./epoch | |-- init -> ./epoch | |-- killall5 -> ./epoch | |-- poweroff -> ./epoch | |-- reboot -> ./epoch | `-- shutdown -> ./epoch |-- CHANGELOG |-- objects | |-- actions.o | |-- config.o | |-- console.o | |-- main.o | |-- membus.o | |-- modes.o | |-- parse.o | `-- utilfuncs.o |-- README |-- src | |-- actions.c | |-- config.c | |-- console.c | |-- epoch.h | |-- main.c | |-- membus.c | |-- modes.c | |-- parse.c | `-- utilfuncs.c `-- UNLICENSE.TXT

Observe the following command:

buildepoch.sh

The preceding command fills in the built and objects trees. If you want to use special arguments to buildepoch.sh, that's doable, but that's getting beyond the scope of this section. Anyway, after running buildepoch.sh, built/sbin/epoch is the Epoch init executable. Copy it wherever you want, but be sure to back up the old version of anything you copy it over. I copied the file temporarily to /e, because I wanted an easily typeable name to put in Grub2's kernel line's init=/e variable.

But don't boot yet. To tell the Epoch init system what to do, you need to configure /etc/epoch/epoch.conf. I started with the Sample epoch.conf file, commented out most of it, uncommented only stuff I absolutely needed, one at a time. Here's the /etc/epoch/epoch.conf I now use to boot up my Manjaro computer to a reasonable desktop computer, with an LXDE window manager and ability to play Youtube videos:

BootBannerText=Sorry, Lennart! BootBannerColor=CYAN Hostname=FILE /etc/hostname DefaultRunlevel=boot EnableLogging=true DisableCAD=true BlankLogOnBoot=true MountVirtual=procfs sysfs devpts+ devshm+ # DEFINE PRIORITY CONSTANTS DefinePriority=Rwfs_Start 30 DefinePriority=Early_Getty_Start 40 DefinePriority=Sysclock_Start 35 DefinePriority=Udev_Start 20 DefinePriority=Network_Start 50 DefinePriority=Dev_Mixer_Start 60 DefinePriority=Typical_Daemon_Start 80 DefinePriority=Typical_Getty_Start 70 DefinePriority=Display_Manager_Start 120 DefinePriority=Display_Manager_Stop 40 DefinePriority=Typical_Daemon_Stop 10 DefinePriority=Sysclock_Stop 80 DefinePriority=Killall5_Soft_Stop 90 DefinePriority=Killall5_Stop 91 DefinePriority=Rwfs_Stop 100 ObjectID=getty2 ObjectDescription=Early getty on /dev/tty2 ObjectStartCommand=agetty tty2 & ObjectStopCommand=NONE ObjectStartPriority=Early_Getty_Start ObjectStopPriority=0 ObjectEnabled=true ObjectOptions=SERVICE AUTORESTART ObjectRunlevels=boot # MAKE readwrite just after udev instantiates, # MAKE readonly again after all killalls ObjectID=rwfs ObjectDescription=root filesystem read-write support ObjectStartCommand=/usr/bin/mount -o remount,rw / ObjectStopCommand=/usr/bin/mount -o remount,ro / ObjectStartPriority=Rwfs_Start ObjectStopPriority=Rwfs_Stop ObjectEnabled=true ObjectRunlevels=boot # KINDLY request all processes to end themselves # Give them 2 seconds to do so. ObjectID=killall5_soft ObjectDescription=Terminating all processes ObjectStopCommand=killall5 -15 && sleep 2 ObjectStartPriority=0 ObjectStopPriority=Killall5_Soft_Stop ObjectEnabled=true ObjectOptions=HALTONLY RAWDESCRIPTION # MURDER all remaining processes # Give them 2 seconds to do so. # Then hand control to service rwfs to # remount / read-only to prevent filesystem corruption ObjectID=killall5 ObjectDescription=Killing all processes ObjectStopCommand=killall5 -9 && sleep 2 ObjectStartPriority=0 ObjectStopPriority=Killall5_Stop ObjectEnabled=true ObjectOptions=HALTONLY RAWDESCRIPTION ObjectID=sysclock ObjectDescription=Configuring system clock ObjectStartCommand=hwclock -s ObjectStopCommand=hwclock -w ObjectStartPriority=Sysclock_Start ObjectStopPriority=Sysclock_Stop ObjectEnabled=true ObjectOptions=RAWDESCRIPTION ObjectRunlevels=boot # RUN simple shellscript to get udev running # Udev is necessary to start network # Run it before remounting / read-write. # Don't bother to stop it at all ObjectID=udevd ObjectDescription=Starting udev ObjectStartCommand=/root/udev_start.sh ObjectStopCommand=NONE ObjectStartPriority=Udev_Start ObjectStopPriority=0 ObjectEnabled=true ObjectOptions=RAWDESCRIPTION ObjectRunlevels=boot # RUN shellscript to set up network. Works on any distro having # the ip command. No config files necessary. # start it after Udev is up and / is read-write # and the early Getty is running ObjectID=network ObjectDescription=Setting up net devices and the network ObjectStartCommand=/root/upnet.sh ObjectStopCommand=NONE ObjectStartPriority=Network_Start ObjectStopPriority=0 ObjectEnabled=true ObjectOptions=RAWDESCRIPTION ObjectRunlevels=boot # LOAD kernel module for /dev/mixer, without ALSA or PULSE # Run after early Getty and network, but before daemons # No need to stop it on shutdown. ObjectID=devmixer ObjectDescription=Setting up /dev/mixer via oss ObjectStartCommand=/usr/bin/modprobe snd_mixer_oss ObjectStopCommand=NONE ObjectStartPriority=Dev_Mixer_Start ObjectStopPriority=0 ObjectEnabled=true ObjectOptions=RAWDESCRIPTION ObjectRunlevels=boot ObjectID=getty3 ObjectDescription=Getty on /dev/tty3 ObjectStartCommand=agetty tty3 & ObjectStopCommand=NONE ObjectStartPriority=Typical_Getty_Start ObjectStopPriority=0 ObjectEnabled=true ObjectOptions=SERVICE AUTORESTART ObjectRunlevels=boot ObjectID=getty4 ObjectDescription=Getty on /dev/tty4 ObjectStartCommand=agetty tty4 & ObjectStopCommand=NONE ObjectStartPriority=Typical_Getty_Start ObjectStopPriority=0 ObjectEnabled=true ObjectOptions=SERVICE AUTORESTART ObjectRunlevels=boot ObjectID=getty5 ObjectDescription=Getty on /dev/tty5 ObjectStartCommand=agetty tty5 & ObjectStopCommand=NONE ObjectStartPriority=Typical_Getty_Start ObjectStopPriority=0 ObjectEnabled=true ObjectOptions=SERVICE AUTORESTART ObjectRunlevels=boot ObjectID=getty6 ObjectDescription=Getty on /dev/tty6 ObjectStartCommand=agetty tty6 & ObjectStopCommand=NONE ObjectStartPriority=Typical_Getty_Start ObjectStopPriority=0 ObjectEnabled=true ObjectOptions=SERVICE AUTORESTART ObjectRunlevels=boot # MANAGE sshd daemon ObjectID=sshd ObjectDescription=Manage sshd daemon ObjectStartCommand=/usr/sbin/sshd ObjectStopCommand=PIDFILE /run/sshd.pid ObjectStartPriority=Typical_Daemon_Start ObjectStopPriority=Typical_Daemon_Stop ObjectEnabled=true ObjectRunlevels=boot ObjectOptions=SERVICE AUTORESTART # MANAGE crond daemon ObjectID=crond ObjectDescription=Manage crond daemon ObjectStartCommand=/usr/bin/crond ObjectStopCommand=PIDFILE /run/crond.pid ObjectStartPriority=Typical_Daemon_Start ObjectStopPriority=Typical_Daemon_Stop ObjectOptions=SERVICE AUTORESTART ObjectEnabled=true ObjectRunlevels=boot # RUN ONCE Display Manager. # Runlevels=boot to enable, Runlevels=gui to disable to GUI via startx # NOTE: Display Manager might invoke dbus. # NOTE: Toggle ObjectEnabled to turn GUI booting on or off. ObjectID=DisplayManager ObjectDescription=DisplayManager ObjectPIDFile=/run/DisplayManager.pid ObjectStartCommand=/usr/bin/lxdm & ObjectStopCommand=PID ObjectStartPriority=Display_Manager_Start ObjectStopPriority=Display_Manager_Stop ObjectEnabled=true ObjectRunlevels=boot

A few comments on the preceding /etc/epoch/epoch.conf:

Start order goes from 1 up. Stop order also goes from 1 up, so the lowest ObjectStopPriority gets stopped first. Therefore, the killalls should have the highest ObjectStopPriority numbers, because they're the last things to get stopped.

gets stopped first. Therefore, the killalls should have the highest numbers, because they're the last things to get stopped. I'm a big fan of named priority constants using DefinePriority . By doing this, I can reorder start and stop order all in one place, and if a lot of services can start at once, with no defined order relative to each other, I can prioritize or de-prioritize the whole batch.

. By doing this, I can reorder start and stop order all in one place, and if a lot of services can start at once, with no defined order relative to each other, I can prioritize or de-prioritize the whole batch. You should run one getty (virtual terminal) very early in the process, so that if something else hangs or fails, you have an easy way to troubleshoot.

ObjectID=DisplayManager works, but it's "not ready for prime time". It shuts down wrong, its reboot and shutdown facilities do nothing. You need to make your own reboot and poweroff shellscripts, and run those (by click or by dmenu_run ) from the window manager itself.

works, but it's "not ready for prime time". It shuts down wrong, its reboot and shutdown facilities do nothing. You need to make your own reboot and poweroff shellscripts, and run those (by click or by ) from the window manager itself. If you're trying to make a dbus-less system, on Manjaro, any way you run X, dbus is started. However, if you use startx, when you quit your window manager, dbus terminates.

IMPORTANT: I recommend reaching this point using a process rather than trying to achieve it all at once.

The Process

The preceding subsection mentioned the preference of reaching a working system using a process rather than just copying and pasting from this web page. That's because your setup will almost certainly be different than mine, so my epoch.conf probably will fail on your system. Here is the high level of the process I recommend:

Hello World: Shell-Only Init

Any complex learning situation begins with a Proof of Concept, sometimes called a POC or a "Hello World". The purpose of a Hello World is to get the new technology to run in its simplest form, so simple as not to be useful. No error messages, no use case adaptations, nothing but some feedback that yes, it ran. To skip the "Hello World" phase when starting a project with an unknown technology is to trudge the Highway to Heartache.

As your "Hello World", you're going to simply use the Epoch Init System to boot into /sbin/sh or /usr/bin/sh.

NOTE: Luminaries in the world of Linux have seen fit to stop the decades-old split between programs needing to run before mounts have been completed, and ordinary programs, and thus have made /sbin and /bin symlinks to /usr/bin. [root@openrc ~]# ls -ldF /sbin lrwxrwxrwx 1 root root 7 12.07.2014 05:57 /sbin -> usr/bin/ [root@openrc ~]# ls -ldF /bin lrwxrwxrwx 1 root root 7 12.07.2014 05:57 /bin -> usr/bin/ [root@openrc ~]# Hey, don't blame me, I'm just reporting the situation. Anyway, when /bin is a symlink, that symlink might not be established early in the init process, so use the real hard location, /usr/bin/sh. If /bin is a completely different directory, use /bin/sh.

Follow these steps:

Copy your built/sbin/epoch to /e . This makes it much easier, quicker, and more error resistant to, at the Grub menu, edit it to append init=/e . I know that putting files in the root directory is normally a worst practice, but when doing experimentation and troubleshooting, fast cycles and error resistance mean the world. When the system is stable, you can move /e to /usr/bin/init or /sbin/init where it belongs. Copy your built/sbin/epoch to /usr/bin/epoch . Once you've done that, you can use the epoch reboot and epoch poweroff commands to reboot and power down your computer. Please remember that without this step, none of the epoch commands described later in this document will work. Create the following /etc/epoch/epoch.conf : BootBannerText=Sorry, Lennart! BootBannerColor=CYAN Hostname=FILE /etc/hostname DefaultRunlevel=boot EnableLogging=true DisableCAD=true BlankLogOnBoot=true MountVirtual=procfs sysfs devpts+ devshm+ # SHELL sh hello world ObjectID=sh ObjectDescription=Hello world via sh ObjectStartCommand=/usr/bin/sh ObjectStopCommand=NONE ObjectStartPriority=1 ObjectStopPriority=0 ObjectEnabled=true ObjectRunlevels=boot Observe that the ObjectStartCommand does not end in an ampersand. This command is not meant to run in the background, and won't deliver a command prompt if run in the background. Observe the line starting with ObjectID . This begins a service, and the comment line before it pertains to that service. The service in this case simply runs the shell, /bin/sh or /bin/sh Reboot the machine. At the Grub prompt, press e to edit the boot, and add the following string to the end of the kernel line: init=/e The machine should quickly bring you to a shell prompt, where you can do commands like ls and mount . NOTE: This environment is nowhere near a complete one. You'll see warnings complaining about ioctl and warnings. You cannot correctly run the epoch command in this environment, so Ctrl+Alt+Del is the only way you can reboot from this environment. If the preceding steps did not bring you to a command prompt, troubleshoot. Understand that you have now proven that the plumbing of the Epoch Init System works on your computer. Your next step is a Hello World that actually creates a Virtual Terminal to log into.

You're done with this experiment. Use Ctrl+Alt+Del to reboot your computer, and let it boot into its normal, functional init.

Hello World: Getty-Only Init

Experiment with your system to determine exactly how to start a virtual terminal. Depending on your distro, it might use getty , agetty , mingetty , getty or something else. Experiment until you have the exact command, with full path, required to throw up a login on Ctrl+Alt+F9. For instance, on my Manjaro system, it's exec /usr/bin/agetty 38400 tty9 linux or exec /usr/bin/agetty tty9 or nohup /usr/bin/agetty tty9 & It's vital you complete this step before going on. This might not be the exact command used in Epoch, and agetty is really weird, but without knowing how to make a getty at the command prompt, you're dead meat on the next step. Make and boot to a getty-only /etc/epoch/epoch.conf . Experiment with it until it works. Here's a hint: The getty line will probably look something like the following: ObjectStartCommand=agetty tty2 & Within your /etc/epoch/epoch.conf replace entire Shell service with a Getty service, which should look something like the following: ObjectID=getty2 ObjectDescription=Early getty on /dev/tty2 ObjectStartCommand=agetty tty2 & ObjectStopCommand=NONE ObjectStartPriority=40 ObjectStopPriority=0 ObjectEnabled=true ObjectOptions=SERVICE AUTORESTART ObjectRunlevels=boot The ObjectStartPriority is fairly high, because other services, such as Read Write File System and Udev, need to happen before it. In the preceding step, if agetty isn't available on your computer or it doesn't work on your computer, substitute one of the getty commands that worked during your previous experimentation on the command line. Reboot, and append init=/e (the location you copied the epoch executable) to the kernel line. When the boot messages stop coming, press Ctrl+Alt+F2 to access tty2. If you can repeatedly log in as both your username and root, you now have a working proof of concept (Hello World) Epoch Getty implementation. If not, troubleshoot. When you're able to repeatedly log into Ctrl+Alt+F2 as both your username and root, you've proven the concept of creating a Getty. Your next step will be to add code to run on reboot or shutdown.

Add in Killall Processes

The closer you get to making everything exit gracefully, the less likely file corruption or other data anomalies become. Add the following services to the bottom of your /etc/epoch/epoch.conf:

# MAKE readwrite just after udev instantiates, # MAKE readonly again after all killalls ObjectID=rwfs ObjectDescription=root filesystem read-write support ObjectStartCommand=/usr/bin/mount -o remount,rw / ObjectStopCommand=/usr/bin/mount -o remount,ro / ObjectStartPriority=30 ObjectStopPriority=100 ObjectEnabled=true ObjectRunlevels=boot # KINDLY request all processes to end themselves # Give them 2 seconds to do so. ObjectID=killall5_soft ObjectDescription=Terminating all processes ObjectStopCommand=killall5 -15 && sleep 2 ObjectStartPriority=0 ObjectStopPriority=90 ObjectEnabled=true ObjectOptions=HALTONLY RAWDESCRIPTION # MURDER all remaining processes # Give them 2 seconds to do so. # Then hand control to service rwfs to # remount / read-only to prevent filesystem corruption ObjectID=killall5 ObjectDescription=Killing all processes ObjectStopCommand=killall5 -9 && sleep 2 ObjectStartPriority=0 ObjectStopPriority=91 ObjectEnabled=true ObjectOptions=HALTONLY RAWDESCRIPTION

Notice that the two Killall services happen on shutdown only. Notice also the order of the three added services. Service rwfs, which remounts read-write on startup and remounts read-only on shutdown, starts before tty2 so that tty2 logging works. However, to prevent damage by rogue device drivers, when Udev is added later on, it will happen before service rwfs, so that if the system is booted read-only, Udev won't be able to write the filesystem.

The shutdown order is especially interesting. With ObjectStopPriority of 100, the rwfs service is intentionally the last to happen. Two steps before that, service killall5_soft executes first, sending every process a SIGTERM and waiting 2 seconds for everything to terminate itself. Then, service killall5 sends every remaining process a SIGKILL (kill -9) to force their immediate termination, and waits 2 seconds for them to die. Finally, service rwfs sets the file system read-only, and Epoch either powers off or reboots, depending on the runlevel passed to the epoch command.

At this point you have a bootable system that's reasonably considerate to your filesystems. No network, no drivers, you can't ssh into it, but it works.

Add in Sysclock

The hardware clock should start up after Udev but before rwfs. So we'll give it a priority of 35. Add the following to the bottom of your existing /etc/epoch/epoch.conf:

ObjectID=sysclock ObjectDescription=Configuring system clock ObjectStartCommand=hwclock -s ObjectStopCommand=hwclock -w ObjectStartPriority=35 ObjectStopPriority=80 ObjectEnabled=true ObjectOptions=RAWDESCRIPTION ObjectRunlevels=boot

The preceding turns on the hardware clock. Note that the hardware clock should start just after Udev and just before rwfs. It should stop late, but before Killall5_soft.

Adding the sysclock service probably won't have especially striking effects, but it's part of your timekeeping system. After using tty2 to make sure everything's OK, go on to the next step.

Run Network

Friends whose opinions I trust tell me that ifconfig is deprecated, and to use the ip command for everything. So that's what this section does.

With the system booted as indicated in the Add in Sysclock sub-subsection, perform the following command:

ip addr | cut -b 1-60

It should show lo and any eth devices, but instead it shows just the lo device, because no network device drivers are loaded. Not only that, the lack of output from the following command proves that the lo device isn't even up:

ip link show up

This sub-subsection gets a wired Ethernet network all set up, and does so without respect to the silly config files that every distro puts in a different place with a different name. In order to run an Ethernet card, you need the following two events:

Load the network card's device drivers Run an ip command based shellscript to set up lo, eth?, and hostname.

The way I did it was to run Udev, which loaded the right drivers. This has the advantage that I needn't know driver names or details. It has the disadvantage that Udev is systemd code. In this case, I chose the easy way. To run Udev, I use a script which is fairly simple but seems fairly universal. The script, called /root/udev_start.sh, follows:

#!/bin/sh /usr/bin/udevd --daemon /usr/bin/udevadm trigger --action=add --type=subsystems /usr/bin/udevadm trigger --action=add --type=devices /usr/bin/udevadm settle

After running the preceding shellscript, the ip addr command shows the Ethernet card(s) too. Now all that's required is to configure the network. The following is a distro-independent way to configure your ethernet address to 192.168.100.87. Notice that on my Manjaro computer, eth1 is configured to 192.168.100.87, and eth0 has no Ethernet cable plugged into it. The script, called /root/upnet.sh, follows:

hostname -F /etc/hostname ip link set dev lo up ip link set dev eth1 down ip addr add 192.168.100.87/24 dev eth1 ip link set dev eth1 up ip route add default via 192.168.100.96

After running the preceding script, you now have a fully functional wired Ethernet network at 192.168.100.87 with gateway 192.168.100.96. You will probably need to change IP addresses to fit your network, but that's OK. If networking works, you're done. If not, troubleshoot.

Troubleshooting

Start by seeing how much or how little network access you have. Assuming you're configured for the 192.168.100.0/24 subnet, try pinging a runninc computer on that subnet. If any of the pings succeed, you know at least your Network card works, and is connected to your LAN.

Next, ping known good Internet address 8.8.8.8. If you succeed, you know you're getting out through your default route to the Internet, and any problems are mere DNS. If the ping to 8.8.8.8 fails, make sure your computer's default route is set to the LAN address of your router. If they are the same, take a look at your router/firewall.

If you can ping 8.8.8.8, but can't ping various Internet URLs such as www.troubleshooters.com, you have a probable DNS failure. Look at /etc/resolv.conf. Back it up, and then replace it with the following pointer to Google's public DNS servers:

nameserver 8.8.8.8 nameserver 8.8.4.4

Once shellscripts /root/udev_start.sh and /root/upnet.sh produce a completely working network, the next step is to make sure they run on boot. Do this by making them Epoch services...

Yes, I Know Yes, I know that this setup guide hard coded the IP address, completely missing both wired DHCP and Roaming Wifi use cases. The former is accomplished with the dhcpd daemon, the latter by a dbus/NetworkManager combo, or preferably a little hand scripting of the wpa_supplicant and iwlist programs. I leave this as an exercise to the reader, because if I have any extra time, I'll devote it to getting nosh or s6 running, or replacing the native init on CentOS or Fedora.

Adding the Udev and Network Services

# RUN simple shellscript to get udev running # Udev is necessary to start network # Run it before remounting / read-write. # Don't bother to stop it at all ObjectID=udevd ObjectDescription=Starting udev ObjectStartCommand=/root/udev_start.sh ObjectStopCommand=NONE ObjectStartPriority=20 ObjectStopPriority=0 ObjectEnabled=true ObjectOptions=RAWDESCRIPTION ObjectRunlevels=boot # RUN shellscript to set up network. Works on any distro having # the ip command. No config files necessary. # start it after Udev is up and / is read-write # and the early Getty is running ObjectID=network ObjectDescription=Setting up net devices and the network ObjectStartCommand=/root/upnet.sh ObjectStopCommand=NONE ObjectStartPriority=50 ObjectStopPriority=0 ObjectEnabled=true ObjectOptions=RAWDESCRIPTION ObjectRunlevels=boot

Reboot, and verify that networking runs as expected.

Make Epoch Your Default Init

Up to this point, your Epoch Init System driven init has been under heavy construction, with a high probability of failure. For this reason, it was set up to init only if you added init=/e to the kernel line in Grub. But of course, time after time, tweaking the Grub kernel line becomes time consuming. So at this point, Epoch is probably stable enough on your system to make it your default. You've already copied your compiled epoch executable to /usr/bin/epoch, now copy it (owner/group root, executable by all) to /usr/bin/init or whatever file your kernel runs after setting itself up.

If you're having trouble determining what program the kernel runs as PID1, you're not alone. The ps command shows "init" regardless of the filename of the program run as PID1. Perhaps the best way of determining this is either looking in your Grub's config, or just trial and error, starting with /usr/bin/init or /sbin/int, depending on whether or not, respectively, your /sbin is just a symlink.

Add in /dev/mixer Capability

The sound card drivers are already loaded, probably by Udev. However, neither ALSA nor Pulseaudio is running, at last on my Manjaro box. And I like it that way! If your results are like mine, there is no /dev/mixer device. This makes volume control problematic. The step you're on now changes that.

Before anything else, use the package manager to install a text mode mixer such as rexima. Don't install Alsamixer because that depends on ALSA, and at this point you want to keep dependencies to a minimum. When you've installed your text mode sound mixer, run it. It will probably fail something like the following:

[root@openrc ~]# rexima rexima: couldn't open mixer device. [root@openrc ~]# ls -ldF /dev/mixer ls: cannot access /dev/mixer: No such file or directory [root@openrc ~]#

The preceding pretty much spells it out. You need /dev/mixer. And here's how you get it:

[root@openrc ~]# modprobe snd_mixer_oss [root@openrc ~]# ls -ldF /dev/mixer crw-rw---- 1 root audio 14, 0 29.12.2014 01:12 /dev/mixer [root@openrc ~]#

When you run rexima again, it gives you the adjustments you expect. Now all that remains is to have this happen on boot. You want it to happen after the early Getty and after network instantiation, so if something goes wrong you can troubleshoot from the console or from ssh. And you want it to happen before any daemons. So add the following to your /etc/epoch/epoch.conf

# LOAD kernel module for /dev/mi