In the olden days Linux administrators had a static /dev directory. It was inflexible and obese, containing 99% irrelevant entries, and we liked it that way. We didn't mind hassling with makedev and struggling with major and minor numbers to enter the devices we actually wanted, or manually deleting the 1,000 useless /dev entries, because Real System Administrators love doing things the hard way. It makes us feel close to our hardware. The best part of the job is spending years acquiring and hoarding arcane bits of knowledge, which are then passed on to eager, fresh-faced noobs with the magical incantation, "RTFM, luser."

Then came devfs, which attempted to replace this increasingly arthritic system with something that was less complex, more efficient, and which was a bit more based in reality. The creator of devfs, Richard Gooch, had this radical idea that the /dev directory should contain only devices actually present on the system, and aim for better performance and simplified device management.

Then descended the unbearded prophet Greg Kroah-Hartman from his mountaintop lair with yet another dev filesystem, called thereforth udev. Since kernel 2.6.13, devfs is no longer included in the mainline kernel.

Endless debates rage about devfs vs. udev, which you can read about in Resources. Chances are you still have /etc/devfs/ on your system, and a few devices that depend on it. A famous example is NVidia drivers. So when you see this directory, don't think you can go on a mad housecleaning spree and get rid of it.

udev handles the task of detecting hardware and creating nodes for it in /dev, and also managing device permissions. It works in concert with the Linux Hardware Abstraction Layer (HAL) and the hotplug subsystem. In effect, all devices, even internal drives and expansion cards, are treated as removable hotplug devices. "Oh no," you say, "this is not good, because removable devices receive different kernel names every time you plug them in."

No problem, for udev lets you create fixed device names so you can make static entries in /etc/fstab, and don't have to play hunt-the-widget every time you reboot. In fact your Linux distribution probably comes with a nice pre-fab configuration that assigns static names to certain devices, like hard drives and PCI network cards.

It's a nice flexible, highly adaptable system, and when it's configured correctly by your distribution maintainer your computing life is easy and fun.

I'm Sorry, Dave, I Can't Allow You to Do That

The downside to all this udev goodness is it's still just a youth, so when you need to make some manual tweaks you have to figure out weirdo command syntax and how to uncover the device information you need. It's simple when you know how. Well, maybe not even then. But let's take a look under the hood anyway.

We still have a /dev directory. This is no longer a static directory, but rather is populated at boot with entries generated by udev rules.

udev's rules are stored in configuration files in /etc/udev. Different distributions mangle this in different ways. Fedora and Ubuntu are sensible. There is /etc/udev/udev.conf, which contains program options, then all rules files are kept in /etc/udev/rules.d/ like they're supposed to. Debian Etch, for gosh-knows-why, puts all the rules files in the top-level directory, and then has to symlink them all to /etc/udev/rules.d. udev rules must go in /etc/udev/rules.d.

The /sys directory is a cousin to /proc, only it's well-organized rather than a chaotic mess. It exports kernel information into a human-browseable and program-parseable structure. Just like /proc uses the /proc filesystem, the /sys directory uses sysfs. You can see this with the mount command: # mount proc on /proc type proc (rw) /sys on /sys type sysfs (rw) udev on /dev type tmpfs (rw)

Notice how /dev uses tmpfs, which means wipeout on reboot.

/sys is chock full o' symlinks, as you can see with the find /sys -type l command. You can browse /sys just like any other directory. Whether you'll understand the contents is another question, but it doesn't hurt to get familiar with its structure.