Fedora and Ubuntu Kernel Config Comparison [May. 9th, 2013|02:29 pm] jwboyer

Every once in a while, I crawl out from under the rock that is bugzilla and I try and look around at what others are doing in the distro kernel space. Today I was curious how Fedora and Ubuntu compare in how they configure the kernel. I've long thought that for all the focus the kernel gets, it should be the most boring package in an entire distro. It should work, work well, and that is about it. It isn't there to differentiate your distro. It's there to let your distro run. So, will my personal belief stand up, or would I find something in the configs that proves one "distro" is better than another? Let's dive in.



I won't give a line by line comparison because 1) that would be very long and 2) that would be utterly boring. You don't need to know that e.g. Ubuntu enables one scsi driver that Fedora doesn't or vice versa. Instead, I'll try to focus on various functional areas and give an overview.



The kernels

For comparison purposes, I used the config from linux-image-3.8.0-19-generic Ubuntu 13.04 package and the 3.8.11-200.fc18 kernel from Fedora. These are the same base upstream kernel version, and both of them are from the latest stable release of their respective distros. I used the x86_64 (amd64 in Ubuntu speak) build, as I found that to be the most relevant in this day and age. I could have used the kernels under development, however those are more apt to change and I thought it would be better to compare things out there in wider use.



Overview

The vast majority of the config is pretty much identical. It's what you would expect from a distribution config. There is a wide range of support for all kinds of devices and buses, with much of it being provided by modules instead of built in. This should be expected, given that they're coming from the same upstream code base and really there is only one sane option for most config options.



Low-level options

A few of the low-level options did differ though. One of the more notable ones is that Ubuntu has CONFIG_NR_CPUS set to 256, whereas Fedora has it set to 128. On the face of it, one might think both values are overkill as people aren't booting on machines with 128 cores very often. However, Ubuntu has transitioned to having a single kernel between it's desktop and server variants (starting with 12.04 I think), so it does make some sense for them to have it set higher. Fedora can revisit our choice if we get a sudden clamoring for massive machine support. I await my mini-supercomputer test system.



Another area where we differed was in the NUMA options. Ubuntu enables CONFIG_X86_NUMACHIP, CONFIG_MOVABLE_NODE, CONFIG_MEMORY_HOTPLUG, and CONFIG_MEMORY_HOT_REMOVE. The CONFIG_NODES_SHIFT value was higher on Ubuntu as well, allowing for more NUMA nodes. Again, given their server variant that probably makes sense for them. Fedora just doesn't deal with that class of machine on a regular basis.



The HZ area also had some differences. Fedora defaults to HZ_1000 compared to Ubuntu's HZ_250. That means the timer tick is 1000 times per second on Fedora, versus 250 on Ubuntu. Typically a higher HZ value is used for better interactive response. On a server system where there isn't typically a desktop or user waiting for interaction, a smaller HZ setting allows more work to get done as there are few timer interrupts to process. Of course, both distros set CONFIG_NO_HZ, which means the timer interrupts are only triggered on an as-needed basis when the CPU is idle. I would expect that both will eventually also go with the new completely tickless options that were introduced in the 3.10 kernel merge window after they have matured a bit.



Ubuntu also enabled CONFIG_FAST_NO_HZ whereas Fedora doesn't. That option lets CPUs with pending RCU callbacks enter the dynamic tick state (idle) by extending the grace period. That slows down synchronize_rcu calls, but by letting the CPUs enter idle state more it improves energy efficiency. Sounds like a great thing, and it really probably is. Fedora disabled this after having it enabled when it was first introduced and hitting numerous issues with it. The option has been upstream for a few kernel releases now, so it's probably something we should revisit.



Default choices

For the most part, the default choices were the same. A couple of the areas where they diverged were in the default I/O scheduler, and the default CPU frequency governor. Ubuntu uses the deadline scheduler by default compared to Fedora's CFQ (Completely Fair Queueing). Ubuntu uses the performance governor instead of the ondemand governor that Fedora uses. To be clear, all of the options are provided by both distros. They just differ in the default setting.



LSM

The security stack is probably one of the larger areas of difference between Ubuntu and Fedora. Fedora is solidly an SELinux camp. It is the only LSM we enable and is therefore naturally the default. Ubuntu on the other hand enables all of them, including SELinux. I was kind of surprised by this, as I expected they would only enable their default LSM which is AppArmor. I wouldn't argue that this makes Ubuntu more secure than Fedora by any means, but it does give users a greater choice in what they use for MAC/DAC security.



Module options

Fedora and Ubuntu both produce signed modules. I was surprised Ubuntu was using this given their statement of it not being needed for Secure Boot purposes. So perhaps they're using it for some other purpose. Looking at their choice of using CONFIG_MODULE_SIG_SHA512 that might well be the case. Fedora uses CONFIG_MODULE_SIG_SHA256 because that is the common hash implementation used in UEFI.



Ubuntu also enables CONFIG_MODVERSIONS. This generates a CRC value for each exported symbol prototype. When a module is loaded, it compares the CRC with that in the kernel and fails the load if they do not match. It is a simple form of ABI check for modules. They also enable MODULE_SRCVERSION_ALL, which embeds a "srcversion" field. Arguably, this lets people build modules against different kernels and then load them while knowing where the module came from.



The old, the obscure, and the crappy

Ubuntu seems to have a tendency to enable a lot more old stuff than Fedora. They have support for acorn, atari, ultrix, and sysv68 partitions. They build DECNET and Parallel IDE support. ARCNET. Things one would not normally expect to find, much less want to use in this day and age. Of course, Fedora has some weird stuff too. We still enable the OSS sound stack and Ubuntu appears to have ditched this entirely.



They also seem to have a lot of "embedded" focused options enabled. Things like full MTD support, all of the random keyboard drivers, SPI, MFD, Regulator, and GPIOLIB. They have JFFS2, F2FS, and VXFS filesystem support. They enable various SoC options for things like sound and media drivers. This seems a bit at odds with their desktop and server focus, but if one wanted to boot an Ubuntu kernel on some random x86 based SoC thing they might be able to.



Fedora also tends to only enable the main DRM drivers that support KMS and very few framebuffer drivers. Ubuntu enables everything almost. Personally, I'm fine with our KMS or go away approach. Oddly, Ubuntu doesn't enable CONFIG_LOGO. WHY DO YOU HATE THE PENGUINS?



Lastly, Ubuntu enables way more staging drivers than Fedora. If you aren't aware, staging is where drivers come in to get cleaned up before being officially mainlined. They literally taint the kernel as TAINT_CRAP, and we tend to stay away from them. Ubuntu is also carrying AUFS, the union fileystem and ALX, an ethernet driver for new Atheros chips that has had a pretty horrible review history upstream.



Summary

So did my personal theory of "kernels should work and that's it" hold? I think it did mostly. There are no shocking differences between the two distribution kernels. The few areas where we differ that might matter are worth reviewing on both sides, but at the end of the day none of those options are going to make any kind of massive improvement. I consider this to be a worthwhile exercise, but one I won't repeat for a while. I can only stare at kernel configs for so long before it makes even bugzilla start looking interesting.



If you've stuck with me thus far, thanks!