Tales of a Stressed Kernel

Spending most of our time (as developers) in the high-level world, it's easy to occasionally forget the true nature of our systems and how fragile they really are. Well, perhaps fragile is not the right word here, more like intricate or perhaps chaotic - you cannot fully predict the effect of one operation on the rest of the system. In this case, a simple file copy got our machines to freeze for many long seconds... Huh?

Our machines are given loads of RAM but no swap space, to ensure deterministic memory access times. The memory is pre-allocated to different processes based on a configuration file, with a small portion reserved for the kernel. Memory is allocated from /dev/shm (mounted as tmpfs), where it's also exposed as files.

When a process crashes we may need some of those /dev/shm files for debugging, so we have a tool that runs whenever there's a crash to collect system info. Among other things, it used to copy some of these shared-memory files to disk (using a simple cp or shutil.copy() ). No surprise there. But every once in a while, and strongly-correlated to times when this collector was running, some time-critical processes timed-out when writing to the disks for no apparent reason, leading to catastrophic results. We've spent about a month trying to pin-point what's taking so many system resources. Many things came to mind: CPU consumption? Accessing special /proc files that got the kernel busy? Running SCSI or other hardware commands? Jamming the IO bus with data from the copy?

The mystery was solved just last Thursday, when I realized copying files from /dev/shm caused all sorts of system-wide hiccups -- not only timeouts when writing to other disks. It seemed that existing processes did get runtime, but no new processes could be forked. Sometimes running ls took ~20 seconds. Other times simple (non-file system) tools like date hanged for a while. When it was apparent it's system-wide, the explanation was quite obvious: the kernel ran out of memory. Since there's no swap space, there's nothing it could do and kernel threads just blocked until there was enough room for their allocations.

But why would the kernel get so low on memory? After all, we're copying files from memory (tmpfs) to the disk. Well, that's seems like a bug:

I think I've finally figured this out. It's a kernel bug -- I'm guessing that under normal circumstances, the "cached" column in the free command "doesn't count" towards how much memory the system thinks it's using. After all, it's just cached copies of stuff that should be elsewhere, and if you run out of memory, you can safely dump that, right? Unfortunately, /dev/shm is counted under cached rather than used memory (as I discovered in an earlier post). https://bbs.archlinux.org/viewtopic.php?pid=390313#p390313

Simplistically, file copy is a simple loop that reads a chunk of data from the source file and writes it to the destination file, until it transfers everything. When we read from a file, the kernel needs to allocate a kernel-space buffer and copy it to userland. And when we write it back to the destination file, the kernel first copies the userland buffer into kernel-space and links is to the device's queue (to be evicted at the driver's discretion). The write() call returns as soon as the kernel places the buffer into the queue, so it might "pile up" there for some time before actually being evicted, depleting kernel memory.

Ugggh. A simple copy brought our system to a halt. The solution was just as simple -- we don't want (and neither do we need) to use kernel buffers here. The source file already resides in memory. Instead of read() ing it, we can just mmap() the whole of it. And as for the destination file, we open it with O_DIRECT, so as not to use kernel buffers along the way. I christened this new tool mmapcp .

The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances. -- Linus

Well, Linus, at least you were kind enough to let it stay :)