>: Measuring Linux at Runtime

Linux needs better open source intrusion detection software.

Beyond the primarily 1990s-style static detection mechanisms a la rkhunter, chkrootkit, ossec and even samhain. And beyond even memory forensic tool volatility, which is solid and always getting better at detecting sophisticated stuff. This post isn’t intended to diss anybody or their software; on the contrary, most of what these and other programs do is well executed. But it’s absolutely not enough for even the most common threats found today, let alone ayy pee tee. Ha.

Whereas offline memory analysis can be dynamic, reliable and sophisticated, capturing it live for regular inspection risks system destablization, takes lots of time, is error-prone, and worst of all, presupposes that you know which system is in need of analysis before the analysis begins. Most open source tools available for this aren’t meant to detect intrusion, they’re meant to explain it.

Raytheon’s Second Look is an agent-based system that can perform on-demand host scans. It has a solid featureset and can apparently do live analysis of partial memory footprints. But it’s commercial and I’m not sure if it’s opensource. I haven’t tested it out personally. I’d love to give it a fair public review if I can get my hands on a demo, non-commercial copy! :)

Given the current state of linux IDS software, and my interest in system security, I sat down one day and wrote a tool that detects some forms of kernel tampering. I wrote it last year, merged it into the OSQuery repo, and wrote a vtable for it. The code has since been relocated here. I’ve finally gotten around to write up this little post. Hopefully it’s of use to some folks, even if only as a learning tool.

What

camb (pronounced kam’bee) is a simple POC kernel module that exposes to user space the current SHA1 of the running kernel. Specifically, it hashes the .text segment, the executable code portion of the kernel. It doesn’t include any data segments, or what is sometimes referred to as non-control data. (This same functionality could be built into the kernel into an LSM proper. Although that’s a less flexible option, and one that’s harder to just try out, it’s preferable. Being built into the upstream kernel is great because you get forward compatibility, you don’t have to manage modules yourself, and LSMs afford a lot of power to the detection software.)

The idea is to more easily determine whether the kernel has been modified, hijacked to do things like provide a persistent backdoor and hide files & network traffic. This is a runtime assessment, only good for the moment the measurement is taken.

TPM-based attestation mechanisms like secure boot assert varying degrees of confidence of boottime integrity. Runtime is a whole other matter. Boottime artifacts are static, the execution sequence known. Runtime control and non-control data alike can be shifty and less than deterministic. So to the extent that boottime is a solved problem, I naturally find the next piece of the integrity puzzle to solve is the runtime bits.

Try it out

Install kernel headers (on ubuntu: $sudo apt-get install linux-kernel-headers ) $git clone https://github.com/unixist/camb.git Build camb:

unixist@ubuntu:~/code/camb >: sudo make SMAP = /boot/System.map-3.16.0-36-generic [ sudo ] password for unixist: make -C /lib/modules/3.16.0-36-generic/build M = /home/unixist/code/camb EXTRA_CFLAGS = "-DTEXT_SEGMENT_START=" 0xffffffff810001c8 " -DTEXT_SEGMENT_END=" 0xffffffff81772444 " -DSYSCALL_BASE_ADDR=" 0xffffffff81801460 "" modules make [ 1 ] : Entering directory ` /usr/src/linux-headers-3.16.0-36-generic ' CC [M] /home/unixist/code/camb/main.o CC [M] /home/unixist/code/camb/sysfs.o CC [M] /home/unixist/code/camb/hash.o LD [M] /home/unixist/code/camb/camb.o Building modules, stage 2. MODPOST 1 modules CC /home/unixist/code/camb/camb.mod.o LD [M] /home/unixist/code/camb/camb.ko make[1]: Leaving directory `/usr/src/linux-headers-3.16.0-36-generic' unixist@ubuntu:~/code/camb >: Install: $sudo insmod camb.ko

See what’s cookin

See the hash of the currently running kernel

unixist@ubuntu:~/code >: cat /sys/kernel/camb/text_segment_hash 86b8e609d011122f449839105255a481357e493d unixist@ubuntu:~/code/osquery >:

Or with OSQuery

unixist@ubuntu:~/code >: osqueryi osquery - being built, with love, at Facebook ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Using a virtual database. Need help , type '.help' osquery> select * from kernel_integrity; +----------------------+------------------------------------------+ | sycall_addr_modified | text_segment_hash | +----------------------+------------------------------------------+ | 0 | 86b8e609d011122f449839105255a481357e493d | +----------------------+------------------------------------------+ osquery> unixist@ubuntu:~ >:

See it in action

Load camb Verify hash Load suterusu rootkit Verify hash changed

unixist@ubuntu:~/code/camb >: sudo insmod camb.ko unixist@ubuntu:~/code/camb >: osqueryi 'select * from kernel_integrity' +----------------------+------------------------------------------+ | sycall_addr_modified | text_segment_hash | +----------------------+------------------------------------------+ | 0 | 430d78d4777628185f47be3ef00a16547e261e18 | +----------------------+------------------------------------------+ unixist@ubuntu:~/code/camb >: sudo insmod ../suterusu/suterusu.ko unixist@ubuntu:~/code/camb >: osqueryi 'select * from kernel_integrity' +----------------------+------------------------------------------+ | sycall_addr_modified | text_segment_hash | +----------------------+------------------------------------------+ | 0 | a13608340e76ca69c459ef8244fafc8798f1370d | +----------------------+------------------------------------------+ unixist@ubuntu:~/code/camb >:

And of course the rootkit performs minor housekeeping to help limit its visibility. You wont find any decent rootkit with lsmod or by looking in debugfs.

unixist@ubuntu:~/code/blog >: sudo lsmod | grep suterusu unixist@ubuntu:~/code/blog >:

Nothing.

Ways to bypass

camb aint perfect by a long shot. If a system is compromised to the point where kernel memory can be tampered with, there is the possibility of thwarting camb’s measurement. This is the quintessential problem with host-based intrusion detection: you’re asking a brainwashed person whether he’s thinking clearly. Anyway, here are a few ways I would defeat this type of detection:

Load your own module and hook file system access routes to maintain a known value in /sys/kernel/camb/text_segment_hash . I’m not sure of the particulars and how it would work with the “file” being found on sysfs, but I’m confident that hooking is possible.

Block the logs from osquery/syslog/whatever from being sent off of the compromised box to syslog/SIEM. Just let camb and osquery do their thing. Who cares if you can prevent or filter the logs from being centrally analyzed? * Stop or kill the logging processes * Network ACL (e.g. iptables) off the logging process for the duration of the backdoor’s life * Temporarily blackhole or redirect the network route to the syslog or SIEM to an attacker controlled host

Use your imagination

If you disable module loads or require modules to be signed, you can prevent easy bypass. You may even detect changes to the kernel that happen by way of a kernel vuln yielding some write primitive, which the .text measurement might catch before full system ownage. It’d be lucky, but possible.

At the end of the day, something akin to camb’s mechanism strikes me as a must-have. It’s like AV: it’s limited and subject to defeat by skilled (?) attackers, but silly not to employ. At the very least you’ll catch kids with powerful kits who don’t know how to customize them or execute bypasses.