>: Detecting hidden files

After system compromise, the attacker’s goal is almost invariably to maintain access for a given period of time. This is usually true even if the attacker only intends to exfiltrate data or perform some form of sabotage. In order to operate undetected for as long as possible, the attacker must limit anomalous behavior: this includes stealthy network communication, judicious use of system resources, hiding any disk presence, and more. I’m herein concerned with detecting an attacker’s disk presence, with a focus on hidden files on an otherwise normal and accessible filesystem. I don’t consider protected or regularly inaccessible portions of storage devices.

Rootkits come in many forms to accomplish the above. Some function as binary replacements for system utilities such as ls , ps , netstat , lsof , df , etc. Some function as userspace in-memory patches of services such as sshd or apache. Still others live in the kernel. Kernel kits are not necessarily the most interesting to an attacker, but are by far the most powerful as operations are carried out with ring 0 privileges. While the replacement of system utilities cannot fool file integrity checkers and in-memory binary patching is limited by process scope and userspace power, the kernel is near-limitless in its stealth abilities.

What

inodeyou is a tool that detects some forms of kernel rootkit file hiding. It detects when certain linux system calls are hooked, replaced by modified versions that shield files from view. It evaluates the disk’s view of the filesystem (via read(2)) and the user’s view of the mounted filesystem (via getdents(2), stat(2), etc.) by comparing the inodes present from both points of view.

I have tested with private kernel modules that hook system calls to hide files, but you can test on kits like Adore or Suckit.

Why

The main reason it’s necessary to take a look at the raw storage device instead of using tools like ls , or find is due to abstractions. The filesystem placed on top of a storage device or volume allows you to easily reference files and directories and hierarchies on it. Reading a filesystem only requires that you speak ls to list directories; reading a raw device requires you to speak filesystem, e.g. EXT2/3/4, FAT, NTFS.

The filesystem gives a user friendly view to the device by way of system calls like read(2) to read file contents, stat(2) to read file metadata, getdents(2) to read directory contents/entries. If certain system calls are hooked, you may receive an incomplete list of files that reside in a directory when you ls it. But if you skip right down to open(2)ing and read(2)ing the storage device, then you access the bytes directly and can interpret them as fs data structures. This is a more difficult problem for the rootkit writer to defend against.

Another reason to employ a file detection technique of this sort is that up-to-date rkhunter and chkrootkit fail to find these files.

Notes

inodeyou requires raw disk access and also read access to the interesting volume mounted somewhere. As such, inodeyou isn’t intended for offline forensics.

requires raw disk access and also read access to the interesting volume mounted somewhere. As such, isn’t intended for offline forensics. inodeyou will yield false positives on file systems with even semi-high file creation or deletion frequency. I'll soon try to parallelize the scan to speed this up. I think a decent approach is to not consider any files created after the scan initiates. I’m open to suggestions on this issue.

will yield false positives on file systems with even semi-high file creation or deletion frequency. I think a decent approach is to not consider any files created after the scan initiates. I’m open to suggestions on this issue. Raw file system support is based upon whatever is supported by The Sleuth Kit on your system. I don’t think EXT4 is officially supported yet :(. inodeyou is only tested on ext2 and ext3.

is only tested on ext2 and ext3. inodeyou only catches rootkits that hook getdents, stat, and other file or directory-related system calls. It is possible to hook system calls read(2) and mmap(2) to hide files, but it is much more difficult to achieve reliably. I haven’t seen this type of behavior in the wild so I plan to prove it out soonish.

only catches rootkits that hook getdents, stat, and other file or directory-related system calls. It is possible to hook system calls read(2) and mmap(2) to hide files, but it is much more difficult to achieve reliably. I haven’t seen this type of behavior in the wild so I plan to prove it out soonish. It currently only works on regular files and directoies. I haven’t yet looked into whether it must be modified to work with sockets, FIFOs, character/block devices, etc.

Gotta run as a suitably privileged user, probably root.

I’ve only tested this on linux, though the same method should apply on other unixes.

Better documentation and examples coming soon

Dependencies

pytsk3. I hoped to write some code against a C API, but it turns out that a bulk of TSK’s codebase–including its API–is C++. So I quickly about-faced and looked for TSK bindings in python :).

Examples

If /dev/sda1 is the volume that you mount at /boot: $ inodeyou.py /dev/sda1 /boot

If your /var is a separate volume and you only want to scan /var/log: $ inodeyou.py /dev/sdb3 /var/log /log

See below example session: First run against /dev/sda1 mounted at /boot, no hidden inodes are found. After the kit is installed we run again and see two inodes hidden. Only one is found when the /boot/grub subdirectory is searched. With the TSK tools icat , fls , and ffind we can see the file name, contents, and path associated with the hidden inodes. * root@debtop:/home/unixist/code/rk_detect# ./inodeyou.py /dev/sda1 /boot root@debtop:/home/unixist/code/rk_detect# insmod sneaky.ko root@debtop:/home/unixist/code/rk_detect# ./inodeyou.py /dev/sda1 /boot 40386 84339 root@debtop:/home/unixist/code/rk_detect# ./inodeyou.py /dev/sda1 /boot/grub /grub 40386 root@debtop:/home/unixist/code/rk_detect# icat /dev/sda1 40386 ohnoes root@debtop:/home/unixist/code/rk_detect# fls /dev/sda1 40385 r/r 40386: secrets root@debtop:/home/unixist/code/rk_detect# root@debtop:/home/unixist/code/rk_detect# ffind /dev/sda1 40386 /boot/grub/asdfasdf/secrets root@debtop:/home/unixist/code/rk_detect#

Tool

Project: inodeyou

Repo: git clone https://bitbucket.org/unixist/inodeyou.git

Catch me on twitter if you’d like to chat about other ways to hide files and try to detect that they’re hidden.