A few weeks back I was analyzing a Solaris 10 (SPARC) raw partition image and was trying to determine from the wtmpx files who had logged into the system, from what/which remote IP addresses and when. To be more precise, I was tracking nagios account that was used to compromise this machine. The problem I encountered was that the file system was completely wiped out - all files were gone.

Fortunately, this was done at filesystem level with rm -rf / command.

1 2 3 4 5 6 7 8 9 10 11 12 [email protected] :~$ sudo mount -o loop,ro -t ufs dd_nj090240-var /mnt [email protected] :~$ ll -R /mnt /mnt: total 6 drwxr-xr-x 3 root sys 1024 jul 1 02:43 ./ drwxr-xr-x 30 root root 4096 aug 4 09:49 ../ drwxr-xr-x 2 root sys 512 jun 28 13:25 run/ /mnt/run: total 2 drwxr-xr-x 2 root sys 512 jun 28 13:25 ./ drwxr-xr-x 3 root sys 1024 jul 1 02:43 ../

This means the data should still be there. But how to recover it?

Solaris wtmpx file format

Solaris uses /var/adm/wtmpx file which is in some way similar to /var/log/wtmp from Linux but unfortunately is incompatible. Also this system is based on SPARC architecture which is big-endian so in contrast to Intel x86 (little-endian) the integers are stored in reverse order. This means we cannot use Linux native tools like last to parse contents of a wtmpx file from Solaris. In order to recover it we need to know the exact structure. The easiest way to understand the format is to look at the source code of programs that read and write to wtmpx files. Since the target system is Solaris, the format is very likely to be found in /usr/include/utmpx.h C include file.

Here is an excerpt from Solaris 10’s utmpx.h:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 struct timeval32 { int tv_sec, tv_usec; }; struct futmpx { char ut_user[32]; /* user login name */ char ut_id[4]; /* inittab id */ char ut_line[32]; /* device name (console, lnxx) */ pid32_t ut_pid; /* process id */ int16_t ut_type; /* type of entry */ struct { int16_t e_termination; /* process termination status */ int16_t e_exit; /* process exit status */ } ut_exit; /* exit status of a process */ struct timeval32 ut_tv; /* time entry was made */ int32_t ut_session; /* session ID, user for windowing */ int32_t pad[5]; /* reserved for future use */ int16_t ut_syslen; /* significant length of ut_host */ char ut_host[257]; /* remote host name */ };

Data carving

Each wtmpx entry is exactly 372-byte long (aligned to 4 bytes!) and it starts with an username trimmed to 32 bytes. Based on this information we can create a pattern for scalpel - well known file carving utility. In this case, we want scalpel to scan for specified string of bytes (header) and then save 372 byte long chunks of data that follow the header. If you want to learn more about the configuration file syntax, I encourage you to review the manual page or the configuration file itself where you will find many examples.

1 wtmpx y 372 nagios\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x74\x73

Let’s run it on the partition image and see the results!

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 [email protected] :~$ scalpel -o scalpel_out/ -O dd_nj090240-var Scalpel version 1.60 Written by Golden G. Richard III, based on Foremost 0.69. Opening target "/home/elceef/dd_nj090240-var" Image file pass 1/2. dd_nj090240-var: 100.0% |***************************************************************************************************| 20.0 GB 00:00 ETA Allocating work queues... Work queues allocation complete. Building carve lists... Carve lists built. Workload: wtmpx with header "\x6e\x61\x67\x69\x6f\x73\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x74\x73" and footer "" --> 8 files Carving files from image. Image file pass 2/2. dd_nj090240-var: 100.0% |***************************************************************************************************| 20.0 GB 00:00 ETA Processing of image file complete. Cleaning up... Done. Scalpel is done, files carved = 8, elapsed = 67 seconds.

After a minute the tool carved eight files out of the image.

1 2 3 4 5 6 7 8 9 10 11 12 13 [email protected] :~/scalpel_out$ ll total 44 drwxrwxr-x 2 elceef elceef 4096 aug 16 05:39 ./ drwxrwxr-x 4 elceef elceef 4096 aug 16 05:36 ../ -rw-rw-r-- 1 elceef elceef 372 aug 16 05:39 00000000.wtmpx -rw-rw-r-- 1 elceef elceef 372 aug 16 05:39 00000001.wtmpx -rw-rw-r-- 1 elceef elceef 372 aug 16 05:39 00000002.wtmpx -rw-rw-r-- 1 elceef elceef 372 aug 16 05:39 00000003.wtmpx -rw-rw-r-- 1 elceef elceef 372 aug 16 05:39 00000004.wtmpx -rw-rw-r-- 1 elceef elceef 372 aug 16 05:39 00000005.wtmpx -rw-rw-r-- 1 elceef elceef 372 aug 16 05:39 00000006.wtmpx -rw-rw-r-- 1 elceef elceef 372 aug 16 05:39 00000007.wtmpx -rw-rw-r-- 1 elceef elceef 963 aug 16 05:39 audit.txt

1 2 3 4 5 6 7 8 9 10 11 12 [email protected] :~/scalpel_out$ hexdump -C 00000000.wtmpx 00000000 6e 61 67 69 6f 73 00 00 00 00 00 00 00 00 00 00 |nagios..........| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000020 74 73 2f 31 70 74 73 2f 31 00 00 00 00 00 00 00 |ts/1pts/1.......| 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000040 00 00 00 00 00 01 b9 00 00 07 00 00 00 00 00 00 |................| 00000050 55 95 ac bb 00 0a d1 5d 00 00 00 00 00 00 00 00 |U......]........| 00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000070 00 0e 31 30 2e 32 30 30 2e 31 32 35 2e 31 34 32 |..10.200.125.142| 00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000174

The entries look valid. We can easily spot account name, console and source IP address this session originated from. We miss other important piece of the puzzle: timestamp and event type. We need to write a parser that will allow us to extract detailed information of each event (entry) similar to how last command does.

Parsing

I created a quick and dirty python script that benefits mostly from struct module to handle binary data. This module has a function called unpack() especially designed to parse binary and structured data according to a given format. Format strings are used to specify the expected layout when unpacking data. They are build up from format characters which specify the type and size of data being unpacked. I strongly encourage you to review documention for struct module first in order to understand better the meaning of format characters.

It is worth mentioning that I had to use pad bytes in the format string in order to maintain proper alignment for the futmpx struct involved. Don’t be surprised if your calculations are not in accordance with sizeof(struct futmpx) - this is the way data structures are stored in the memory.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 #!/usr/bin/env python import struct import sys import datetime def type(x): return { 0: 'EMPTY', 1: 'RUN_LVL', 2: 'BOOT_TIME', 3: 'NEW_TIME', 4: 'OLD_TIME', 5: 'INIT_PROCESS', 6: 'LOGIN_PROCESS', 7: 'USER_PROCESS', 8: 'DEAD_PROCESS', 9: 'ACCOUNTING' }.get(x, 'UNKNOWN') data = open(sys.argv[1], 'rb') while True: chunk = data.read(372) if not chunk: break s = struct.Struct('>32s 4s 32s i H H H b b I I I 5I H 257s b') unpacked = s.unpack(chunk) #TODO: timezone timestamp = datetime.datetime.fromtimestamp(int(unpacked[9])).strftime('%Y-%m-%d %H:%M:%S') print(str(unpacked[0]) + '\t' + str(unpacked[3]) + '\t' + str(unpacked[2]) + '\t' + str(timestamp) + '\t' + str(unpacked[18]) + '\t' + type(unpacked[4]))

Now it’s time to see this code in action. My script takes only a single file as an argument so I use the following command line kung-fu to parse all files (in this case single wtmpx entries) at once and sort by the timestamp:

1 2 3 4 5 6 7 8 9 [email protected] :~$ for i in $(ls *.wtmpx); do readutmpx.py $i; done | tr -d '\000' | sort -k 4 nagios 257138 pts/1 2015-02-14 04:12:35 10.212.6.160 USER_PROCESS nagios 257138 pts/1 2015-02-14 04:12:35 10.212.6.160 USER_PROCESS nagios 257138 pts/1 2015-02-14 04:15:35 DEAD_PROCESS nagios 257138 pts/1 2015-02-14 04:15:35 DEAD_PROCESS nagios 112896 pts/1 2015-07-02 17:27:23 10.200.125.142 USER_PROCESS nagios 112896 pts/1 2015-07-02 17:27:23 10.200.125.142 USER_PROCESS nagios 112896 pts/1 2015-07-02 17:32:32 DEAD_PROCESS nagios 112896 pts/1 2015-07-02 17:32:32 DEAD_PROCESS

Works like a charm! But there is still area for improvement. This code does not convert the time to the correct time zone. Take this into account before building a timeline.

Happy end

Solving this case would not be possible without this promising technique. The compromised system was configured to keep track of only unsuccessful authentication attempts leaving wtmpx records as the only reliable source of information about the origin of the attack. The person responsible for the destruction of this system was too confident - deleting all files is not enough to cover all tracks. Now personal details of this individual are known and the case is closed. Cheers!