Contributed by tbert on 2015-01-21 from the Fuzz Aldrin dept.

I wanted to test the afl fuzzer that sort of recently entered the ports collection, ever since this webpage talked about how they give a jpeg decoder the string "Hello" in a file which it twists and mutates until the jpeg decoder no longer croaks on it, and it ends up actually being a valid jpeg image (though not very pretty).

Still, the general idea of instrumenting the code and then fuzz the inputs in such a way that it detects when a program starts taking new (perhaps less tested?) paths through the binary was new to me.

So, I dug in to how to set this up in an OpenBSD environment. First of all, whatever porting effort needed to make it run was already fixed, so "sudo pkg_add afl" takes care of that. Then you need to have a space to run the tests in, and since the fuzzer is going to create a huge amount of junk files to throw at your program, you really want this to be inside a tmpfs or mfs. This affects the speed a lot. It doesn't need to be very big, just fast in creating and deleting files.

Next step is to make a "in" and an "out" dir (for each program you intend to fuzz, in case you want to exercize a bunch of cores in parallel), and lastly to recompile your program with the appropriate afl-gcc wrapper.

In theory, this part could have been hard and cumbersome if the OpenBSD system makefiles were looking for stuff in odd places or if they generally didn't honor CC or whatever since the docs basically assumes everyone will run ./configure and have the flags trickle down into the generated makefiles, but I found it to be very easy on OpenBSD. Just copy the /usr/src/usr.bin/name_of_program directory next to the in and out dirs in your tmpfs and go:

prompt$ cp -r /usr/src/usr.bin/name_of_program . prompt$ cd name_of_program prompt$ CC=afl-gcc make

For many of the programs this will just work fine, since they are very self-contained in their directories. Some have several directories so you may need to copy a bunch of extra files (mopchk for instance, which is the mopd/ directory). In this example I chose cap_mkdb.

Prep a infile, not too big (so it can change each bit and each string and so on), in this case, I took a small part out of the main termcap (which normally is 21k lines long):

dumb|80-column dumb tty:\ :am:\ :co#80:\ :bl=^G:cr=^M:do=^J:sf=^J: unknown|unknown terminal type:\ :gn:tc=dumb: lpr|printer|line printer:\ :bs:hc:os:\ :co#132:li#66:\ :bl=^G:cr=^M:do=^J:ff=^L:le=^H:sf=^J: glasstty|classic glass tty interpreting ASCII control characters:\ :am:bs:\ :co#80:\ :bl=^G:cl=^L:cr=^M:do=^J:kd=^J:kl=^H:le=^H:nw=^M^J:ta=^I: vanilla|dumb tty:\ :bs:\ :bl=^G:cr=^M:do=^J:sf=^J:

then we start the fuzzer:

prompt$ nice afl-fuzz -i in -o out -f termcap -- cap_mkdb/cap_mkdb -f outfile @@

The two @ signs tells afl-fuzz to add the temporary name of the current fuzzed infile there, if you omit them afl-fuzz will feed input via stdin instead.

As it runs, it gives you a simple ascii info pane to look at, which tells you how it is doing. As per the documentation, it tells people to have some patience, which will be needed, since it can take a day or two to find all possible variants for that particular input, all depending on your CPU and I/O of course.

For me, it ended up like this:

american fuzzy lop 0.73b (cap_mkdb) ┌─ process timing ─────────────────────────────────────┬─ overall results ─────┐ │ run time : 2 days, 20 hrs, 28 min, 11 sec │ cycles done : 4 │ │ last new path : 1 days, 15 hrs, 54 min, 26 sec │ total paths : 82 │ │ last uniq crash : 2 days, 14 hrs, 21 min, 13 sec │ uniq crashes : 4 │ │ last uniq hang : 2 days, 19 hrs, 52 min, 53 sec │ uniq hangs : 8 │ ├─ cycle progress ────────────────────┬─ map coverage ─┴───────────────────────┤ │ now processing : 77 (93.90%) │ map density : 84 (0.26%) │ │ paths timed out : 0 (0.00%) │ count coverage : 5.58 bits/tuple │ ├─ stage progress ────────────────────┼─ findings in depth ────────────────────┤ │ now trying : havoc │ favored paths : 39 (47.56%) │ │ stage execs : 4983/7500 (66.44%) │ new edges on : 11 (13.41%) │ │ total execs : 13.0M │ total crashes : 166 (4 unique) │ │ exec speed : 52.63/sec (slow!) │ total hangs : 22.7k (8 unique) │ ├─ fuzzing strategy yields ───────────┴───────────────┬─ path geometry ────────┤ │ bit flips : 12/405k, 0/405k, 0/405k │ levels : 4 │ │ byte flips : 0/50.6k, 0/50.6k, 0/50.5k │ pending : 34 │ │ arithmetics : 2/3.55M, 0/330k, 0/16.6k │ pend fav : 0 │ │ known ints : 0/452k, 0/1.87M, 0/2.52M │ own finds : 81 │ │ havoc : 70/1.47M, 0/1.36M │ imported : 0 │ │ trim : 39.0 kB/27.8k (44.08% gain) │ variable : 0 │ └─────────────────────────────────────────────────────┴────────────────────────┘ ^C +++ Testing aborted by user +++ [+] We're done here. Have a nice day! prompt$

At this point, you can see it has 4 unique crashes, which means that in my out/ dir, there now exist 4 fuzzed versions of the above termcap file (quite damaged versions, I might add), that will crash cap_mkdb if you ask it to make a DB file out of them:

prompt$ ls out/crashes/ README.txt id:000000,sig:11,src:000048,op:havoc,rep:16 id:000001,sig:06,src:000059,op:havoc,rep:8 id:000002,sig:11,src:000059,op:havoc,rep:128 id:000003,sig:06,src:000059,op:havoc,rep:32

Now, the not-completely-automated-part enters, where you need to use your own smarts and rebuild this binary with debug info and run it from gdb to figure out where in the code it bombs out on you:



$ gdb cap_mkdb/cap_mkdb GNU gdb 6.3 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-unknown-openbsd5.6"... (gdb) run out/crashes/id:000000,sig:11,src:000048,op:havoc,rep:16 Starting program: /home/jj/afl2/cap_mkdb/cap_mkdb out/crashes/id:000000,sig:11,s rc:000048,op:havoc,rep:16 cap_mkdb: no name field: lpr? cap_mkdb: ignored duplicate: lpr cap_mkdb: ignored duplicate: lin?print? cap_mkdb: no name field: gl cap_mkdb: no name field: g cap_mkdb: ignored duplicate: cap_mkdb: ignored duplicate: glasst?y cap_mkdb: Record not tc expanded: ... <a lot of output from cap_mkdb goes here, since it complains about the mangled entries a lot> Program received signal SIGABRT, Aborted. 0x00000d10f694a34a in kill () at <stdin>:2 2 <stdin>: No such file or directory. in <stdin> Current language: auto; currently asm (gdb) bt full #0 0x00000d10f694a34a in kill () at <stdin>:2 No locals. #1 0x00000d10f69839e4 in __stack_smash_handler ( func=0xd10f6ab0aaa "cgetnext", damaged=Variable "damaged" is not available. ) at /usr/src/lib/libc/sys/stack_protector.c:61 sdata = {log_stat = 0, log_tag = 0xd0e597086a0 "cap_mkdb", log_fac = 8, log_mask = 255} message = "stack overflow in function %s" sa = {__sigaction_u = {__sa_handler = 0, __sa_sigaction = 0}, sa_mask = 0, sa_flags = 0} mask = 4294967263 #2 0x00000d10f6952cc3 in cgetnext (cap=0x7f7ffffd5530, db_array=0x6) at /usr/src/lib/libc/gen/getcap.c:801 len = Variable "len" is not available.

the binary "mopchk" also has issues with strange input:



(gdb) run out4/crashes id:000000,sig:11,src:000009,op:flip1,pos:14 Starting program: /home/jj/afl2/mopd/mopchk/obj/mopchk out4/crashes id:000000,sig:11,src:000009,op:flip1,pos:14 Checking: out4/crashes Checking: id:000000,sig:11,src:000009,op:flip1,pos:14 Unknown file. Program exited normally. (gdb) run out4/crashes/id:000000,sig:11,src:000009,op:flip1,pos:14 Starting program: /home/jj/afl2/mopd/mopchk/obj/mopchk out4/crashes/id:000000,sig:11,src:000009,op:flip1,pos:14 Checking: out4/crashes/id:000000,sig:11,src:000009,op:flip1,pos:14 Program received signal SIGSEGV, Segmentation fault. 0x0000174079c0a588 in GetMopFileInfo (dl=0x7f7ffffd9400, info=1) at /home/jj/afl2/mopd/mopchk/../common/file.c:280 280 isize = (header[isd+EISD_L_SECSIZE+3]*0x1000000 + (gdb) bt full #0 0x0000174079c0a588 in GetMopFileInfo (dl=0x7f7ffffd9400, info=1) at /home/jj/afl2/mopd/mopchk/../common/file.c:280 header = "?\0000", '\0' <repeats 11 times>, "\200\000\001", '\0' <repeats 197 times>, "t", '\0' <repeats 295 times>, "\005" image_type = Variable "image_type" is not available.

Looking at the mangled infiles and of course all the normal debug procedures will be of help to fix these issues. stdhosts(8) seems to have issues with a lot of single-char lines (i.e lots of blank lines), but after trying out this patch for it:

diff -u stdhosts*/stdhosts.c --- stdhosts/stdhosts.c Tue Dec 2 13:19:02 2014 +++ stdhosts2/stdhosts.c Fri Jan 2 21:24:35 2015 @@ -46,6 +46,9 @@ int len = strlen(buf); done += len; + + if (len == 1 && buf[0] == '

') continue; + if (len > 1 && buf[len-2] == '\\' && buf[len-1] == '

') { int ch;

Program received signal SIGSEGV, Segmentation fault. main (argc=2, argv=0x7f7ffffbd718) at stdhosts.c:118 118 while (!isspace(*p)) /* find first "space" */ (gdb) bt full #0 main (argc=2, argv=0x7f7ffffbd718) at stdhosts.c:118 data_line = "0.0.5\000\000\000st\000\000\000\000\000l\000$O\000$\000\000\225\225\225\225\225\225\225\225\000\200\000\000", '\225' <repeats 35 times>, "#t\000\000\2\ 25\225\225\225\225\225\225\225?", '\225' <repeats 43 times>, "?\225\225\225\225\225\225\225.5\000\000", '\225' <repeats 129 times>, "\221\225\225\225\225\225\225\225\225j"\ , '\225' <repeats 36 times>, "\206\225\225\225\225\225?", '\225' <repeats 29 times>, "?", '\225' <repeats 65 times>, "?", '\225' <repeats 67 times>, '?' <repeats 22 times>\ , '\225' <repeats 162 times>, "\235\225\225\225", '?' <repeats 17 times>, '\225' <repeats 23 times>, ":16d", '#' <repeats 11 times>... p = 0x7f7fffffc000 <Address 0x7f7fffffc000 out of bounds> line_no = 79 len = Variable "len" is not available.

it seems that the fuzzer still beats it at times (not as fast though) and produces even stranger inputs for stdhosts to croak on:

So, this afl-fuzzer is a neat way for me and you to become that elephant in the porcelain store that smashes something everytime you turn around. The hard part is actually fixing the found issues. The cap_mkdb seems to relate to libc line parsing code, so it could potentially affect other similar programs too.

If anyone wants to take a stab at fixing the things I found so far, I've posted the crashy inputs here: http://c66.it.su.se:8080/afl-fuzz/