As we mentioned in our our last blog post, Tobias Ospelt of Pentagrid AG lately broke down the idea of Java fuzzing with JQF into simple steps and presented it at Swiss Cyber Storm in Bern, Switzerland and at the Black Alps conference in Yverdon-Les-Bain, Switzerland. The recordings of the talks are not online, yet.

We wanted to describe our work in written form as well, so you get a better idea of what can be done by fuzzing Java with JQF. It also allows you to copy and paste commands if you want to try it yourself. We also uploaded the talk slides as a PDF and the video with the JQF tutorial and Bouncycastle ASN.1 parser fuzzing run, which were shown during the talks.

Introduction to fuzzing Fuzzing is a topic in the IT security world that was able to keep its mysteriousness. Every now and then a new fuzzer tool is created and every new fuzzer finds new bugs. Various organisations run fuzzing projects for various reasons and use CPU-years to find security issues. We believe it is still by far the most common way to find memory corruption security issues. The reason why fuzzing is so successful is simple: Fuzzing is the very abstract concept of feeding input in an automated, clever and partially random way to a target program and try to observe undesired behaviour. We are sure there are more exact definitions of fuzzing, but they do not matter for this blog post. The fuzzing field is very big and a lot of people spend their entire professional lives with it. There are many fuzzers available online. If you want to learn about fuzzing in general, you can check out the fuzzing book for example.

AFL The AFL fuzzer was first released in 2014 by Michal Zalewski when he was still at Google. There were many other fuzzers before it, but the combination of an easy way to instrument the code, ease of use in general and the usage of a genetic algorithm with a feedback loop that favours inputs that trigger new code paths revolutionized the field. If you want to know more about AFL and see the impressive bugs it found, you better head over to the original AFL project page. There are as well numerous forks of AFL, the one we currently recommend for runs is AFL++ that seems to be the best maintained one while still aiming for the same goals as the original AFL. The thing is, AFL was written to be able to fuzz programs written in memory-unsafe languages, especially C and C++. It operates on files and feeds them to target programs. For this blog post it is important to know that AFL will usually put files that result in crashes of targeted programs into the crash directory. Hangs refer input files that took too long to process, as AFL sets a timeout limit on how long an input can take to process. This means a hang file might just take a little longer than AFL was willing to wait, but it might also mean you found an infinite loop or similar in a program. There is also the path metric which shows how many code paths in the target program were triggered by the input files that were tested so far. This is a very important metric to see if instrumentation works and the genetic algorithm can do its magic.

Fuzzing Java The ideas of what kind of security issues can be found with Java fuzzing came up already before Pentagrid AG was founded (blog post in Tobias' blog: Java Bugs with and without Fuzzing – AFL-based Java fuzzers and the Java Security Manager). However, since then the fuzzers evolved. While most fuzzers that were evaluated back then didn't move much on until today, the JQF fuzzer developed by Rohan Padhye at University of Berkeley took a big leap forward. And that's why we only talk about JQF here.

JQF-Zest JQF-Zest is the newer approach that still does the genetic algorithm approach of AFL. Hangs and "crashes" (uncaught exceptions) as well assertion violations are all now named "unique failures" and "total paths" are are now in the "total coverage" metric as branches. Apart from that it behaves similarly to JQF-AFL. However, it implements a couple of improvements and is aiming at developers and people who would like to use fuzzing in their continuous integration. But it also kept its usefulness for security researchers. Developers will be more familiar with this JUnit test approach and might be able to reuse large parts of their Unit Tests for fuzzing. JQF-Zest supports other Java types than InputStreams. It will automatically understand types such as int, String and Map because there are JUnit quickcheck generators for them. If a test case you have in mind takes an argument where no generator is available, you can still write your own generator. In the simple case of a single argument of InputStream the input file will correspond directly to the InputStream. However, in all other cases the mapping of an input file to Unit Test arguments is non-trivial and therefore you will later need the jqf-repro program to reproduce function arguments from input files. The good thing is that this file approach will still allow you to reuse past runs for subsequent runs with JQF-Zest to improve coverage. However, as the input and output files do not contain direct values but rather represent seed values for JUnit Quickcheck Generators, it is hard to reuse existing corpus data that wasn't produced by JQF-Zest. When writing tests, the assume* methods can be used to abort a test early, if the (often randomly) generated inputs don't have a certain form. While on one hand this can be regarded as additional instrumentation and you have to take care that you don't prevent bugs from being found by assuming too much, it can be really powerful on the other hand. By adding an assume statement at the beginning of the Unit Test, JQF-Zest will try to pass the assume statements with its inputs and therefore make as many tests pass the assume statement as possible. This allows to target certain tests cases a little more focused than it would otherwise be possible. A good example is the PatriciaTrie example in the JQF repository: @Fuzz public void testCopy ( Map < String , Integer > map , String key ) { assumeTrue ( map . containsKey ( key )); // Create new trie with input `map` Trie trie = new PatriciaTrie ( map ); // The key should exist in the trie as well assertTrue ( trie . containsKey ( key )); } The test is requiring a Map and a String to be passed as arguments and the first assume statement will make sure that the String is in the Map (as a key). JQF-Zest will know this requirement and will try to comply with it. On the other hand, we will never be able to find issues in PatriciaTrie's containsKey method with this test that only occurs when a non-existing key is passed to PatriciaTrie's containsKey . While JQF-Zest is available as a command line tool, it is also available as a maven plugin, meaning you don't have to install JQF yourself and maven will do it transparently for you. It allows easy continuous integration and you can choose to do a fuzzing run for a limited time by specifying the -Dtime=5m argument to the maven jqf-zest plugin. In our experience JQF-Zest is also faster than JQF-AFL, although we haven't benchmarked it.

Bug classes As already said, a broader picture of what can be found with fuzzing in the Java world can be found in another blog post. But let's focus on JQF here. So far JQF was able to find bugs in the area of the following bug classes: Exception issues potentially affecting availability, for example ClassCastException or IndexOutOfBoundsException.

Denial of Service issues, for example infinite loops in code and out of memory conditions (OutOfMemoryError).

Logic bugs, if the correct assert statements are written such as for the above PatriciaTrie example. We assume the following bug classes could be found as well by using JQF: Any security issues that would lead to an uncaught exception, for example an SQLSyntaxErrorException for SQL injections. However, this also means that the exceptions is not allowed to be caught before it reaches JQF.

Any security issues that can be expressed by assert statements, such as differential fuzzing for cryptographic libraries.

Any security issues that can be prohibited by the Java Security Policy Manager, such as Server-Side Request Forgery (SSRF) by preventing network access and subsequently this would throw an Exception when network access is attempted. In the end, a lot of different bug classes can be found as long as the correct assert statement is created, but it is more a question of how much effort it takes.