Fuzzing NetBSD Filesystems via AFL. [Part 2]

Recently I started working on Fuzzing Filesystems on NetBSD using AFL.
You can take a look at the previous post to learn more details about background of this project.
This post summarizes the work that has been done in this area, and is divided into 3 sections:

1. Porting AFL kernel mode to work with NetBSD
2. Running kernel fuzzing benchmark
3. Example howto fuzzing particular Filesystem

AFL Port for NetBSD

AFL is a well known fuzzer for user space programs and libraries, but with some changes it can be used for fuzzing the kernel binary itself.

For the first step to fuzz the NetBSD kernel via AFL, I needed to modify it to use coverage data provided by the kernel instead of compiled instrumentations. My initial plan was to replace the coverage data gathered via afl-as with that provided by kcov(4). In this scenario, AFL would just run a wrapper and see the real coverage from the kernel.

I also saw previous work done by Oracle in this area, where instead of running the wrapper as a binary, the wrapper code was included in a custom library (.so object).

Both approaches have some pros and cons. One thing that convinced me to use a solution based on the shared library with initialization code was the potentially easier integration with remote fork server. AFL has some constraints in the way of managing fuzzed binary, and keeping it on a remote VM is less portable than fuzzing using a shared library and avoiding introducing changes to the original binary fuzzing.
Porting AFL kernel fuzzing mode to be compatible with NetBSD kernel mainly relied on how the operating system manages the coverage data. The port can be found currently on github.

Writing a kernel fuzzing benchmark

Performance is one of the key factors of fuzzing. If performance of the fuzzing process is not good enough, it's likely that the entire solution won't be useful in practice. In this section we will evaluate our fuzzer with a practice benchmark.

One exercise that I want to perform to check the AFL kernel fuzzing in practice is similar to a password cracking benchmark. The high level idea is that a fuzzer based on coverage should be much smarter than bruteforce or random generation.

To do this, we can write a simple program that will take a text input and compare it with a hardcoded value. If the values match, then the fuzzer cracked the password. Otherwise, it will perform another iteration with a modified input.

Instead of "password cracker", I called my kernel program "lottery dev". It's a character device that takes an input and compares it with a string.

The chance to find one 6 byte combination (or the "lucky byte" combination, because of the name) are similar to winning big in the lottery: every byte contains 8 bits, thus we have 2**(8*6) => 281,474,976,710,656 combinations. The coverage based fuzzer should be able to do this much quicker and in fewer iterations, as feedback from code instrumentations will show, compared to blindly guessing.

I performed a similar test using a simple C program: the program read stdio and compared it with the hardcoded pattern. If the pattern matches, the program panics, and otherwise returns zero. Such a test took an AFL about a few hours on my local laptop to break the challenge (some important details can make it faster). The curious reader who wants to learn some basics of AFL should try to do run a similar test on their machine.

I ran the fuzzer on my lottery dev for several days, and after almost a week it was still not able to find the combination. So something was fundamentally not right. The kernel module with the wrapper code can be found here.

Measuring Coverage for a particular function

In the previous article, I mentioned that the NetBSD kernel seems to be 'more verbose' in terms of coverage reporting. I ran my lottery dev wrapper code (the code that writes a given input to the char device) to check the coverage data using standard kcov(4) without the AFL module. My idea was to check the ratio between entries of my code that I wanted to track and other kernel functions that can be considered noise from other subsystems. These operations are caused by the context services, such as Memory Management, File Systems or Power Management, that are executed in the same process.

To my surprise, there was a lot of data, but I could not find any of functions from lottery dev. I quickly noticed that the amount of addresses is equal to the size of kcov(4) buffer, so maybe my data didn't fit in the buffer inside kernel space?

I changed the size of the coverage buffer to make it significantly larger and recompiled the kernel. With this change I reran the test. Now, with the buffer being large enough, I collected the data and printed top 20 entries with a number of occurrences. There were 30578 entries in total.

1544 /usr/netbsd/src/sys/uvm/uvm_page.c:847
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:869
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:890
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:880
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:858
1281 /usr/netbsd/src/sys/arch/amd64/compile/obj/GENERIC/./machine/cpu.h:70
1281 /usr/netbsd/src/sys/arch/amd64/compile/obj/GENERIC/./machine/cpu.h:71
 478 /usr/netbsd/src/sys/kern/kern_mutex.c:840
 456 /usr/netbsd/src/sys/arch/x86/x86/pmap.c:3046
 438 /usr/netbsd/src/sys/kern/kern_mutex.c:837
 438 /usr/netbsd/src/sys/kern/kern_mutex.c:835
 398 /usr/netbsd/src/sys/kern/kern_mutex.c:838
 383 /usr/netbsd/src/sys/uvm/uvm_page.c:186
 308 /usr/netbsd/src/sys/lib/libkern/../../../common/lib/libc/gen/rb.c:129
 307 /usr/netbsd/src/sys/lib/libkern/../../../common/lib/libc/gen/rb.c:130
 307 /usr/netbsd/src/sys/uvm/uvm_page.c:178
 307 /usr/netbsd/src/sys/uvm/uvm_page.c:1568
 231 /usr/netbsd/src/sys/lib/libkern/../../../common/lib/libc/gen/rb.c:135
 230 /usr/netbsd/src/sys/uvm/uvm_page.c:1567
 228 /usr/netbsd/src/sys/kern/kern_synch.c:416

It should not be a surprise that the coverage data does not much help our fuzzing with AFL. Most of the information that the fuzzer sees is related to UVM page management and machine-dependent code.

I decided to remove instrumentation from these most common functions to check the difference. Using the attribute no_instrument_function tells the compiler to not put instrumentation for coverage tracing inside these functions.

Unfortunately, after recompiling the kernel, the most common functions did not disappear from the list. As I figured out, the support in GCC 7 may not be fully in place.

GCC 8 for help

To solve this issue, I decided to work on reusing GCC 8 for building the NetBSD kernel. After fixing basic build warnings, I got my basic kernel working. This still needs more work to get kcov(4) fully functional. Hopefully, in the next report, I will be able to share these results.

Fuzzing Filesystem

Given what we already know, we can run Filesystem fuzzing. As a target I chose FFS as it is a default FS that is delivered with NetBSD.

The reader may ask the question: why would you run coverage based fuzzer if the data is not 100% accurate? So here is the trick: For coverage based fuzzers, it is usually recommended to leave the input format as is, as genetic algorithms can do a pretty good job here. There is great post on Michal Zalewski's Blog about this process applied for the JPEG format: "Pulling JPEGs out of thin air". But what will AFL do if we provide an input format that is already correct? We already know what a valid FS image should look like (or we can simply just generate one). As it turns out, AFL will start performing operations on the input in a similar way as mutation fuzzers do - another great source that explains this process can be found here: "Binary fuzzing strategies: what works, what doesn't"

Writing mount wrapper

As we discussed in the previous paragraph, to fuzz the kernel itself, we need some code to run operations inside the kernel. We will call it a wrapper, as it wraps the operations of every cycle of fuzzing. The first step to write a wrapper for AFL is to describe it as a sequence of operations. Bash style scripting is usually good enough to do that.
We need to have an input that will be modified by the fuzzer, and be able to mount it. NetBSD comes with vnd(4) that allows exposing regular files as block devices. The simplest sequence can be described as:

# Expose file from tmpfs as block device
vndconfig vnd0 /tmp/rand.tmp

# Create a new FS image on the blk dev that we created
newfs /dev/vnd0

# Mount our fresh FS
mount /dev/vnd0 /mnt

# Check if FS works fine
echo "FFS mounted!" > /mnt/test

# Undo mount
umount /mnt

# Last undo step
vndconfig -u vnd0

From bash to C and system calls

At this point, the reader has probably figured out that a shell script won't be the best approach for fuzzer usage. We need to change it to C code and use proper syscall/libc interfaces.

vndconfig uses the opendisk(3) combined with vnd_ioctl. mount(2) is a simple system call which can operate directly after file is added to vnd(4)

Below is an example conceptual code for mounting an FS:

	// Structure required by mount()
	struct ufs_args ufs_args;

	// VNConfigure step
	rv = run_config(VND_CONFIG, dev, fpath);
	if (rv) 
		printf("VND_CONFIG failed: rv: %d\n", rv);

	// Mount FS
	if (mount(FS_TYPE, fs_name, mntflags, &ufs_args, 8) == -1) {
		printf("Mount failed: %s", strerror(errno));
	} else {
		// Here FS is mounted
		// We can perform any other operations on it
	
		// Umount FS
		if (unmount(fs_name, 0) == -1) printf("#: Umount failed!\n");
	}

	// VNC-unconfigure
	rv = run_config(VND_UNCONFIG, dev, fpath);
	if (rv) {
		printf("VND_UNCONFIG failed: rv: %d\n", rv);
	}

The complete code can be found here

Ready to fuzz FFS! aka Running FS Fuzzing with a predifined corpus

The first thing that we need to do is to have a wrapper that provides mount/umount functionality. In the previous section, we have already shown how that can be done. For now, we will be fuzzing the same kernel that we are running on. Isn't that dangerous? Taking a saw to the branch we are sitting on? Of course it is! In this exercise I want to illustrate an idea from a technical perspective so the curious reader is able to understand it better and do any modifications by their own. The take away from this exercise is that the fuzzing target is the kernel itself, the same binary that is running the fuzzing process.

Let's come back to the wrapper code. We've already discussed how it works. Now we need to compile it as a shared library. This is not obvious, but should be easy to understand after we already brought this sawing-off metaphor.

To compile the so object:

gcc -fPIC -lutil  -g -shared ./wrapper_mount.c -o wrapper_mount.so 

Now we need to create the input corpus, for the first attempt we will use a large enough empty file.

dd if=/dev/zero of=./in/test1 bs=10k count=8

And finally run. The @@ tells AFL to put here the name of input file that will be used for fuzzing.

./afl-fuzz -k -i ./in -o ./out -- /mypath/wrapper_mount.so @@

Now, as we described earlier, we need a proper FS image to allow AFL perform mutations on it. The only difference is the additional newfs(8) command.

# We need a file, big enough to fit FS image but not too big
dd if=/dev/zero of=./in/test1 bs=10k count=8

# A block is already inside fuzzer ./in
vndconfig vnd0 ./in/test1

# Create new FFS filesystem
newfs /dev/vnd0

vndconfig -u vnd0

Now we are ready for another run!

./afl-fuzz -k -i ./in -o ./out -- /mypath/wrapper_mount.so @@



                  american fuzzy lop 2.35b (wrapper_mount.so)

┌─ process timing ─────────────────────────────────────┬─ overall results ─────┐
│        run time : 0 days, 0 hrs, 0 min, 17 sec       │  cycles done : 0      │
│   last new path : none seen yet                      │  total paths : 1      │
│ last uniq crash : none seen yet                      │ uniq crashes : 0      │
│  last uniq hang : none seen yet                      │   uniq hangs : 0      │
├─ cycle progress ────────────────────┬─ map coverage ─┴───────────────────────┤
│  now processing : 0 (0.00%)         │    map density : 17.28% / 17.31%       │
│ paths timed out : 0 (0.00%)         │ count coverage : 3.53 bits/tuple       │
├─ stage progress ────────────────────┼─ findings in depth ────────────────────┤
│  now trying : trim 512/512          │ favored paths : 1 (100.00%)            │
│ stage execs : 15/160 (9.38%)        │  new edges on : 1 (100.00%)            │
│ total execs : 202                   │ total crashes : 0 (0 unique)           │
│  exec speed : 47.74/sec (slow!)     │   total hangs : 0 (0 unique)           │
├─ fuzzing strategy yields ───────────┴───────────────┬─ path geometry ────────┤
│   bit flips : 0/0, 0/0, 0/0                         │    levels : 1          │
│  byte flips : 0/0, 0/0, 0/0                         │   pending : 1          │
│ arithmetics : 0/0, 0/0, 0/0                         │  pend fav : 1          │
│  known ints : 0/0, 0/0, 0/0                         │ own finds : 0          │
│  dictionary : 0/0, 0/0, 0/0                         │  imported : n/a        │
│       havoc : 0/0, 0/0                              │ stability : 23.66%     │
│        trim : n/a, n/a                              ├────────────────────────┘
┴─────────────────────────────────────────────────────┘             [cpu:  0%]

Future work

Support for the NetBSD kernel fuzzing was developed as a part of the AFL FileSystems Fuzzing project, which aims to improve quality of filesystems and catch various issues.
The very next thing that I have on my todo list is to provide support for kernel tracing on GCC 8 to turn off coverage data from other functions that generate a lot of noise.
During the FFS fuzzing, I found a few issues that I need to analyze in detail.
Last but not least, for the next report I plan to show a remote setup of AFL running on a VM, reporting crashes, and being remotely rebooted by the master controller.