On the Usability of Coverage-Based Fuzzing of C/C++ Programs

Abstract

Even though the foundations for fuzzing were laid more than 30 years ago, it did not play a role in industry or academia for a long time. Interestingly, the popularity of fuzzing has risen for top-tier companies and academia in recent years. There are no firm findings on the reasons for this, but a continued increase in awareness of the need for secure software systems and access to more computing power may have played a role. In addition, fuzzing has proven to be capable of finding severe software vulnerabilities that slipped through the established security process. In academia, the focus on the topic of fuzzing lies heavily in improving performance, e.g., better execution speed or covering more code faster or making fuzzing applicable to more software and systems. However, examining the usability of fuzzing has yet to be touched.<br /> A drawback of the current state of fuzzing is that especially small or medium-sized companies are not benefiting from it because they do not use it. The reasons for this still need to be explored. Motivated by the lack of understanding about the usability of fuzzing and its potential negative influence on widespread adoption, I examine in this thesis the usability of fuzzers in the context of C and C++ programs by conducting four user studies.<br /> In the first qualitative study with computer science (CS) students and Capture the Flag (CTF) players, I examined the usability of a fuzzer and a static code analysis tool, with static code analysis being a technique that is already very commonly used. This way, I found first insights into potential problems that hinder the adoption of fuzzing. The results revealed several usability issues, which I converted into recommendations.<br /> I strengthened the qualitative findings of my first study with a quantitative comparison of the two most popular fuzzers, AFL and libFuzzer, in a second user study with CS students.<br /> Large parts of the original recommendations could be confirmed and supplemented. Moreover, I could show that AFL performed better overall and was rated higher than libFuzzer.<br /> To be able to make a more generalizable statement on the findings, I replicated my second study, but this time with freelance developers. With this study, I provided insights into the methodological implications of substituting freelancers with students in fuzzing studies and solidified the quantitative results for both fuzzers.<br /> In my fourth study, I investigated whether developers should fuzz their self-written code or code written by others. This was motivated by existing examples in related fields for and against analyzing someone's own code. In the results, I could not find any evidence that a practical difference exists between fuzzing self-written code and the code of others.<br /> Based on the results of the four studies, I additionally give recommendations to improve the usability of fuzzers.