This section explains the process of installing CodeQL, as well as how to build and analyze a CodeQL database.| Introduction on Testing Handbook
This section explains the process of installing Semgrep and how to start using it.| Introduction on Testing Handbook
libFuzzer # libFuzzer is the clear and easy choice if you need to fuzz your C/C++ program, because it is part of the LLVM project and is available on most platforms. We recommend fuzzing on Linux if possible because it is the platform with the best support for libFuzzer (e.g., it is not preinstalled in XCode with macOS). Microsoft’s MSVC compiler has recently gained support for libFuzzer. Note that libFuzzer has been in maintenance-only mode since late 2022, so no new features will be added.| Introduction on Testing Handbook
Writing harnesses # The following section showcases some techniques to successfully write a fuzzing harness—the most important part of any fuzzing setup. If written poorly, critical parts of your application may not be covered. Beyond byte arrays # Often the code you want to fuzz not only takes a plain byte array as input, but has more complex input. A very basic example is the following fuzz test that contains a division by 0.| Introduction on Testing Handbook
AFL++ # The AFL++ fuzzer is a fork from the AFL fuzzer. It offers better fuzzing performance and more advanced features while still being a very stable alternative to libFuzzer. A major benefit over libFuzzer is that AFL++ has stable support for running fuzzing campaigns on multiple cores (see Multi-core fuzzing). This section of the Testing Handbook is based on fuzzing binaries written in C/C++ on Ubuntu on x64_64. AFL++ supports different environments like macOS, but there are caveats.| Introduction on Testing Handbook
cargo-fuzz # The cargo-fuzz tool is the de facto choice for fuzzing your Rust project when using Cargo. It uses libFuzzer as the back end. Note that if you are not using Cargo, you cannot use the cargo-fuzz tool. By installing the cargo-fuzz crate, a Cargo subcommand is installed. Therefore, cargo-fuzz depends on using Cargo. The subcommand also automatically enables relevant compilation flags for your Rust project and even supports enabling sanitizers like AddressSanitizer.| Introduction on Testing Handbook
Coverage analysis # Gaining confidence in your code coverage archived during fuzzing is essential for two reasons. Firstly, you want to assess which parts of your applications your fuzzing harnesses execute. For example, a magic value check, like the one shown in the following figure, may be hard for a fuzzer to overcome. Discovering such a check is important so that the values can be provided to the fuzzer through a dictionary or test cases in the seed corpus.| Introduction on Testing Handbook
Coverage analysis # Gaining confidence in your code coverage archived during fuzzing is essential for two reasons. Firstly, you want to assess which parts of your applications your fuzzing harnesses execute. For example, a magic value check, like the one shown in the following figure, may be hard for a fuzzer to overcome. Discovering such a check is important so that the values can be provided to the fuzzer through a dictionary or test cases in the seed corpus.| Introduction on Testing Handbook
Fuzzing dictionary # A dictionary can be used to guide the fuzzer. A dictionary is usually passed as a file to the fuzzer. The simplest input accepted by libFuzzer is a ASCII text file where each line consists of a quoted string. Strings can contain escaped byte sequences like “\xF7\xF8". Optionally, a key-value pair like hex_value="\xF7\xF8" can be used for documentation purposes. Comments are supported by starting a line with #.| Introduction on Testing Handbook
SUT patching: Overcoming obstacles # Codebases are often not fuzzing-friendly. This can happen if, for example, the code uses checksums or depends on a global state like a system-time seeded PRNG (i.e., by using rand) that causes the code to behave differently for the same input. Refer to Practical harness rules to learn more about potential problems in your SUT. If you encounter checksums or a global state in your SUT, you may want to apply fuzzing-specific patches to change the behavior of ...| Introduction on Testing Handbook
SUT patching: Overcoming obstacles # Codebases are often not fuzzing-friendly. This can happen if, for example, the code uses checksums or depends on a global state like a system-time seeded PRNG (i.e., from the rand) that causes the code to behave differently for the same input. Refer to Practical harness rules to learn more about potential problems in your SUT. If you encounter checksums or a global state in your SUT, you may want to apply fuzzing-specific patches to change the behavior of ...| Introduction on Testing Handbook
Writing harnesses # In the following, we will go over Rust specific tips to optimize the results from your harnesses. For general advice, refer to Writing harnesses. Structure-Aware fuzzing with the arbitrary crate # The arbitrary crate simplifies writing fuzzing harnesses. By deriving a macro, Rust structs can be targeted for fuzzing. For example, the following code requires constructing a Name struct that owns a String. We derived the Arbitrary macro to facilitate the construction of such a...| Introduction on Testing Handbook
AddressSanitizer # AddressSanitizer (ASan) is a widely adopted tool in the realm of software testing, particularly during fuzzing. Fuzzing greatly benefits from the use of ASan because it helps detect memory errors that might otherwise go unnoticed, such as buffer overflows and use-after-free errors. While ASan is a standard practice in fuzzing due to its effectiveness in identifying such vulnerabilities, it does come with certain drawbacks. One significant downside is that it can make the fu...| Introduction on Testing Handbook
LibAFL # The LibAFL fuzzer implements features from AFL-based fuzzers like AFL++. Similarly to AFL++, LibAFL provides better fuzzing performance and more advanced features over libFuzzer. However, with LibAFL, all functionality is provided in a modular and customizable way—in fact, LibAFL is a library that can be used to implement custom fuzzers. Because LibAFL is a library, there is no single-line command to install LibAFL like there is with libFuzzer (apt install clang) and AFL++ (apt ins...| Introduction on Testing Handbook
Python # We recommend using Atheris to fuzz Python code. Installation # Atheris supports 32-bit and 64-bit Linux, and macOS. We recommend fuzzing on Linux because it’s simpler to manage and often faster. If you’d like to run Atheris in a Linux environment on a Mac or Windows system, we recommend using Docker Desktop. If you’d like a fully operational Linux environment, see the Dockerfile section below. If you’d like to install Atheris locally, first install a recent version of clang, ...| Introduction on Testing Handbook
Fuzzing environments # Like any software, the choice of fuzzer will depend on factors such as the operating system, architecture, software versions, and hardware. This section will review factors that influence the choice of the environment used for fuzzing. Choice of hardware. If your fuzzer supports running on multiple cores, choose hardware that has many cores available. Renting or purchasing a bare-metal server might be worthwhile if you plan to run campaigns regularly.| Introduction on Testing Handbook
Ruby # We recommend using Ruzzy to fuzz Ruby code. Installation # Ruzzy supports Linux x86-64 and AArch64/ARM64. If you’d like to run Ruzzy on a macOS or Windows, you can build the Dockerfile and/or use the development environment. Ruzzy requires a recent version of clang (tested back to 14.0.0), preferably the latest release. Install Ruzzy with the following command: MAKE="make --environment-overrides V=1" \ CC="/path/to/clang" \ CXX="/path/to/clang++" \ LDSHARED="/path/to/clang -shared" \...| Introduction on Testing Handbook
FAQ (Fuzzily Asked Questions) # The fuzzer is showing crashes, but when I run the SUT on the test cases outside of the fuzzer, then no crash is shown. What is happening? # There are several potential reasons: The SUT is behaving nondeterministically. The SUT depends on global state (e.g., it may read or write to disk or use in-memory singletons). You are experiencing crashes because your system is running out of memory and killing processes.| Introduction on Testing Handbook
OSS-Fuzz # OSS-Fuzz is an open-source project developed by Google that aims to improve the security and stability of open-source software by providing free distributed infrastructure for continuous fuzz festing. Using a pre-existing framework like OSS-Fuzz has many advantages over manually running harnesses: it streamlines the process and facilitates simpler modifications. Although only select projects are accepted into OSS-Fuzz, because the project’s core is open-source, anyone can host th...| Introduction on Testing Handbook
Additional resources # Awesome fuzzing list 1. GitHub-hosted list focused on scientific publications about fuzzing. Awesome fuzzing list 2. GitHub-hosted list about fuzzers and fuzzing related books. Fuzzing handbook. A fuzzing handbook written from an academic perspective. CNCF-Fuzzing handbook.. Handbook created by the CNCF. Fuzzing101. Tutorial and training for various fuzzing methods by GitHub Security Lab.| Introduction on Testing Handbook
This section describes how to create and test custom CodeQL queries and query packs.| Introduction on Testing Handbook
This section explores the advanced usage of Semgrep, including how to create new rules.| Introduction on Testing Handbook
This section describes the process of integrating CodeQL into your continuous integration and continuous delivery (CI/CD) pipeline.| Introduction on Testing Handbook
This section explains the process of integrating Semgrep into your continuous integration and continuous delivery (CI/CD) pipeline.| Introduction on Testing Handbook
This section discusses the process of introducing Semgrep to your organization.| Introduction on Testing Handbook
Wycheproof testing harness example in Python # The following section will showcase how to write a simple testing harness to test a Python library implementing AES in GCM mode. We will use the cryptography package as an example of an AES-GCM implementation. For testing, we will use the pytest package, one of the most popular testing frameworks in Python. Prerequisites # Add the Wycheproof repository as a submodule to your existing repository.| Introduction on Testing Handbook
Dudect # Overview # Dudect is a statistical constant-time analysis tool that measures the execution time of a specific code section for two input classes and aims to find the statical difference between the measurements of the two classes. If the timing measurements for the two input classes deviate from one another, it would suggest that the code is dependent on the input and, therefore, not constant time. The two most commonly used input classes are:| Introduction on Testing Handbook
Timecop (Valgrind) # Timecop is a wrapper around Valgrind designed to detect potential timing leaks dynamically. It allows developers to mark memory regions as secret, and if during runtime, a branching instruction or memory access is performed that is dependent on the secret memory region, Valgrind will report the behavior, helping to identify timing vulnerabilities. Overview # Timecop is a C macro wrapper around functions provided by Valgrind. Setup # To use Timecop, you must first install ...| Introduction on Testing Handbook
Learn more about CodeQL using the external resources listed on this page.| Testing Handbook
Learn more about Semgrep using the external resources listed on this page.| Testing Handbook
Snapshot fuzzing enables security engineers to effectively test software that is traditionally difficult to analyze, such as kernel-level software (though the technique is not limited to such software). Whether you’re auditing drivers or other kernel-mode components, including antivirus software, snapshot fuzzing provides a robust way to discover critical vulnerabilities. Consult this section for a walkthrough on how to conduct snapshot fuzzing on your system.| Testing Handbook