dredd-webgpu-testing

Generate new tests for the WebGPU CTS using mutation testing and fuzzing.

Build

Prerequisites:

• Python 3.10
• pip

Note you must clone recursively in order to get the submodules.

git clone --recursive --depth 1 https://github.com/mc-imperial/dredd-webgpu-testing.git && cd dredd-webgpu-testing

Next, create a Python virtual environment and install the dependencies, which are listed in pyproject.toml.

python -m venv venv
source venv/bin/activate
pip install . 

Build Dredd

Get Clang and LLVM:

sudo apt install -y libzstd-dev
cd external/dredd/third_party
curl -Lo clang+llvm.tar.xz https://github.com/llvm/llvm-project/releases/download/llvmorg-17.0.6/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04.tar.xz
tar xf clang+llvm.tar.xz
rm -rf clang+llvm
mv clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04 clang+llvm
rm clang+llvm.tar.xz
cd ..

Build Dredd:

mkdir build && cd build
cmake -G Ninja .. -DCMAKE_BUILD_TYPE=Release
cmake --build . --config Release
cp src/dredd/dredd ../third_party/clang+llvm/bin

Check it worked:

dredd/third_party/clang+llvm/bin/dredd --help

Mutate the subject

The mutation subject can be Dawn or Mesa.

First, get two checkouts of the subject version that you want to mutate. Build a mutated and tracked version once without any Dredd intervention in order to produce a compile commands database. For Mesa:

git clone https://gitlab.freedesktop.org/mesa/mesa.git mesa_mutated
git clone https://gitlab.freedesktop.org/mesa/mesa.git mesa_tracked

Follow the build instructions. For Ubuntu 24.04, first get the Mesa prerequisites:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
sudo apt-get build-dep mesa

Install SPIRV-Tools (for Ubuntu 24.04 or see instructions here to install the Vulkan SDK):

wget -qO- https://packages.lunarg.com/lunarg-signing-key-pub.asc | sudo tee /etc/apt/trusted.gpg.d/lunarg.asc
sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-noble.list http://packages.lunarg.com/vulkan/lunarg-vulkan-noble.list
sudo apt update
sudo apt install vulkan-sdk

Next, use the script to build and mutate the mutated and tracked subjects. This will:

• Perform a clean build with the required build options
• Inject mutants into the mutation version of the subject
• Inject mutant coverage instrumentation into the tracking version of the subject

cd dredd-webgpu-testing
source venv/bin/activate
cd src
python -m mutate mesa /path/to/mesa_mutated /path/to/mesa_tracked \
    --mutation_dir src/gallium/drivers/llvmpipe \
    --dredd /path/to/dredd