x32-fuzzer

The Behringer X32 digital mixer has a rare crash bug. Let's find it.

Table of Contents

• Motivation
• Findings
• Installation and Usage

Motivation

The Behringer X32 is a very powerful and surprisingly affordable digital mixer. It's what GDQ uses for all of our shows.

This mixer also can also be controlled over the network via its OSC protocol. This is where the bug lies. There is some kind of race condition or resource contention which can cause the mixer to freeze, requiring a power cycle to restore normal operation.

The goal of this repo is to build a fuzzer which can automatically discover a repro for this crash/freeze bug.

Findings

This fuzzer is now complete and can crash the NIC of an X32 in about 30 seconds or less. It does this by generating a random OSC packet, and then sending copies of that same packet over and over again, as fast as possible, from 9 different client sockets.

This means that the contents of the packets being sent doesn't matter too much, and what matters more is the volume of packets being sent. Perhaps this indicates that the root issue is a race condition or memory leak.

We also discovered that just sending a high volume of random bytes won't crash the mixer. The data being sent does have to be valid OSC packets for the crash to occur.

Most importantly, we could not reproduce the crash when running only a single fuzzing client (achieved via setting fuzzing.numFuzzers to 1 in config.json). The fuzzer was left running for 19 hours in this configuration, and no crash occurred. However, when running even just 2 fuzzing clients, the crash occurs within 5 minutes. This suggests that the issue is only present when multiple clients are connected to the X32.

Interestingly, when the mixer is in this crashed state, sending OSC over MIDI still works. The MIDI jacks seem totally unaffected by this particular crash that this fuzzer causes.

You'll know when the crash has occurred because the mixer will stop responding to pings, and the fuzzer script will say that it has stopped receiving heartbeats. Additionally, you'll see an error like this on the X32's main display:

Installation and Usage

1. Install Node.js 8 or later (10 is recommended).

2. Clone this repository:

   git clone https://github.com/GamesDoneQuick/x32-fuzzer.git

3. Install this project's dependencies:

   cd x32-fuzzer
   npm install

4. Configure the fuzzer by creating a config.json file in the x32-fuzzer directory. The only parameter you need is mixerIp, the rest are optional and documented here.

   {
   	"mixerIp": "192.168.1.62"
   }

5. Run the fuzzer:

   # From the x32-fuzzer directory:
   npm start