Breakpoints on multithreaded linux user-mode programs

[rliebig]

Currently, if the

emu.set_breakpoint(guest_addr):

Feature is used to define a breakpoint, the execution is only yielded back if the same thread that set the breakpoint hits the target address. If another thread hits this breakpoint, this does stop this specific thread but does not yield execution to the rust wrapper code back.

The reason is likely inside libafl/exit.c prepare_qemu_exit, which is used by the breakpoint api to throw cpu_loop_exit:

static void prepare_qemu_exit(CPUState* cpu, target_ulong next_pc)
{
    expected_exit = true;
    last_exit_reason.cpu = cpu;
    last_exit_reason.next_pc = next_pc;

#if !defined(CONFIG_USER_ONLY) && defined(AS_LIB)
    qemu_system_return_request();
#endif

     if (cpu->running) {
            cpu->exception_index = EXCP_LIBAFL_EXIT;
            cpu_loop_exit(cpu);
     }
        
}

A workaround seems to be available in iterating over all CPUs, which are in linux user-mode actually threads, and using qemu_cpu_kick to request a shutdown:

    CPU_FOREACH(cpu) {
        if (cpu->running) {
            cpu->exception_index = EXCP_LIBAFL_EXIT;
            qemu_cpu_kick(cpu);
        }
    }

This does however suffer from race conditions. Calling cpu_loop_exit(cpu) while iterating over the different CPUs seems to result in memory failures, therefore I presume some kind of lock is needed.

@rmalmain I would be more than happy to contribute a fix for this, but would like to hear some input in what you would believe to be a good solution for this problem.

[rliebig]

Further experimentation yielded the conclusion that expected_exit is marked as THREAD_MODIFIER and goes to thread local storage. It seems that only setting this variable as volatile instead of THREAD_MODIFIER is enough to shutdown all threads, however I suspect there is a reason that it is marked thread local.

[rliebig]

Hmm, cpu_exit() also seems like a candidate. Unfortunately, right now, the code looks like it is focused on single-threaded applications. Resuming from the breakpoint again only triggers the CPU with cpu_index = 0, because during context switches the libafl_qemu_env does not seem to be updated.

The clone_func in linux-user/syscall.c calls libafl_hook_new_thread_run which does call libafl_set_qemu_env, which is marked with __thread, but returning from the breakpoint by calling qemu.run() inside the rust code relaunches then cpu_index = 0. I believe there need to be either a global state which tracks the which thread was interrupted and then upon relaunch either switches the necessary structures to that thread or one could build something like qemu.run_thread(cpu_index = 3) to only run interesting threads in seclusion.

[rliebig]

Beyond current_cpu there is also a thread_cpu structure which is heavily used in linux-user/signal.c.