Deadlock: inst_in_pipeline leak when L1I miss resolves after warp completion

[jzhou1318]

Deadlock Error

When an L1I cache miss response arrives after all threads in a warp have already completed functionally, decode() still runs and incrementsinst_in_pipeline. The instruction is placed in the ibuffer but can never issue because the scheduler skips warps where waiting() is true. dec_inst_in_pipeline never fires, so inst_in_pipeline stays at 1 permanently, hardware_done() never returns true, the warp exit path never fires, and the core deadlocks.

I saw this error when running multi-kernel workloads (more kernels = higher chance of hitting the race) & deadlock typically appears at kernel boundaries when all warps need to exit. Example disreptancy when all pipeline stages empty, L1I MSHR empty, response FIFO empty. Instruction is in ibuffer but will never be issued.:

w04 npc: 0x00a0, done:fs  :32 i:1 s:0 a:0
active=00000000000000000000000000000000

Race between L1I miss latency and warp completion. Sequence:

1. Warp fetches instruction → L1I miss
2. Other instructions execute, all threads complete (functional_done() = true)
3. L1I miss response arrives late → decode() runs → ibuffer_fill + inc_inst_in_pipeline
4. Scheduler skips warp forever (waiting() = true)
5. dec_inst_in_pipeline never fires → hardware_done() blocked → deadlock

Proposed Fix

In decode(), after filling ibuffer, flush it if the warp is already functional_done(). ibuffer_flush() calls dec_inst_in_pipeline() for each valid entry, keeping the counter balanced. Instructions flushed here would never have executed anyway since the scheduler skips completed warps.

void shader_core_ctx::decode() {
  if (m_inst_fetch_buffer.m_valid) {
    // decode 1 or 2 instructions and place them into ibuffer
    address_type pc = m_inst_fetch_buffer.m_pc;
    const warp_inst_t *pI1 = get_next_inst(m_inst_fetch_buffer.m_warp_id, pc);
      if (pI1) {
      m_warp[m_inst_fetch_buffer.m_warp_id]->ibuffer_fill(0, pI1);
      m_warp[m_inst_fetch_buffer.m_warp_id]->inc_inst_in_pipeline();
      m_stats->m_num_decoded_insn[m_sid]++;
      // ADD THIS:
      // if (m_warp[m_inst_fetch_buffer.m_warp_id]->functional_done()) {
      //   fprintf(stderr, "BUG pI1 : decode on functional_done warp %u, sid=%u, cycle=%llu\n",
      //          m_inst_fetch_buffer.m_warp_id, m_sid,
      //          m_gpu->gpu_tot_sim_cycle + m_gpu->gpu_sim_cycle);
      //   fflush(stderr);  
      //   abort();
      // }
      if ((pI1->oprnd_type == INT_OP) ||
          (pI1->oprnd_type == UN_OP)) {  // these counters get added up in mcPat
                                         // to compute scheduler power
        m_stats->m_num_INTdecoded_insn[m_sid]++;
      } else if (pI1->oprnd_type == FP_OP) {
        m_stats->m_num_FPdecoded_insn[m_sid]++;
      }
      const warp_inst_t *pI2 =
          get_next_inst(m_inst_fetch_buffer.m_warp_id, pc + pI1->isize);
      if (pI2) {
        m_warp[m_inst_fetch_buffer.m_warp_id]->ibuffer_fill(1, pI2);
        m_warp[m_inst_fetch_buffer.m_warp_id]->inc_inst_in_pipeline();
        // ADD THIS:
        // if (m_warp[m_inst_fetch_buffer.m_warp_id]->functional_done()) {
        //   fprintf(stderr, "BUG pI2 : decode on functional_done warp %u, sid=%u, cycle=%llu\n",
        //         m_inst_fetch_buffer.m_warp_id, m_sid,
        //         m_gpu->gpu_tot_sim_cycle + m_gpu->gpu_sim_cycle);
        //   fflush(stderr);  
        //   abort();      
        // }
        m_stats->m_num_decoded_insn[m_sid]++;
        if ((pI1->oprnd_type == INT_OP) ||
            (pI1->oprnd_type == UN_OP)) {  // these counters get added up in
                                           // mcPat to compute scheduler power
          m_stats->m_num_INTdecoded_insn[m_sid]++;
        } else if (pI2->oprnd_type == FP_OP) {
          m_stats->m_num_FPdecoded_insn[m_sid]++;
        }
      }
    }
    // FIX
    if (m_warp[m_inst_fetch_buffer.m_warp_id]->functional_done()) {
        m_warp[m_inst_fetch_buffer.m_warp_id]->ibuffer_flush();
    }
    m_inst_fetch_buffer.m_valid = false;
  }
}

[JRPan]

sounds like related to this one? #503
Do you think this can be it?

[jzhou1318]

Yes! That PR looks like the cleaner fix. Thanks!