Skip to content

[BUG] RKVDEC2 driver crash in H.264/H.265 decode due to unsafe PPS scaling list handling #362

@chrisbeeze

Description

@chrisbeeze

Description

We are seeing reproducible kernel crashes during H.264/H.265 decoding on RK3588 when using the RKVDEC driver.
The issue appears in the decoder cleanup path (mpp_dma_release → mpp_task_finalize → rkvdec_free_task).

Kernel Log
Example trace:

kernel: rk_gmac-dwmac fe1c0000.ethernet ethO: Link is Up - 1Gbps/Full - flow control rx/tx
kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ethO: link becomes ready
kernel: ttyFIQ ttyFIQ0: tty_port_close_start: tty->count = 1 port count = 2
kernel: dwc3 fc000000.usb: request 0000000006337ea4 was not queued to epdout
kernel: android_work: did not send uevent (0 © 0000000000000000)
kernel: of_dma_tequest_slave_channel: dma-names property of node '/serialafeb60000' missing or empty
kernel: dw-apb-uart feb60000.serial: failed to request DMA, use interrupt mode
kernel: vec3v3_lcd0_n: disabling
kernel: vec_1v8_cam_s0: disabling
kernel: vec_2v8_cam_s0: disabling
kernel: Unable to handle kernel paging request at virtual address 0000000000001260
kernel: Mem abort info:
kernel: ESR = 0x96000005
kernel: EC = 0x25: DABT (current EL), IL = 32 bits
kernel: SET = 0, FnV = 0
kernel: EA = 0, S1PTW = 0
kernel: Data abort info:
kernel: ISV = 0, ISS = 0x00000005
kernel: CM = 0, WnR = 0
kernel: user pgtable: 4k pages, 39-bit VAs, pgdp=00000001f9035000
kernel: [0000000000001260] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
kernel: Internal error: Oops: 96000005 [#1] SMP
kernel: Modules Linked in:
kernel: CPU: 1 PID: 149 Comm: queue_work9 Not tainted 5.10.160 #28
kernel: Hardware name: Rockchip RK3588S MARS3500s (DT)
kernel: pstate: 40c00009 (nZcv daif +PAN +UAO -TCO BTYPE=--)
kernel: pe : mutex_Lock+0x28/0x70
kernel: ir : mutex_Lock+0x1c/0x70
kernel: sp : ffffffcodadd3c60
kernel: x29: FFFFFfcOdadd3c60 x28: FFFFFFE1F9752000
kernel: x27: FFFFFfCOOa20cb48 x26: FFFFFFCOVa20c000
kernel: x25: 0000000000000107 x24: FfFfFF81FOC9fcaO
kernel: x23: fFFFFF81f96fa00 x22: FFFFFFE1F9752038
kernel: x21: FfffFf81f975a088 x20: 0000000000001260
kernel: x19: 0000000000001a60 x18: 0000000000000000
kernel: x17: 0000000000000000 x16: 0000000000000000
kernel: x15: 0000007f8904e8c8 x14: 0000000000000000
kernel: x13: 0000000000000000 x12: o000000000000000
kernel: x11: 0000000000000001 x10: 0000000000000990
kernel: x9 : FFfFFFCOO9ICFSE4 x8 : FFTFFFBIfidibfro
kernel: x7 : 0000000000000001 x6 : FFFTFF81F975ae48
kernel: x5 : 0000000000000000 x4 : 0000000000000000
kernel: x3 : 0000000000000000 x2 : 0000000000000000
kernel: x1 : 000000000000000 xo : FFFfFF81fidib600
kernel: Call trace
kernel: mutex_lock+0x28/0x70
kernel: mpp_dma_release+0x28/0xac
kernel: mpp_task_finalize+0x60/Oxb4
kernel: rkvdec2_free_task+0x1c/0:
kernel: mpp_free_task+0x50/0xfO
kernel: rkvdec2_Soft_ccu_worker+0x254/Oxcf4
kernel: kthread_worker_fn+0x12c/0x280
kernel: kthread+0x12c/0x130
kernel: ret_from_fork+0x10/0x24

Suspected Problem Areas

File: drivers/video/rockchip/mpp/mpp_rkvdec.c

1- fill_scaling_list_pps()

pps = map.vaddr + offset;
memcpy(&scaling_fd, pps + base, sizeof(scaling_fd));

No bounds checking on offset / base. If userspace provides a bad PPS buffer or offset, pps + base can point outside the mapped area, leading to memory corruption and later kernel Oops during cleanup.

2- rkvdec_alloc_task() fail path

mpp_task_finalize(session, mpp_task);
kfree(task);

Always calls mpp_task_finalize(), even if the task wasn’t fully initialized. This can trigger invalid releases in mpp_dma_release().

Conditions

  • Happens during H.264/H.265 decoding (not using VP8/VP9).
    *Triggered when PPS scaling list handling is active (e.g., corrupted PPS buffer or unusual offset).
  • Platform: RK3588, kernel 5.10.160 (also relevant for 6.1 branch).

Suggested Fix

  • Add proper bounds checks in fill_scaling_list_pps() to ensure (offset + base + sizeof(u32)) <= buffer_size.
  • Guard mpp_task_finalize() so it only runs when the task is fully initialized.

Impact

This leads to hard kernel crashes during decoding. Fixing it would improve decoder robustness against bad/corrupt input streams.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions