AMD Renoir Stuck at 100%

[simonyel]

I'm curious if anyone has run into this issue:

Environment:

• Frigate 0.16 beta3
• In PVE unprivileged container
• Mesa Gallium driver 22.3.6 for AMD Radeon Graphics (renoir, LLVM 15.0.6, DRM 3.57, 6.8.12-11-pve)

Frigate Config:

• 5 cameras
• 2 x M2 Coral
• 10TB NFS Storage

All is good, EXCEPT after some time (minutes to hours), the integrated AMD GPU pegs out at 100% and doesn't go down not even after all the containers on that PVE node are shut down.

[NickM-27]

what are you using for object detection? I have seen this happen a couple times while testing, it seems radeontop has a bug that makes it get stuck showing 100% GPU usage even though it is not being used

[simonyel]

2 x Google Coral TPUs via M2 bus. It's not just faulty showing 100% util, it's doing something. I'm seeing the load via amdgpu_top. 15W consumption, full clock, etc., but I don't see the PID that's doing it.

[NickM-27]

I'm seeing the load via amdgpu_top

well, like I said I see the same thing (load via utils) but I can tell it is not actually doing anything because the fans are not spinning fast. In any case the only way I have found to fix it is reboot

[simonyel]

Correct, the only thing that helps is a complete reboot of the node. I'm at a loss as to where to look next. I've updated the node and verified the GPU passthrough. Will probably update the Mesa driver in the Docker container next.

[simonyel]

It seems to be Semantic Search and\or Face Recognition. I turned those two things off and haven't seen the issue in over a day.