AMD iGPU bad scaling - lower detection rate

[ortofan]

Describe the problem you are having

I have two systems for redundency. Both systems are fed the same cameras. One of them is an intel 14500T the other is an AMD 8845HS. Both have Coral Tpu and other than the cpu (iGPU) they are essentially the same. h.264 and vaapi used for both.
With any sort of scaling enabled on the AMD the image gets a bit granier and straight lines get jagged etc...typical scaling problem.
My prefred way of using it all would be to have go2rtc scale the main channel form each camera (1920x1080 to 1280x720). This saves on network streams and also provide the same resolution for each camera (not all have substream with correct aspect ratio).

However with scaling enabled, detection rate on the AMD iGPU is about 10% lower than on the intel iGPU, clearly becase the image is less clear.

If I chose a substream from each camera for detect using birdseye, even though the resolution is set for the same as the substream (640x360), randomly some of the cameras get scaled up. I know this becasue the camera text will get smaller. When this scaling happens it is clear that the AMD iGPU scaling is super bad comapred to the intel system. (Sorry I guess I described 2 problems, 1 with AMD scaling and an other about Birdseye wanting to do extra work for no reason and also effected by scaling).

Version

14.1

Frigate config file

database:
  path: /data/frigate.db

detectors:
  coral1:
    type: edgetpu
    device: pci:0
  coral2:
    type: edgetpu
    device: pci:1

objects:
  track:
    - person
    - car
    - dog
    - cat

record:
  enabled: true
  retain:
    days: 30
    mode: motion
  events:
    retain:
      default: 30
      mode: active_objects

snapshots:
  enabled: true
  bounding_box: true

birdseye:
  enabled: true
  width: 640
  height: 360
  mode: objects
  inactivity_threshold: 10
  layout:
    scaling_factor: 3.0

go2rtc:
  streams:
    Cam_F_01:
      - rtsp://username:password(redected)@192.168.101.201:554/Streaming/Channels/101#video=h264#hardware=vaapi
    Cam_F_01_lowres:
      - rtsp://username:password(redected)@192.168.101.201:554/Streaming/Channels/102#video=h264#hardware=vaapi

# + 9 more cameras configured the same way


cameras:
#
  Cam_F_01:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Cam_F_01
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/Cam_F_01_lowres
          input_args: preset-rtsp-restream
          roles:
            - detect
# + 9 more cameras configured the same way

version: 0.14

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped #AMD GPU CRASH PROTECTION
    image: ghcr.io/blakeblackshear/frigate:0.15.0-beta2
    shm_size: "512" # update for your cameras based on calculation above
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /App_Containers/Frigate/config/config.yml:/config/config.yml
      - /App_Containers/Frigate/data:/data # Need matching statment in config.yml
      - /z_ld_NVR:/media/frigate
      - type: tmpfs # Optional: 512MB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 512000000

    devices:
      # - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/apex_1:/dev/apex_1 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      # - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
      # - /dev/dri/card0 # for intel hwaccel, needs to be updated for your hardware


    environment:
      FRIGATE_RTSP_PASSWORD: "password"

    stop_grace_period: 120s


networks:
  default:
    name: "frigate" # name of the mcvlan network
    external: true

Relevant Frigate log output

No log output for bad scaling :(

Relevant go2rtc log output

No relevant logs

FFprobe output from your camera

Here is the VAinfo

Vainfo Output
Return Code: 0

Process Output:

vainfo: VA-API version: 1.17 (libva 2.10.0)
vainfo: Driver version: Mesa Gallium driver 22.3.6 for AMD Radeon Graphics (gfx1103_r1, LLVM 15.0.6, DRM 3.57, 6.8.0-51-generic)
vainfo: Supported profile and entrypoints
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
VAProfileAV1Profile0 : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc

Install method

Docker Compose

Object Detector

Coral

Network connection

Wired

Camera make and model

Sevarl newer and older Hikvision models.

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

Birdseye not scaled same as substreem resolution

Birdseye scaled up (and down???) text is unreadably.

[ortofan]

The above config was for using substreams from the cameras, where is my go2rtc rescale option

• "ffmpeg:Forras_01#video=h264#width=640#height=360#raw=-fpsmax 10#hardware=vaapi"

[NickM-27]

You might need to define an extra argument to try a different scaling technique than the default.

Also using a birdseye resolution that small is just recommended

[ortofan]

On the BirdsEye issue,
I understand it basically uses the "detect" stream.
If I setup detect to use the cameras sub stream (640x360) so no AMD iGPU scaling issue, when I open a feed from the birdseye view it is also at that same low resolution.
Interestinlgy enough, if I use go2rtc for both record and detect (scaled to lower res and frame rate) when I open a camera feed from birdseye it then displays full HD feed. So there must be some logic in the background that switches streams, but that logic does not work without go2rtc. Perhaps this is a feature request and not a bug, that no matter what the detect (and birdseye uses), when you open a camerafeed from birdseye it switches to the "record" stream whcih will alwas be the highest resolution.

[NickM-27]

Birdseye can only ever use the detect stream, because decoding is required. It does not make sense for birdseye to decode the higher res recording stream just for a composite stream.

[ortofan]

Aha, so go2 rtc will always decode then? All streams setup in go2rtc are decoded streams? Even with no connections to them? I thought they are created on the fly.
Again I can have go2rtc feed both record and detect (low res low frame rate). and when I click on a camera I get full hd.
If I use direct paths to camera high res and low res streams when I click on a camera I get low res feed. this is both intel, amd and 14 and 15. But I get it. Will open a feature request, (Start high res Decode on camera feed when opened from Birdseye).

[NickM-27]

No, go2rtc does not decode, Frigate does. If you setup go2rtc correctly then go2rtc will be used in the Frigate webUI. This is unrelated to birdseye

Will open a feature request, (Start high res Decode on camera feed when opened from Birdseye).

this is not something we are looking to support. The resource usage will be far too high not to mention many other challenges with feeds being out of sync.

[ortofan]

Well, just installed 15 hoping ffmpeg 7 helps. Sadly it was the same.
On the other hand just noticed that power efficiency with 15 is better :).

I think there may be something borken with AMD iGPU ffmpeg becase now I see other artifacts not just scaling.
I have basically set both record and detect on the main streams 1920x1080 for all cameras, ditched go2rtc completly, to my surprise power consumption was less than before evven with 20fps full HD decode! Detection rate is much better, but still image quality in general with the AMD iGPU is crap compared to intel. Left to right Intel iGPU in Frigate, Hikvision Camera Webview, AMD iGPU in firgate this is about a 25%crop. How could there be such a huge difference? (I am pretty sure this isnt your fault though :), but I am pretty upset that my shiny new AMD host is so bad. I think for most people this is ok, and frankly If you do not have both system types side by side you may never notice it. But clearly with a cleaner image everything works mucht better.

[ortofan]

Well after using the search term "amd gpu h.264 quality" I got a bunch of hits...

Endless number of articles, reviews and complaints on how video on AMD hardware sucks. Especially h.264. Apperetnly it is the worst on Linux in general.
No amount of knob twisting can make it better according to the internet experts.

What is even more troubling is that the loss of resolution and scaling artifects are not linear. Meaning the worse the initial video quailty the larger the image quality gap between amd, intel and nvidia.
Given security cameras in general have pretty aweful image quality, it is no surprise to see such huge difference even with my best and latest camera.

This is clearly a problem for every single AMD igpu user. They might not know about it (no side by side systems), but it is a problem.

*** It is not Just contrast. *** It is *** overall detail (resolution) *** as well as the *** scaling artifacts ***.

Garbage in garbage out. No matter what model or AI you use the issue is prevelent. If you feed your AI worst quality video it would make sense it would be less successful with detection.

Due to this amd graphics quality issue, frigate will perform far worse using AMD then intel or Nvidia.
In my case

During the day test run 6 hours mid day.
Intel iGPU 262 detection.
AMD iGPU 243 detection.

Night time dusk till dawn (I got foxes, cats and dogs etc...)
Intel iGPU 66
AMD iGPU 51

*** I think this is a serius shortcomings of somethings the community should be aware of even if it isnt really a Frigate issue.

For me the cost of the system was the same, if I had knonw this prior I would not have purchased and AMD hoping for a bit lower power consumption. (Even if by findings these knobs can be fixed, for sure my otherwise clean config file will be a mass).

[NickM-27]

No amount of knob twisting can make it better according to the internet experts.

Let's be clear, I've spent years working on ffmpeg in frigate, I also personally have multiple devices with AMD iGPUs (as well as other GPUs from all vendors). In general AMD is worse at encoding but in the vast majority of Frigate use cases encoding is not necessary, and the decoding quality for Frigate's use case is just fine.

You'll likely see more changes on the Intel side as well if you use preset-vaapi compared to qsv. This is something that became more prevalent in ffmpeg 7 with the scale_vaapi filter.

As far as knobs to turn:

• There are plenty of options listed right in the ffmpeg docs https://trac.ffmpeg.org/wiki/Hardware/VAAPI
• You can also see there's different scaling modes available https://www.mltframework.org/plugins/FilterAvfilter-scale_vaapi/#avmode
• And you can also look into applying sharpening and other filters after the VAAPI scale

Due to this amd graphics quality issue, frigate will perform far worse using AMD then intel or Nvidia.

As someone who has tested and used AMD iGPUs as well as Intel / Nvidia GPUs in frigate I definitely disagree and I've not seen this with my hardware or any users who have requested support, definitely nowhere near the level that you suggest it is.

Also, due to the nature of how object detection works the regions that are sent to the object detectors are often cropped / resized anyway so this difference in quality won't make as much of an impact as you suggest. We would want to dig a lot deeper into the specifics to see in what cases some detections were missed in your case and what the config was, it is possible the difference is unrelated to that.

I'm not suggesting you should spend time trying to fix things or that what you are seeing is invalid, but there are many different factors at play and I'd suggest avoiding sweeping generalist statements with only a singular hardware experience. We'd need to see a lot more information about the config, driver versions, system metrics, etc. to get a better understanding of the specifics in this circumstance.

[NickM-27]

also worth mentioning that in your config you are re-encoding the recording and detect streams to H264

go2rtc:
  streams:
    Cam_F_01:
      - rtsp://username:password(redected)@192.168.101.201:554/Streaming/Channels/101#video=h264#hardware=vaapi
    Cam_F_01_lowres:
      - rtsp://username:password(redected)@192.168.101.201:554/Streaming/Channels/102#video=h264#hardware=vaapi

which does not make any sense in general considering your camera should support h264 OOTB. With each change made it will be helpful to see a full config so we can understand if there may be some mistakes being made that are further causing issues

[ortofan]

Lets try to reboot issue simplify and hopefully make some progress.
I am no expert I simply share my experience, observation and looking for a solution. The links provided talks about shortocming of the AMD ffmpeg implementation, but do not help me understand what to do in FRIGATE about them. I do not think this is an isolated case of only my two different gpus working differently, but rather a real difference between the present capablities of AMD and Intle iGPUs and ffmpeg implementation.

I have two identical system with the only difference being

• Intel 14500T cpu/gpu
• AMD 8845HS cpu/gpu

Using the same settings in FRIGATE the following was observed:

AMD gpu has worse contrast, much worse scaling and overall less detail in the image. However this seems to only effect Frigate display in live, birdseye etc views. (When looking at framegrabs of the already recorded stream image quality seem to be identical.)

Can you you please provide me with some specific examples on what to put where in my FRIGATE configuration to imporove the AMD iGPU

• Contrast? (Example in Frigate camera or main config?)
• Scaling? (Example in Frigate camer or main config?)
• Overall image quality. (Example in Frigate camera or main config?)

With those examples I can take it further and report back on what settings produced identical results using both the intel iGPU (defailt no tweeks and knobs turned) and the AMD iGPU).

Thanks a million really appriciate it yout taking the time and help me get better quality video and detection rate with my AMD iGPU.

Suggestion: Perhaps have these examples in the documentation.

[NickM-27]

I need to start by seeing the current configs you are using for each device and what differences there are between the images. There are a lot of factors at play here so it's important not to make any assumptions.