hardware acceleration for Amlogic A311D mainline kernel support

[jimmyhon]

Thanks to @dsx724, I have gotten a hold of a Libre Computer Alta (https://libre.computer/products/aml-a311d-cc/) which uses the Amlogic A311D SoC.

That SoC contains a 5 TOPS Etnaviv NPU supported by Mesa's Teflon since 25.0.
https://docs.mesa3d.org/teflon.html

The mainline kernel enables NPU in the device trees in 6.17
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/arch/arm64/boot/dts/amlogic?h=v6.17-rc1

Going into Frigate 17.0 is support for using Mesa Teflon as an Object Detector (i.e. teflon_tfl). #18310
The inference time on the A311D using the cpu_model.tflite is ~20ms.
Note: the tflite delegate on A311D takes a long time to compile the cpu_model.tflite (60,000+ ms). Whereas the tflite delegate on a Rockchip RK3588 takes less than 100ms to compile the cpu_model.tflite.

The SoC contains video decoder with staging driver support. (meson_vdec https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/media/meson/vdec?h=v6.16)
FFmpeg can use hwaccel_args: -c:v h264_v4l2m2m for hardware acceleration. However, it doesn't seem to lower CPU usage much. With or without it, the ffmpeg capture process uses 20% cpu utilization when used for detect. Also, it seems like meson_vdec can only by used by one client at a time.

With 3 cameras (2 x 1920x1080 detection stream and a 1440p camera with 640x360 detection stream), the Libre Computer Alta uses 5w (measured using a Kill A Watt). Running with just the passive heatsink (no fan), the CPU sensor reports 60 C.

Configuration

config.yaml

detectors:
  npu:
    type: teflon_tfl
ffmpeg:
  hwaccel_args: -c:v h264_v4l2m2m

docker-compose.yml

    devices:
      - /dev/dri/:/dev/dri/ # etnaviv
      - /dev/video0:/dev/video0 # meson-vdec

Related

• hardware acceleration for Rockchip RK3588 mainline kernel support #18311

[jimmyhon]

Since Frigate 16 is based on Debian bookworm, you could modify an existing Frigate 16 image to include support for teflon_tfl

Dockerfile

FROM ghcr.io/blakeblackshear/frigate:stable-standard-arm64

RUN echo "deb http://deb.debian.org/debian bookworm-backports main" > /etc/apt/sources.list.d/backports.list
RUN apt update && apt install -y mesa-teflon-delegate/bookworm-backports

COPY teflon_tfl.py /opt/frigate/frigate/detectors/plugins/
COPY detector_utils.py /opt/frigate/frigate/detectors/
COPY config.py /opt/frigate/frigate/config/

Pull python files from #18310

[jimmyhon]

Adventures in trying to use Frigate 16 using Teflon on Libre Computer Alta

I'm using a stable mainline kernel Linux 6.16.7-1-aarch64-ARCH

If I use the cpu object detector, everything runs fine for more than a day. The object detector will use 240% cpu on the 4xA73 cores.

If I use the teflon_tfl object detector, frigate will run detections for ~30 seconds, and then the whole board will lock up. Network gets disconnected. HDMI output freezes. Keyboard input does not respond.

To debug this, I've modified the cpu_tfl.py to pickle the detection tensor_input into a file, and also the (boxes, class_ids, scores) output to a file. If I run those files through an another script to invoke the delegate, the board does not lock up and completes all the inferences. So I'm having a trouble recreating the board lockup with a simpler/smaller test case.

Unfortunately, when the board locks up, nothing shows up in the dmesg -w

Also locks up when running on Libre Computer 6.12 kernel 6.12.42-gfed5d838bb74

Note: the same frigate software while running on the Rockchip RK3588 rocket driver is fine.

[jimmyhon]

Limited both policy0 and policy2 scaling_max_frequency to 1.2 GHz...board still locks up.

Are there any kernel level debug flags I should turn on so soem output spits out on the UART when the board locks up?

[jimmyhon]

As a debugging step, I went into the frigate container and have a script run through 13000 invocations() in 8 minutes using the NPU. (Previously pickle dumped from Frigate's cpu_tfl.py detector.) The CPU temperature (with a fan blowing) only went up to 50 C.

[jimmyhon]

Okay, I can reliably cause the board to lockup without using frigate.

The key behavior is to add a 200 ms sleep between invocations. It's even more noticeable if wait is increased to 400ms. With a 100 ms sleep, the board continues running.

jimmyhon/TensorFlow-Lite-Object-Detection-on-Android-and-Raspberry-Pi#1

[jimmyhon]

As a hacky workaround, have a script that invokes the teflon delegate every 100ms, and that will keep the board from locking up.

[dsx724]

I will send this to Tomeu. There is a lot of dimensions for this IP that was RE and maybe not parameterized correctly.