[Detector Support]: ONNX on Jetson Orin NX: 320x320 working, 640x640 problematic with no obvious/explicit error

[zeroflow]

Describe the problem you are having

I'm running 0.16.0 Beta 3 on my Orin NX 16GB. Jetpack 6.2 / L4T 36.4.3 (latest as of writing).

With some great help from NickM, the ONNX backend for the -jp6 image was fixed.
Now with support for ONNX, I purchased Frigate+ and trained on my own images.

Now the 320x320 model works fine, both with the base model and with my fine-tuned mode.
Since the Orin NX should have enough power, I tried the 640x640 model, but that did not work.

EDIT: TensorRT 640 models, like the Yolo7x-640 work fine (but obviously slower), ONNX models, e.g. YOLO-NAS S 640 does not.

Version

0.16.0-f141b58

Frigate config file

mqtt:
  enabled: true
  host: 192.168.4.11

ffmpeg:
  hwaccel_args: preset-jetson-h264

# ---------------------
detectors:
  onnx:
    type: onnx
model:
  # PLUS
  # path: plus://94cf39f8a42bf67ad1892b4925e8d2bf # yolonas	640x640
  # path: plus://7efe1d2424d8522a3f292a4efbb92324 # yolonas	320x320
  # path: plus://811a797d7a31cdb4c0e2cff7e35d64d9 # yolonas 320 custom 03/06/2025	2025.1
  path: plus://f6d350fef1ae10a679256a23e845ee9e # yolonas 640 custom 03/06/2025 2025.1
#-------------------

# model:
#   model_type: yolonas
#   width: 320 # <--- should match whatever was set in notebook
#   height: 320 # <--- should match whatever was set in notebook
#   input_pixel_format: bgr
#   input_tensor: nchw
#   # YOLO-NAS-S
#   #path: /config/model_cache/yolo_nas_s.onnx
#   labelmap_path: /labelmap/coco-80.txt

# # ---------------------
# # Settings for TensorRT (known good)
# detectors:
#   tensorrt:
#     type: tensorrt
#     device: 0
# model:
#   path: /config/model_cache/tensorrt/yolov7x-320.trt
#   #path: /config/model_cache/yolo_nas_l.trt
#   input_tensor: nchw
#   input_pixel_format: rgb
#   width: 320
#   height: 320
# #-------------------

cameras:
  Terrasse_Garage: # --------------------------------------------------
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Terrasse_Garage
          input_args: preset-rtsp-restream
          roles:
            - detect
    onvif:
      host: 192.168.5.26
      port: 2020
      user: <redacted>
      password: <redacted>
    detect:
      enabled: true
    objects:
      filters:
        car:
          min_area: 50000
      mask: 0,0,0.549,0,0.511,0.031,0.045,0.409,0,0.446
    notifications:
      enabled: true
    zones:
      Main:
        coordinates: 0.045,0.409,0.311,0.157,0.511,0.031,0.709,0.051,1,0.176,1,1,0.478,1,0.217,0.683
        loitering_time: 0
        inertia: 3
    motion:
      threshold: 30
      contour_area: 5
      improve_contrast: true
      mask:
        - 0,0,0.365,0,0.365,0.05,0,0.05
        - 0,0,0.549,0,1,0,1,0.176,0.707,0.051,0.511,0.031,0,0.446
  Terrasse: # --------------------------------------------------
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Terrasse
          input_args: preset-rtsp-restream
          roles:
            - detect
    onvif:
      host: 192.168.5.27
      port: 2020
      user: <redacted>
      password: <redacted>
    detect:
      enabled: true
    notifications:
      enabled: true
    zones:
      Main:
        coordinates: 0.001,0.295,0.473,0.066,1,0.177,1,1,0,1
        loitering_time: 0
        inertia: 3
    motion:
      mask:
        - 0,0,0.365,0,0.365,0.05,0,0.05
        - 0,0,1,0,1,0.177,0.473,0.066,0.001,0.295
  Eingang: # --------------------------------------------------
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Eingang
          input_args: preset-rtsp-restream
          roles:
            - detect
    onvif:
      host: 192.168.5.25
      port: 2020
      user: <redacted>
      password: <redacted>

    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: true
      mask:
        - 0,0,0.365,0,0.365,0.05,0,0.05
        - 0,0,1,0,1,0.182,0.657,0.085,0.279,0.089,0,0.159
    zones:
      Main:
        coordinates: 0,0.169,0.263,0.089,0.648,0.083,1,0.182,1,1,0,1
        loitering_time: 0
        inertia: 3
    notifications:
      enabled: true
    objects:
      mask: 0,0,1,0,1,0.223,0.662,0.122,0.287,0.122,0,0.205
    review:
      alerts:
        required_zones: Main
      detections:
        required_zones: Main
  Einfahrt: # --------------------------------------------------
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Einfahrt
          input_args: preset-rtsp-restream
          roles:
            - detect
    onvif:
      host: 192.168.5.23
      port: 2020
      user: <redacted>
      password: <redacted>
    detect:
      enabled: true
    notifications:
      enabled: true
    zones:
      Main:
        coordinates: 0,0.15,0.132,0.05,0.365,0,1,0,1,1,0,1
        loitering_time: 0
        inertia: 3
  Wohnzimmer: # --------------------------------------------------
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Wohnzimmer
          input_args: preset-rtsp-restream
          roles:
            - record
        # - path: rtsp://127.0.0.1:8554/Wohnzimmer_sub
        #   input_args: preset-rtsp-restream
        #   roles:
        #     - detect
    onvif:
      host: 192.168.5.20
      port: 2020
      user: <redacted>
      password: <redacted>
    record: # Don't keep constant recordings
      retain:
        days: 0
    detect:
      enabled: true
    notifications:
      enabled: true
    zones:
      Main:
        coordinates: 0,0.158,1,0.143,1,1,0,1
        loitering_time: 0
        inertia: 3
    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: true
  Keller: # --------------------------------------------------
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Keller
          input_args: preset-rtsp-restream
          roles:
            - record
        # - path: rtsp://127.0.0.1:8554/Keller_sub
        #   input_args: preset-rtsp-restream
        #   roles:
        #     - detect
    onvif:
      host: 192.168.5.21
      port: 2020
      user: <redacted>
      password: <redacted>
    record: # Don't keep constant recordings
      retain:
        days: 0
    detect:
      enabled: true
    notifications:
      enabled: true
    zones:
      Main:
        coordinates: 0,0.196,1,0.202,1,1,0,1
        loitering_time: 0
        inertia: 3
  Garage: # --------------------------------------------------
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Garage
          input_args: preset-rtsp-restream
          roles:
            - record
        # - path: rtsp://127.0.0.1:8554/Garage_sub
        #   input_args: preset-rtsp-restream
        #   roles:
        #     - detect
    onvif:
      host: 192.168.5.22
      port: 2020
      user: <redacted>
      password: <redacted>
    record: # Don't keep constant recordings
      retain:
        days: 0
    detect:
      enabled: true
    notifications:
      enabled: true
    objects:
      track:
        - person
      filters:
        motorcycle:
          mask: 0.163,0.467,0.44,0.459,0.443,1,0.171,1
        car:
          mask: 0.043,0.465,0.325,0.462,0.328,0.89,0.049,0.883
    zones:
      Main:
        coordinates: 0.394,0,1,0,1,1,0.319,1,0.405,0.578
        loitering_time: 0
    motion:
      mask:
        - 0,0,0.365,0,0.365,0.05,0,0.05
        - 0,0.05,0.127,0.05,0.146,0.424,0,0.433
  Arbeitszimmer2: # --------------------------------------------------
    enabled: false
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Arbeitszimmer2
          input_args: preset-rtsp-restream
        # - path: rtsp://zone:<redacted>@192.168.160.20:80/videoMain
        #   input_args: preset-rtsp-restream
          roles:
            - detect
    # onvif:
    #   host: 192.168.5.26
    #   port: 2020
    #   user: <redacted>
    #   password: <redacted>
    detect:
      enabled: true
    objects:
      filters:
        car:
          min_area: 50000
    notifications:
      enabled: true
    zones:
      Main:
        coordinates: 0,0,1,0,1,1,0,1
        loitering_time: 0
        inertia: 3
    motion:
      threshold: 30
      contour_area: 5
      improve_contrast: true
# ---------------
# go2rtc restream
# ---------------
go2rtc:
  streams:
    # RTSP & Tapo two-way
    Terrasse_Garage:
      - rtsp://<redacted>@192.168.5.26:554/stream1
      - tapo://<redacted>@192.168.5.26
      - ffmpeg:Terrasse_Garage#audio=aac#audio=opus
    Terrasse:
      - rtsp://<redacted>@192.168.5.27:554/stream1
      - tapo://<redacted>@192.168.5.27
      - ffmpeg:Terrasse_Garage#audio=aac#audio=opus
    Eingang:
      - rtsp://<redacted>@192.168.5.25:554/stream1
      - tapo://<redacted>@192.168.5.25
      - ffmpeg:Terrasse_Garage#audio=aac#audio=opus
    Einfahrt:
      - rtsp://<redacted>@192.168.5.23:554/stream1
      - tapo://<redacted>@192.168.5.23
      - ffmpeg:Terrasse_Garage#audio=aac#audio=opus
    Wohnzimmer:
      - rtsp://<redacted>@192.168.5.20:554/stream1
      - tapo://<redacted>@192.168.5.20
      - ffmpeg:Terrasse_Garage#audio=aac#audio=opus
    #Wohnzimmer_sub:
    #  - rtsp://<redacted>@192.168.5.20:554/stream2
    Keller:
      - rtsp://<redacted>@192.168.5.21:554/stream1
      - tapo://<redacted>@192.168.5.21
      - ffmpeg:Terrasse_Garage#audio=aac#audio=opus
    #Keller_sub:
    #  - rtsp://<redacted>@192.168.5.21:554/stream2
    Garage:
      - rtsp://<redacted>@192.168.5.22:554/stream1
      - tapo://<redacted>@192.168.5.22
      - ffmpeg:Terrasse_Garage#audio=aac#audio=opus
    #Garage_sub:
    #  - rtsp://<redacted>@192.168.5.22:554/stream2
    #Arbeitszimmer2: rtsp://zone:<redacted>@192.168.160.20:80/videoMain
  webrtc:
    candidates:
      - 192.168.4.13:8555

# -----------------------
# General Motion Settings
# -----------------------
motion:
  mask: 0,0,0.365,0,0.365,0.05,0,0.05 # Timestamps

# ----------------------
# Detector Settings / AI
# ----------------------
detect:
  enabled: true
  stationary:
    interval: 50
    threshold: 50

objects:
  track:
    - person
    - face
    # Cars
    - motorcycle
    - bicycle
    - license_plate
    - car
    - amazon
    - fedex
    - ups
    - package
    # Animals
    # - dog
    - cat
    - deer
    - horse
    #- bird
    - raccoon
    - fox
    - bear
    - cow
    - squirrel
    - goat
    - rabbit
  filters:
    person:
      threshold: 0.9
      min_area: 10000
    car:
      threshold: 0.9
      min_area: 10000
    bus:
      threshold: 0.9
      min_area: 10000
    bicycle:
      min_area: 10000
    dog:
      threshold: 0.85
    cat:
      threshold: 0.86
    squirrel:
      threshold: 0.85
    deer:
      threshold: 0.85
    raccoon:
      threshold: 0.85
    horse:
      threshold: 0.85

# Optional: Configuration for semantic search capability
semantic_search:
  # Optional: Enable semantic search (default: shown below)
  enabled: false
  # Optional: Re-index embeddings database from historical tracked objects (default: shown below)
  reindex: false
  # Optional: Set the model size used for embeddings. (default: shown below)
  # NOTE: small model runs on CPU and large model runs on GPU
  model_size: large

# ---------------------
# Recording & Snapshots
# ---------------------

record:
  enabled: true
  retain:
    days: 7
    mode: all
  alerts:
    pre_capture: 5
    post_capture: 5
    retain:
      days: 62
      mode: active_objects
  detections:
    pre_capture: 5
    post_capture: 5
    retain:
      days: 62
      mode: active_objects

snapshots:
  enabled: true
  bounding_box: true

# ------
# Review
# ------
review:
  alerts:
    labels:
      - car
      - person
    #  - bicycle
      - motorcycle
    required_zones:
      - Main
  detections:
    labels:
      - dog
      - cat
      - deer
      - horse
      - bird
      - raccoon
      - fox
      - bear
      - cow
      - squirrel
      - goat
      - rabbit

classification:
  bird:
    enabled: false
    #threshold: 0.75

face_recognition:
  enabled: true

lpr:
  enabled: true
  device: GPU
  model_size: small # default small, optionally large
  known_plates:
    E-Klasse:
      - <redacted>
      - <redacted>

version: 0.16-0

# -------------
# Camera Groups
# -------------

camera_groups:
  Outdoor:
    order: 1
    icon: LuTrees
    cameras:
      - Terrasse_Garage
      - Terrasse
      - Eingang
      - Einfahrt
  Einfahrt:
    order: 2
    icon: LuCar
    cameras:
      - Terrasse_Garage
      - Eingang
      - Einfahrt
  Monitoring:
    order: 3
    icon: LuMonitorCheck
    cameras:
      - Terrasse_Garage
      - Terrasse
      - Eingang
      - Einfahrt
      - Wohnzimmer
      - Keller
      - Garage
notifications:
  enabled: true
  email: <redacted>

docker-compose file or Docker CLI command

version: '2.4'
services:
  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    stop_grace_period: 30s # allow enough time to shut down the various services
    image: ghcr.io/blakeblackshear/frigate:0.16.0-beta3-tensorrt-jp6
    shm_size: "2048mb" # 920MB in use
    runtime: nvidia 
    volumes:
      - /var/frigate/config:/config
      - /var/frigate/storage:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
      - /etc/localtime:/etc/localtime:ro
      - /usr/lib/aarch64-linux-gnu/tegra:/usr/lib/aarch64-linux-gnu/tegra:ro
    ports:
      - "1984:1984" # go2rtc
      - "8971:8971" # HTTPS port
      - "5000:5000" # Internal unauthenticated access. Expose carefully.
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      YOLO_MODELS: "yolov7-640,yolov7-416,yolov7-320,yolov7x-640,yolov7x-320,yolov7-640-dla,yolov7-416-dla,yolov7-320-dla,yolov7x-640-dla,yolov7x-320-dla"
      #YOLO_MODELS: "yolov7x-640,yolov7x-640-dla"
      USE_FP16: true
      PLUS_API_KEY: "<redacted>"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              #device_ids: ['0'] # this is only needed when using multiple GPUs
              count: 1 # number of GPUs
              capabilities: [gpu]

Relevant Frigate log output

warning | 2025-06-06 07:08:17 | frigate.record.maintainer      | Too many unprocessed recording segments in cache for Eingang. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...
warning | 2025-06-06 07:08:17 | frigate.record.maintainer      | Too many unprocessed recording segments in cache for Einfahrt. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...
info    | 2025-06-06 07:08:21 | detector.onnx                  | Exited detection process...
info    | 2025-06-06 07:08:21 | root                           | Detection process has exited...
info    | 2025-06-06 07:08:21 | detector.onnx                  | Starting detection process: 2557
info    | 2025-06-06 07:08:21 | frigate.detectors.plugins.onnx | ONNX: loaded onnxruntime module
info    | 2025-06-06 07:08:21 | frigate.detectors.plugins.onnx | ONNX: loading /config/model_cache/f6d350fef1ae10a679256a23e845ee9e
info    | 2025-06-06 07:08:21 | frigate.detectors.plugins.onnx | ONNX: /config/model_cache/f6d350fef1ae10a679256a23e845ee9e loaded
warning | 2025-06-06 07:08:27 | frigate.record.maintainer      | Too many unprocessed recording segments in cache for Terrasse. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...
warning | 2025-06-06 07:08:27 | frigate.record.maintainer      | Too many unprocessed recording segments in cache for Eingang. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...
warning | 2025-06-06 07:08:27 | frigate.record.maintainer      | Too many unprocessed recording segments in cache for Einfahrt. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...
warning | 2025-06-06 07:08:37 | frigate.record.maintainer      | Too many unprocessed recording segments in cache for Terrasse. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...
warning | 2025-06-06 07:08:37 | frigate.record.maintainer      | Too many unprocessed recording segments in cache for Eingang. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...
warning | 2025-06-06 07:08:37 | frigate.record.maintainer      | Too many unprocessed recording segments in cache for Einfahrt. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...
info    | 2025-06-06 07:08:41 | frigate.watchdog               | Detection appears to be stuck. Restarting detection process...
info    | 2025-06-06 07:08:41 | root                           | Waiting for detection process to exit gracefully...

Install method

Docker Compose

Object Detector

Other

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

In jtop, I see the memory use for the detector spike up rather high to ~5gig and the iGPU usage is stuck at 100%.
From how it looks, the model seems to be crashing.
Also, it seems to cause problems with the recordings, since frigate.record.maintainer is reporting unprocessed recording segments.

My main question is: Is there some way to debug the model and see where the problem is coming from?

[banthungprong]

One question: why do you use your main streams for detection? In the logs the are hints showing issues with these detect streams.
So maybe you can try to switch to the sub Streams.

[zeroflow]

I am indeed, since it worked fine with TensorRT yolo7x-640.
But I'll try enabling sub streams and use those for detection.

[zeroflow]

No, switching to substreams did not bring any improvement.

But I noticed a warning in the logs that I may have glanced over before:

unknown | 2025-06-06 07:41:14 | unknown | 2025-06-06 07:41:14.773604693 [W:onnxruntime:, transformer_memcpy.cc:74 ApplyImpl] 2 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.

EDIT: That warning also existis with the 320 size model.

[NickM-27]

Yolo7x is a smaller model than yolo-nas. Works fine on my Nvidia GPU so it must just be some overflow or other issue. I'm not aware of any deeper logs to give more info. 320x320 is recommended in most cases anyway