Frigate+ and Yolov9 (9s, 9t, 320, 640). What Does it Mean?

[moogoos]

I am currently looking at switching to the Yolov9 models and can't get a good feel for the different options. I tried reading the docs and asking the AI agent but didn't quite get the clarity I need. Can someone help me understand the pros/cons for the 3 following openvino Yolov9 models. I do notice inference speeds with my 15th gen 265k to be about 7ms for the 320 and 14ms for the 640. Not sure what's the trade off would be here but..:

yolov9t | 320x320 | onnx,rocm,openvino
yolov9t | 640x640 | onnx,rocm,openvino
yolov9s | 640x640 | onnx,rocm,openvino

This doc says use s or t "sizes" but doesn't really specify what the difference means or the significance.
https://docs.frigate.video/plus/#yolov9-details

I reviewed these docs:
Base Models:
https://plus.frigate.video/dashboard/models/

Other Docs
https://docs.frigate.video/plus/#available-model-types
https://docs.frigate.video/configuration/object_detectors/#yolo-v3-v4-v7-v9
https://docs.frigate.video/integrations/plus/#use-models

[inkeep-ai-assistant[bot]]

Based on the Frigate+ documentation, YOLOv9 models are available in two sizes: s (small) and t (tiny) (1). Here's what these options mean:

Model Sizes

YOLOv9s (small): The s size is most similar to the current yolonas models in terms of inference times and accuracy, and a good place to start is the 320x320 resolution model for yolov9s (1).

YOLOv9t (tiny): This is a smaller, faster variant that will have quicker inference times but potentially lower accuracy compared to the s size (1).

Resolution Options

The numbers (320x320 vs 640x640) refer to the input resolution of the model (2). Based on your inference speed observations:

• 320x320: ~7ms inference time on your 15th gen i5-12600K
• 640x640: ~14ms inference time on your 15th gen i5-12600K

Trade-offs

Higher resolution (640x640):

• Better detection of smaller objects
• More detailed analysis of the scene
• Slower inference times (14ms vs 7ms in your case)

Lower resolution (320x320):

• Faster inference times
• May miss smaller objects or fine details
• Better for systems with limited processing power

Recommendation for OpenVINO

For OpenVINO detector with Intel hardware, YOLOv9 is the recommended model type (1). The documentation suggests that OpenVINO runs best with tiny, small, or medium models (3).

Given your inference speeds, you have good performance headroom with either option. The choice depends on whether you prioritize speed (320x320) or detection accuracy for smaller objects (640x640).

📚 Sources:

1. Models
2. Supported Hardware
3. Recommended hardware

---

Have another question? Just tag @inkeep.

[moogoos]

is 14ms really that bad for inference speed @inkeep

[inkeep-ai-assistant[bot]]

14ms is actually quite good for inference speed. For context, the Beelink EQ13 with Google Coral EdgeTPU achieves 5-10ms inference speeds (1). Your 14ms with OpenVINO on a 15th gen Intel processor is well within acceptable ranges.

The inference speed metric is measured in milliseconds (2). To put this in perspective, other users report much higher inference times - for example, one user with an Nvidia A2000 GPU running YOLOv9 NAS at 640x640 reported 20ms inference speeds and found this acceptable (3). Another user experienced problematic speeds of 162ms with a Coral TPU that required troubleshooting (4).

Your 14ms inference speed means your detector can theoretically handle around 71 detections per second (1000ms ÷ 14ms), which is more than sufficient for most camera setups. The key metric to watch is whether you're experiencing skipped frames - if your system can keep up with the detection load without skipping frames, then 14ms is perfectly fine (3).

The documentation shows that OpenVINO runs best with tiny, small, or medium models (1), and your performance suggests the system is working well within its capabilities.

📚 Sources:

1. Recommended hardware
2. Inference Speed
3. inference speed problematic?
4. The coral is very slow. It always shows 162ms

---

Have another question? Just tag @inkeep.

[moogoos]

How do I use a second onyx detector? @inkeep

[inkeep-ai-assistant[bot]]

To use multiple ONNX detectors in Frigate, you can configure them in your detectors section (1). Here's how to set up two ONNX detectors:

detectors:
  onnx_0:
    type: onnx
  onnx_1:
    type: onnx

(1)

If you want to specify which GPU each detector should use, you can add the device parameter with the GPU index number (2):

detectors:
  onnx_0:
    type: onnx
    device: "0"
  onnx_1:
    type: onnx
    device: "1"

(2), (3)

When using multiple detectors, they will run in dedicated processes but pull from a common queue of detection requests from across all cameras (1). This allows you to spread the GPU load across multiple GPUs - for example, you could have ffmpeg and semantic search running on cuda:0, while the two detectors use different GPUs (2).

Note that multiple detectors cannot be mixed for object detection (for example, OpenVINO and Coral EdgeTPU cannot be used for object detection at the same time), but this doesn't affect using hardware for accelerating other tasks such as semantic search (4).

📚 Sources:

1. Supported Hardware
2. Define which gpu to use for onnx
3. [Config Support]:
4. Supported Hardware

---

Have another question? Just tag @inkeep.

[moogoos]

is there a benefit to running 2 openvino detectors? @inkeep

[inkeep-ai-assistant[bot]]

Yes, there can be benefits to running multiple OpenVINO detectors when using many cameras (1). The documentation specifically mentions this use case:

detectors:  
 ov_0:  
 type: openvino  
 device: GPU  
 ov_1:  
 type: openvino  
 device: GPU  

(1)

The benefit comes when one detector may not be enough to keep up with the detection load from multiple cameras (1). Multiple detectors can be defined assuming GPU resources are available (1).

However, there are some important considerations:

Performance Context: Based on your previous message showing 7ms inference speed for 320x320 and 14ms for 640x640 models, you likely have sufficient performance headroom with a single detector. One user reported that their system can theoretically handle up to 13 cameras at 5fps detection with similar performance (2).

Resource Requirements: Running multiple OpenVINO detectors requires sufficient GPU resources to be available (1). Each detector will consume additional GPU memory and processing power.

When It's Beneficial: Multiple detectors are most useful when you have many cameras and are experiencing detection bottlenecks - such as skipped frames or delayed detections due to the detection queue backing up.

Given your 14ms inference speeds, a single OpenVINO detector should handle multiple cameras efficiently. You'd typically only need multiple detectors if you're running a large number of cameras and notice performance issues with a single detector.

📚 Sources:

1. Supported Hardware
2. [Detector Support]: Openvino - Is it really so good or there is something i don't understand

---

Have another question? Just tag @inkeep.

[moogoos]

Can you configure multiple openvino detectors on a single GPU @inkeep

[inkeep-ai-assistant[bot]]

Yes, you can configure multiple OpenVINO detectors on a single GPU (1). The documentation shows an example configuration where multiple OpenVINO detectors both use the same GPU device:

detectors:  
 ov_0:  
 type: openvino  
 device: GPU  
 ov_1:  
 type: openvino  
 device: GPU  

(1)

This configuration allows you to run multiple detector processes on the same GPU when you have many cameras and one detector may not be enough to keep up (1). Multiple detectors can be defined assuming GPU resources are available (1).

When using multiple detectors, they will run in dedicated processes but pull from a common queue of detection requests from across all cameras (1). This means both detectors will share the workload from all your cameras while utilizing the same GPU hardware.

The key requirement is that your GPU has sufficient resources to handle multiple detector instances running simultaneously (1). Given your previous mention of 7-14ms inference speeds, your GPU should have adequate performance headroom to support multiple detectors if needed.

📚 Sources:

1. Supported Hardware

---

Have another question? Just tag @inkeep.

[moogoos]

Will you marry me @inkeep?

[inkeep-ai-assistant[bot]]

I'm sorry, but as an AI Assistant knowledgeable about Frigate, I only have the ability to answer questions about Frigate.

---

Have another question? Just tag @inkeep.

[moogoos]

This AI is awesome and so good.

[hawkeye217]

You can use it directly with the "Ask AI" button at https://docs.frigate.video as well.