Support for Non-CUDA GPUs

[jmiguelv]

Currently, BVQA has hardcoded CUDA references in several places throughout the codebase. This limits the tool to CUDA devices only.

Would it make sense to use PyTorch's device detection to automatically identify available compute devices (CUDA, MPS, CPU) and send models to the appropriate device? This would enable support for multiple compute backends while maintaining CUDA support.

For example, instead of hardcoded CUDA logic, we could use:

import torch

device = (
    torch.accelerator.current_accelerator().type
    if torch.accelerator.is_available()
    else "cpu"
)

model = AutoModelForCausalLM.from_pretrained(...).to(device)

I have tested this approach in the moondream describer.

[geoffroy-noel-ddh]

Thanks that's a good point. We should definitely offer broader GPU support. The only difficulty here for me is the get hold of the correct device for development and testing.

[jmiguelv]

I'd be happy to help implement and test the device detection logic on a Mac. My suggestion to use PyTorch's device detection would somewhat alleviate the testing burden across different devices, though I understand your concern about ensuring reliability.

I can help implement the changes and test on MPS, and we could add clear documentation about which devices have been directly tested vs. which are theoretically supported through PyTorch's auto-detection.

[geoffroy-noel-ddh]

Any help is great to have, thank you very much for the offer! PyTorch is indeed a good way to do it. There are conditions in some of the describers to check if cuda is present so we could move that detection back to the core functions available to all describers. Note that some describers won't need it (e.g. Ollama), so it should remain optional.