[Proposal] Contribute to nanoALM – Add Audio Modality Support

[haidog-yaqub]

Hi nanoVLM team,

Thank you for your excellent work on nanoVLM – it's a very impressive project.

I’m currently working on audio-language models, and one thing I’ve noticed is the lack of simple, well-structured pipelines for building such models. I believe nanoALM, with its clean and modular design, could serve as a strong foundation for supporting the audio modality in a lightweight and extensible way.

I would love to contribute by extending the framework to include audio modality support, leveraging the existing strengths of the nanoVLM architecture.

Please let me know if you'd be open to this contribution — or potentially collaborating on a new repository focused on lightweight audio-language modeling.

[lusxvr]

Exciting! This would definitely be an interesting addition, we are very interested in so-called Omni-Models, that can take multiple different modalities as in- and output. Do you have a specific idea in mind on how to integrate Audio into nanoVLM? I would suggest we brainstorm and plan a bit before jumping into the code.

[tsdocode]

Interested

[carankt]

I have an initial implementation of the nanoVLM adapted for the audio domain. I was able to use the CLAP model as the feature extractor and make it work. Would love to be a part of the discussion and brainstorm.

[aceliuchanghong]

Interested +1

[leo1oel]

Interesting

[WangHaoyuuu]

Interesting