[models] Add AudioFlamingo3 integration

[lashahub]

This PR adds support for AudioFlamingo3 (AF3) — NVIDIA’s open large audio language model capable of reasoning over speech, sounds, and music.

• Paper: Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
• Model Weights: nvidia/audio-flamingo-3 on Hugging Face
• Original Code: NVIDIA Audio Flamingo GitHub (branch: audio_flamingo_3)

It introduces the following components:

• AudioFlamingo3 model class
• AudioFlamingo3Processor for preprocessing text + audio
• Configuration, modeling, and processing utilities
• Example usage

With this integration, AF3 can be loaded directly from the Hugging Face Hub:

from transformers import AudioFlamingo3Processor, AudioFlamingo3

processor = AudioFlamingo3Processor.from_pretrained("nvidia/audio-flamingo-3")
model = AudioFlamingo3.from_pretrained("nvidia/audio-flamingo-3")

prompt = "What is happening in the audio?"
audio = "clap.wav"

input_ids, media, media_meta = processor(prompt, audio)
output_ids = model.generate(
    input_ids=input_ids,
    media=media,
    media_meta=media_meta,
    generation_config=model.default_generation_config,
)
print(processor.decode(output_ids))
# Example output: "A crowd is applauding and cheering."

[ebezzam]

Thanks @lashahub for your PR! This is very a exciting model to add to the Transformers library!

I see that you've taken inspiration from Llava, which makes sense as you combine modules of different modalities.

Most of my comments are about rearranging your modules so that they fit the Transformers convention, such that it will be more convenient for others to use and to test your new model. To that end, Llava's and this PR (of another audio model) might serve as a useful example to see what files will be added/modified.

Below are my suggested steps.

1. Refactoring / reorganizing current files according to Transformers convention

You can take inspiration from the above models for refactoring your configuration, modeling and processing files, specifically:

• Consolidating your configurations. From what I see you may need only one config like in Llava or two like in Dia for AudioFlamingo3Config and AudioFlamingo3EncoderConfig.
• Processor: you can take inspiration from Dia to group your feature extractor, text tokenizer, and audio tokenizer into a single component. Loading pre-trained feature extractors and LLMs can be directly handled by the processor without you (or the user) having to manually load the corresponding weights (like you've done below).

# Load components
fe = WhisperFeatureExtractor.from_pretrained("openai/whisper-large-v3")
tok = AutoTokenizer.from_pretrained(
    llm_dir,
    padding_side="right",
    use_fast=True,
    legacy=False,
)

To this end, you'll need to create a model conversion script (with something like this) so the configuration files are generated for the processor to know where to pull the relevant models.

• The modeling file will diminish quite significantly because we won't apply the audio/text tokenizer inside it but rather in the processor, and the configuration file will also handle pulling the relevant LLM config.

2. Conversion script

This will be needed to convert your model weights / configuration to one that is compatible with the one defined in the above files.

This script can also handle uploading the Transformer compatible model and its configuration to the Hugging Face Hub. You can again take inspiration from Llava and Dia.

3. Testing, documentation, etc

Once your model implementation is consistent with other models implemented in Transformers, there's a lot of boilerplate code we can reuse to make using model convenient and to apply various testing suites. For example, you can see the docs, src/transformers/models/auto, and tests/models/dia folders of the Dia PR on how to prepare / modify the relevant files.

Hope that helps and let me know if you have any questions!

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

[lashahub]

@ebezzam The tests were converting the batch to bf16 before generation, so I removed all those conversions, it’s working fine on my end now. I also replaced one of the audio files, even though the previous one would’ve worked fine.

[ebezzam]

run-slow: audioflamingo3

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ["models/audioflamingo3"]
quantizations: []

[github-actions]

CI Results

Workflow Run ⚙️

✅ No failing test specific to this PR 🎉 !

[ebezzam]

@lashahub we still need to keep the models in bf16 for the tests, otherwise they won't load properly (see here)

[ebezzam]

run-slow: audioflamingo3

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ["models/audioflamingo3"]
quantizations: []

[ebezzam]

@lashahub could you also add this training snippet: 8bf40fa

to the model page: https://huggingface.co/nvidia/audio-flamingo-3-hf

[github-actions]

CI Results

Workflow Run ⚙️

✅ No failing test specific to this PR 🎉 !

[ArthurZucker]

Kudos everyone and thanks @eustlb and @ebezzam very clean 😉

[ebezzam]

run-slow: audioflamingo3

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ["models/audioflamingo3"]
quantizations: []

[github-actions]

CI Results

Workflow Run ⚙️

✅ No failing test specific to this PR 🎉 !

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

[ebezzam]

Thanks @lashahub and @Sreyan88 for the great work, and @eustlb and @ArthurZucker for the feedback 🤗

Merging!

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, voxtral

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.