Live api: add to context when outputaudiotranscription is enabled

[kaiseenn]

When modality is set to AUDIO, and outputaudiotranscription is enabled, one will notice that the outputaudiotranscription is output faster than the audio. However, if the model is interrupted, only the spoken part is added to context, and the rest of the output audio transcription is not added to context. This makes sense for an audio-only use case; if the user hasn't heard something, it shouldn't be added to context.

However, with outputaudiotranscription enabled, users will also become aware of something faster than they listen to it. Then, if a user attempts to interrupt the audio output but has seen the complete text transcription, the model will not be aware of the unspoken parts and will repeat information to the user.

This does not occur when modality is set to TEXT.

It would be nice to add a feature that allows us to enable adding to context based on transcription output.

[shmishra99]

Hi @kaiseenn ,

Could you please provide a minimal reproducible example so I can investigate the behavior? Thank You!!

[kaiseenn]

In AI studio, try the 2.5 live model, and ask it to tell you a 75 word story with response modality is AUDIO + TEXT. Then, before Gemini finishes speaking, ask it for the last word in the story. You will notice that Gemini will claim the last word in the story as approximately the last word it speaks aloud, even if you see more of the story as text.

Now switch modality to TEXT only, and ask it the same questions. It will now output the actual last word in the story