Real-Time Speech-to-Text Translation Support

[hu-ke]

Description of the feature request:

Instead of waiting for a turn of speech to complete (VAD mode), would it be possible to stream the generated results in real-time?

What problem are you trying to solve with this feature?

Suppose I am currently in a Japanese interview, but my Japanese skills are not very strong. I would like to build a app with the Gemini Multimodal API to assist me with real-time speech-to-text translation.

Any other information you'd like to share?

No response

[ViaAnthroposBenevolentia]

Currently, the easiest (and free) way to do this:

1. Get a free ($200 credit) API from Deepgram;
2. Establish a web socket connection to wss://api.deepgram.com/v1/listen;
3. Send the base64 audio from Gemini to the web socket;
4. Get real-time transcript from Deepgram.

[OptionIA]

or for free, but the price is the latency, make a function calling with 2 parameters Input: output, and add in the model instruction that the model in all of their ressponses must use the function call. then, add a system that get it and print it

[OptionIA]

well, they early added the native function, but there is no documentation aviable (or easy to found)