How to process text only mode?

[david-ng-hk]

I play around with your code in huggingface talk-to-gemini

I am using AsyncStreamHandler
I remove below code
const stream = await navigator.mediaDevices.getUserMedia({ audio: true }); stream.getTracks().forEach(track => peerConnection.addTrack(track, stream)); if (!audioContext || audioContext.state === 'closed') { audioContext = new AudioContext(); } if (source_input) { try { source_input.disconnect(); } catch (e) { console.warn("Error disconnecting previous input source:", e); } source_input = null; } source_input = audioContext.createMediaStreamSource(stream); analyser_input = audioContext.createAnalyser(); source_input.connect(analyser_input); analyser_input.fftSize = 64; dataArray_input = new Uint8Array(analyser_input.frequencyBinCount); updateAudioLevel();
from
https://huggingface.co/spaces/fastrtc/talk-to-gemini/blob/main/index.html

and found I can not send over the text.

do some research, and from
https://github.com/gradio-app/fastrtc/blob/main/docs/reference/stream_handlers.md?plain=1
it has audio and video receive() call back function, but there is no text call back function.

is there any text only sample with AsyncStreamHandler ？
thank you

[freddyaboulton]

Hi @david-ng-hk !
[
You can send any arbitrary data via the a post request. In the gemini example we use a route called /input_hook. js and python

However, I don't think gemini live api supports text input

[david-ng-hk]

HI @freddyaboulton
thank you for the reply, I check the code.
it seems the start_up() inside the AsyncStreamHandler never got trigger due to I did not do "peerConnection.addTrack()"
so after I call the /input_hook , the AsyncStreamHandler receive() or emit() inside the AsyncStreamHandler never got trigger.

gemini live api is support text only input, I got this example from their doc
await session.send_client_content(turns={"parts": [{"text": prompt}]})

[freddyaboulton]

Hi @david-ng-hk ! So is it working for you now?

[david-ng-hk]

Hi @freddyaboulton
does not work.
I try to create an dummy audio, and pass it to the peerConnection,
after /input_hook get the data,
I still not yet figure out how to use GeminiHandler to those the data.