[ Feature Request ] Support the ability to continuously stream a dynamic list of mp4 files for Stream

[BajaMexico]

I am trying to create a visual avatar that can interact with the user. The avatar is comprised of numerous short videos that are played one after another to look reasonably life-like. If an audio question is asked the avatar will nod its head and then respond with the verbal answer using a wav2lip generated video. Gradio can play videos back but the screen goes black at the end of a video or multiple videos - so dynamically introducing a response to the user looks very disjointed. I was hoping to use fastRTC web camera video stream to make it appear that the multiple dynamic video files are just one stream. Currently, when using modality "video" and mode "receive" the files can be played back using cv2 but it is not possible to maintain continuous stream without it ending and having to press "Start Streaming" again. Ideally, a thread could be created that could iterate through a dynamically changing file list and stream video to the display.

[freddyaboulton]

Hi @BajaMexico ! I think you need to use the AudioVideoStreamHandler here

[freddyaboulton]

Have you tried that^?

[BajaMexico]

Hi freddyaboulton,

I tried but my Python skills were not up to the task and I couldn't follow the documentation (user error on my part). I think your suggestion would be better than what I implemented. I used the WebRTC component in my Gradio app. I use a virtual camera to trigger an incoming frame and then I stream frames from various video files of my avatar. I can change the files and essentially create the illusion of an avatar. If you ask the avatar a question, I generate a new video with Wav2Lip and insert that as the return video stream. This solution is significantly better than using Gradio straight - so thank you for providing. Unfortunately, the WebRTC class only sends video frames so I have to play the audio independently (via embedded VLC) which creates some synch issues. I haven't given up on the AudioVideoStreamHandler. If you update the documentation, I think an example of AudioVideoStreamHandler running on localhost streaming files would be a very useful example. Also, its not clear to me what is trigger the Handler - is it Audio or is it Video? I assume its the video and the audio comes along for the ride? And why do you have to implement video_receive and video_emit? In my application, I think that I only need video_emit to send the frames to my client.

BTW, here is my current code with WebRTC

with gr.Blocks(title="Parrot") as app:

gr.HTML("<center><h1>Parrot</h1></center>")

with gr.Row():
    with gr.Column(scale=1):
        with gr.Row():
            languageDD = gr.Dropdown (choices=list(language_dict.keys()), value=default_language, label="Language",
                interactive=True)
            speakerDD = gr.Dropdown(choices=[], value=default_speaker, label="Speaker", interactive=False)
        with gr.Row():
            levelDD = gr.Dropdown(levels, label = "Level", interactive=True)
            conversationDD = gr.Dropdown(["Ongoing", "Single"], value = "Single", label = "Conversation",
                interactive=True)
        audioInput = gr.Audio(label = "Input", sources="microphone", type="filepath", format="mp3",
                              waveform_options=gr.WaveformOptions( waveform_color="#01C6FF",
                              waveform_progress_color="#0066B4", skip_length=2, show_recording_waveform=False,))
    with gr.Column(scale=2):
        with gr.Group():
            video = WebRTC (mode="send-receive", modality="video")
        video.stream (fn=playWaitingVideos, inputs=[video], outputs=[video])

with gr.Row():
    responseText = gr.Text(label = "Written Response")
    englishText = gr.Text(label = "English Translation")

with gr.Row():
    configureB = gr.Button("Configure", interactive = True)
    converseB = gr.Button("Converse", interactive = True)
    displayResponseB = gr.Button("Response", interactive = True)
    displayEnglishB = gr.Button("Translation", interactive = True)
    clearB = gr.ClearButton([audioInput, responseText, englishText])

    languageDD.change (get_speakers, inputs=[languageDD], outputs=speakerDD)
    configureB.click (fn = configureOutput, inputs = [speakerDD, languageDD, levelDD, conversationDD])
    converseB.click (fn = processUserQuery, inputs = [audioInput])
    displayResponseB.click (fn = displayResponse, inputs = displayResponseB, outputs = responseText)
    displayEnglishB.click (fn = displayEnglish, inputs = displayEnglishB, outputs = englishText)

if name == "main":
#app.launch(share=True)
app.launch(server_name="127.0.0.1", server_port=42000)