[talking avatar] update real time sample to support using cached loca…

…l video for idle status, to help customer save cost. (#2236)

* [TalkingAvatar] Add sample code for TTS talking avatar real-time API

* sample codes for batch avatar synthesis

* Address repository check failure

* update

* [Avatar] Update real time avatar sample code to support multi-lingual

* [avatar] update real time avatar chat sample to receive GPT response streamingly

* [Live Avatar] update chat sample to make some refinements

* [TTS Avatar] Update real-time sample to support 1. non-continuous recognition mode 2. a button to stop speaking 3. user can type query without speech

* [TTS Avatar] Update real time avatar sample to support auto-reconnect

* Don't reset message history when re-connecting

* [talking avatar] update real time sample to support using cached local video for idle status, to help save customer cost

* Update chat.html and README.md

* Update batch avatar sample to use mp4 as default format, to avoid defaultly showing slow speed with vp9

* A minor refinement

* Some refinement

* Some bug fixing

---------

Co-authored-by: Yulin Li <[email protected]>

samples/batch-avatar/python/synthesis.py

@@ -42,7 +42,7 @@ def submit_synthesis():
         'description': DESCRIPTION,
         "textType": "PlainText",
         'synthesisConfig': {
-            "voice": "en-US-AndrewNeural",
+            "voice": "en-US-JennyNeural",
         },
         # Replace with your custom voice name and deployment ID if you want to use custom voice.
         # Multiple voices are supported, the mixture of custom voices and platform voices is allowed.
@@ -59,10 +59,10 @@ def submit_synthesis():
             "customized": False, # set to True if you want to use customized avatar
             "talkingAvatarCharacter": "lisa",  # talking avatar character
             "talkingAvatarStyle": "graceful-sitting",  # talking avatar style, required for prebuilt avatar, optional for custom avatar
-            "videoFormat": "webm",  # mp4 or webm, webm is required for transparent background
-            "videoCodec": "vp9",  # hevc, h264 or vp9, vp9 is required for transparent background; default is hevc
+            "videoFormat": "mp4",  # mp4 or webm, webm is required for transparent background
+            "videoCodec": "h264",  # hevc, h264 or vp9, vp9 is required for transparent background; default is hevc
             "subtitleType": "soft_embedded",
-            "backgroundColor": "transparent",
+            "backgroundColor": "#FFFFFFFF", # background color in RGBA format, default is white; can be set to 'transparent' for transparent background
         }
     }
 

samples/js/browser/avatar/README.md

@@ -65,15 +65,16 @@ This sample demonstrates the chat scenario, with integration of Azure speech-to-
         * Avatar Style - The style of the avatar. You can update this value to use a different avatar style. This parameter is optional for custom avatar.
         * Custom Avatar - Check this if you are using a custom avatar.
         * Auto Reconnect - Check this if you want to enable auto reconnect. If this is checked, the avatar video stream is automatically reconnected once the connection is lost.
+        * Use Local Video for Idle - Check this if you want to use local video for idle part. If this is checked, the avatar video stream is replaced by local video when the avatar is idle. To use this feature, you need to prepare a local video file. Usually, you can record a video of the avatar doing idle action. [Here](https://ttspublic.blob.core.windows.net/sampledata/video/avatar/lisa-casual-sitting-idle.mp4) is a sample video for lisa-casual-sitting avatar idle status. You can download it and put it to `video/lisa-casual-sitting-idle.mp4` under the same folder of `chat.html`.
 
-* Step 3: Click `Open Video Connection` button to setup video connection with Azure TTS Talking Avatar service. If everything goes well, you should see a live video with an avatar being shown on the web page.
+* Step 3: Click `Open Avatar Session` button to setup video connection with Azure TTS Talking Avatar service. If everything goes well, you should see a live video with an avatar being shown on the web page.
 
 * Step 4: Click `Start Microphone` button to start microphone (make sure to allow the microphone access tip box popping up in the browser), and then you can start chatting with the avatar with speech. The chat history (the text of what you said, and the response text by the Azure OpenAI chat API) will be shown beside the avatar. The avatar will then speak out the response of the chat API.
 
 # Additional Tip(s)
 
 * If you want to enforce the avatar to stop speaking before the avatar finishes the utterance, you can click `Stop Speaking` button. This is useful when you want to interrupt the avatar speaking.
 
-* If you want to clear the chat history and start a new round of chat, you can click `Clear Chat History` button. And if you want to stop the avatar service, please click `Close Video Connection` button to close the connection with avatar service.
+* If you want to clear the chat history and start a new round of chat, you can click `Clear Chat History` button. And if you want to stop the avatar service, please click `Close Avatar Session` button to close the connection with avatar service.
 
 * If you want to type your query message instead of speaking, you can check the `Type Message` checkbox, and then type your query message in the text box showing up below the checkbox.

samples/js/browser/avatar/chat.html

@@ -78,14 +78,17 @@ <h2 style="background-color: white; width: 300px;">Avatar Configuration</h2>
   <div style="background-color: white; width: 200px;">
     <input type="checkbox" id="autoReconnectAvatar">Auto Reconnect</input><br />
   </div>
+  <div style="background-color: white; width: 200px;">
+    <input type="checkbox" id="useLocalVideoForIdle" onchange="window.updateLocalVideoForIdle()">Use Local Video for Idle</input><br />
+  </div>
   <br />
 </div>
 
-<button id="startSession" onclick="window.startSession()">Open Video Connection</button>
+<button id="startSession" onclick="window.startSession()">Open Avatar Session</button>
 <button id="microphone" onclick="window.microphone()" disabled>Start Microphone</button>
 <button id="stopSpeaking" onclick="stopSpeaking()" disabled>Stop Speaking</button>
 <button id="clearChatHistory" onclick="window.clearChatHistory()">Clear Chat History</button>
-<button id="stopSession" onclick="window.stopSession()" disabled>Close Video Connection</button>
+<button id="stopSession" onclick="window.stopSession()" disabled>Close Avatar Session</button>
 
 <div id="videoContainer" style="position: relative; width: 960px;">
   <div id="overlayArea" style="position: absolute;">
@@ -98,10 +101,15 @@ <h2 style="background-color: white; width: 300px;">Avatar Configuration</h2>
         background-color: transparent;
         overflow: hidden;" hidden></textarea>
   </div>
+  <div id="localVideo" hidden>
+    <video src="video/lisa-casual-sitting-idle.mp4" autoplay loop muted></video>
+  </div>
   <div id="remoteVideo"></div>
 </div>
 
-<input type="checkbox" id="showTypeMessage" onchange="window.updateTypeMessageBox()" disabled>Type Message</input><br />
+<div id="showTypeMessageCheckbox">
+  <input type="checkbox" id="showTypeMessage" onchange="window.updateTypeMessageBox()" disabled>Type Message</input><br />
+</div>
 <textarea id="userMessageBox" style="width: 960px; height: 40px" hidden></textarea><br/>
 </body>
 </html>