Skip to content

Improving TTS Project #1078

@kevmo314

Description

@kevmo314

This is going to be a project to overhaul how TTS works in RealtimeChat. Today, it works by attempting to keep the main activity alive while the active channel is streaming. This comes with some complications, namely:

  • It's hard to keep that activity alive
  • It's battery intensive since the activity runs other things
  • It's hard to ensure message parity since the message history fetches things that should not be read
  • TTS logic is intermingled with other logic
  • It's not so easy to use native APIs to customize TTS voices as we are reliant on third-party packages, so we run into issues like this: [tts] tts does not work when app is in the background on ios #388

This project will have four components:

  1. The iOS TTS engine
  2. The Android TTS engine
  3. The TTS circular buffer
  4. The TTS isolate

The Android and iOS TTS engines will have the same Dart APIs following:

class Engine:
  getLanguages(): string[]
  getVoices(languageId: string): string[]
  speak({ text: string, languageId: string, voiceId: string }): Promise<void>
  stop() // immediately stop speaking
### Milestone 1: Native APIs
- [ ] https://github.com/muxable/rtchat/issues/1080
- [ ] https://github.com/muxable/rtchat/issues/1081
- [ ] https://github.com/muxable/rtchat/issues/1083
- [ ] https://github.com/muxable/rtchat/issues/1086
- [ ] https://github.com/muxable/rtchat/issues/1087
- [ ] https://github.com/muxable/rtchat/issues/1088
- [ ] https://github.com/muxable/rtchat/issues/1092
- [ ] https://github.com/muxable/rtchat/issues/1093
- [ ] https://github.com/muxable/rtchat/issues/1097
### Milestone 2: TTS circular buffer
- [ ] https://github.com/muxable/rtchat/issues/1098
- [ ] https://github.com/muxable/rtchat/issues/1099
- [ ] https://github.com/muxable/rtchat/issues/1100
- [ ] https://github.com/muxable/rtchat/issues/1101
- [ ] https://github.com/muxable/rtchat/issues/1102
### Milestone 4: Running in an isolate
- [ ] https://github.com/muxable/rtchat/issues/1103
- [ ] https://github.com/muxable/rtchat/issues/1104
- [ ] https://github.com/muxable/rtchat/issues/1105
- [ ] https://github.com/muxable/rtchat/issues/1106
- [ ] https://github.com/muxable/rtchat/issues/1151
- [ ] https://github.com/muxable/rtchat/issues/1152
### Milestone 5: Migrating the new TTS system
- [ ] https://github.com/muxable/rtchat/issues/1153
- [ ] https://github.com/muxable/rtchat/issues/1154
### Milestone 6: Testing and cleanup
- [ ] https://github.com/muxable/rtchat/issues/1189
- [ ] Ship a new release with the new TTS engine
- [ ] https://github.com/muxable/rtchat/issues/1166
- [ ] https://github.com/muxable/rtchat/issues/1165
- [ ] https://github.com/muxable/rtchat/issues/1183
- [ ] https://github.com/muxable/rtchat/issues/1184
- [ ] https://github.com/muxable/rtchat/issues/1185
- [ ] https://github.com/muxable/rtchat/issues/1187
- [ ] https://github.com/muxable/rtchat/issues/1188
- [ ] https://github.com/muxable/rtchat/issues/1186

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions