Feat: Implement Custom Pause Tags and Automatic Newline Pauses

[mylukin]

Description:

This PR introduces two significant enhancements to the TTS audio generation process:

1. Custom Pause Tag Support: Users can now insert pauses of specific durations directly within the input text using the format [pause:Xs], where X is the duration in seconds (e.g., [pause:0.5s], [pause:2s]).
2. Automatic Newline Pauses: A default pause of 0.5 seconds is automatically inserted after each newline character (\n) encountered in the input text, typically signifying paragraph breaks.

These features provide greater control over the rhythm and pacing of the generated speech.

Key Changes:

• api/src/services/text_processing/text_processor.py:
  • Modified smart_split to recognize and parse [pause:Xs] tags using regex (PAUSE_TAG_PATTERN).
  • Updated smart_split to yield tuples indicating whether a chunk is text (Optional[float] is None) or a pause (Optional[float] contains duration).
  • Modified get_sentence_info and smart_split's chunking logic to preserve trailing newline characters (\n) from the original text segments, allowing the service layer to detect them.
  • Improved handling of custom phoneme markers ([word](/ipa/)) during normalization and splitting, ensuring they are correctly restored before yielding text chunks.
  • Fixed logic errors in smart_split related to handling oversized sentences and clauses, ensuring correct chunking around token limits.
• api/src/services/tts_service.py:
  • Modified generate_audio_stream to consume the new output format from smart_split.
  • Added logic to generate silent audio chunks of the specified duration when a pause chunk is received.
  • Added logic to check for trailing newlines (\n) in yielded text chunks and insert an additional 0.5-second silent chunk if found.
  • Ensured accurate accumulation of the current_offset for timestamps, accounting for both generated speech and inserted silence.
  • Refactored _get_voices_path for more robust parsing of combined voices with weights (e.g., voice1(0.7)+voice2(0.3)) using regex, improved weight normalization handling based on settings, and ensured correct device placement for combined tensors.
  • Fixed NameError in generate_audio by removing the incorrect check for output_format.
  • Corrected async handling and normalization logic within generate_from_phonemes.
  • Improved error logging (using logger.exception) and added resource cleanup (writer.close()) in generate_audio_stream.
  • Removed duplicated/incorrect logic blocks.

How to Test:

1. Generate speech with inputs containing [pause:Xs] tags (e.g., "Hello [pause:1.5s] world."). Verify the pause duration in the output audio.
2. Generate speech with inputs containing newline characters (e.g., "First paragraph.\n\nSecond paragraph."). Verify the 0.5s pauses between paragraphs.
3. Test inputs combining both pause tags and newlines.
4. Test edge cases like pauses at the beginning/end of text or multiple consecutive newlines.
5. Test with various base and combined voices (including weighted combinations).
6. Test with inputs containing custom phoneme markers alongside pauses or newlines.

[tanhv90]

@mylukin Could you please help check case calling API dev/captioned_speech with stream = false?

2025-04-07 18:54:14 kokoro-tts-1  | 11:54:14 AM | ERROR    | audio:198 | Error converting audio stream to mp3: I/O operation on closed file.
2025-04-07 18:54:14 kokoro-tts-1  | 11:54:14 AM | WARNING  | development:360 | Invalid request: Failed to convert audio stream to mp3: I/O operation on closed file.
2025-04-07 18:54:14 kokoro-tts-1  | INFO:     172.24.0.1:46290 - "POST /dev/captioned_speech HTTP/1.1" 400 Bad Request

Sample request body:

{
        "model": "kokoro",
        "input": "Hello [pause:5s] World",
        "voice": "af_alloy",
        "speed": 1.0,
        "response_format": "mp3",
        "stream": false
}

[mylukin]

@mylukin Could you please help check case calling API dev/captioned_speech with stream = false?

2025-04-07 18:54:14 kokoro-tts-1  | 11:54:14 AM | ERROR    | audio:198 | Error converting audio stream to mp3: I/O operation on closed file.
2025-04-07 18:54:14 kokoro-tts-1  | 11:54:14 AM | WARNING  | development:360 | Invalid request: Failed to convert audio stream to mp3: I/O operation on closed file.
2025-04-07 18:54:14 kokoro-tts-1  | INFO:     172.24.0.1:46290 - "POST /dev/captioned_speech HTTP/1.1" 400 Bad Request

Sample request body:

{
        "model": "kokoro",
        "input": "Hello [pause:5s] World",
        "voice": "af_alloy",
        "speed": 1.0,
        "response_format": "mp3",
        "stream": false
}

This fork solves this problem: https://github.com/EasyMetaAu/Kokoro-FastAPI

[fireblade2534]

@mylukin Please fix the test issues

[RBEmerson970]

@fireb;lade2534 - Is there a way to contact you off-list?

[fireblade2534]

off list uhh I'm on the official kokoro discord under the same name

[RBEmerson970]

off list uhh I'm on the official kokoro discord under the same name

Gotit

[mylukin]

All of this code is generated by Gemini 2.5 Pro 😂

[fireblade2534]

I suspected as much based on your description

[fireblade2534]

Its worse now @mylukin

[fireblade2534]

@mylukin Still doesn't work

[fireblade2534]

Please read CONTRIBUTING.md

[fireblade2534]

@mylukin The tests still fail. Again please read CONTRIBUTING.md

[mylukin]

@mylukin The tests still fail. Again please read CONTRIBUTING.md

Finally fixed it, but still can't rely entirely on AI, I have to write the code myself! 😂

[fireblade2534]

Ok so um it doesn't work now I tried "Hello [pause:5s] world." with this code:

import base64
import json

import requests

text = """Hello [pause:5s] world."""


Type = "wav"

response = requests.post(
    "http://localhost:8880/v1/audio/speech",
    json={
        "model": "kokoro",
        "input": text,
        "voice": "af_heart+af_sky",
        "speed": 1.0,
        "response_format": Type,
        "stream": False,
    },
    stream=True,
)

with open(f"output.{Type}", "wb") as f:
    f.write(response.content)

It pronouces pause 5s

[mylukin]

Ok so um it doesn't work now I tried "Hello [pause:5s] world." with this code:

import base64
import json

import requests

text = """Hello [pause:5s] world."""


Type = "wav"

response = requests.post(
    "http://localhost:8880/v1/audio/speech",
    json={
        "model": "kokoro",
        "input": text,
        "voice": "af_heart+af_sky",
        "speed": 1.0,
        "response_format": Type,
        "stream": False,
    },
    stream=True,
)

with open(f"output.{Type}", "wb") as f:
    f.write(response.content)

It pronouces pause 5s

I fixed this.

curl -X 'POST' \
  'http://127.0.0.1:8880/v1/audio/speech' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "kokoro",
  "input": "Hello [pause:5s] world.",
  "voice": "af_heart+af_sky",
  "response_format": "wav",
  "speed": 1,
  "stream": false
}'
Warning: Binary output can mess up your terminal. Use "--output -" to tell curl to output it to your terminal anyway, or consider "--output <FILE>" to save to a file.

[Copilot]

Copilot reviewed 5 out of 6 changed files in this pull request and generated no comments.

Files not reviewed (1)

• run-tests.sh: Language not supported

Comments suppressed due to low confidence (2)

api/tests/test_text_processor.py:53

• The function name 'test_get_sentence_info_phenomoes' contains a spelling error. Consider renaming it to 'test_get_sentence_info_phonemes' for clarity.

def test_get_sentence_info_phenomoes():

api/src/services/text_processing/text_processor.py:93

• The parameter name 'custom_phenomes_list' appears to be misspelled. Consider renaming it to 'custom_phonemes_list' to maintain consistency and clarity.

def get_sentence_info(text: str, custom_phenomes_list: Dict[str, str]) -> List[Tuple[str, List[int], int]]: