How to fix missing phonemes?

[pvagner]

Hello,
I am working on a slovak voice together with a friend. He has recorded some 3630 high quality samples of his voice.

The training is running and now after some 1000 steps the voice sounds verry promising however it does not correctly pronounce some numbers such as
4, 40, 41, 45, 5, 50 and so on.
When writing these numbers as words, it's being pronounced correctly.

I guess some of the eSpeak phonemes are not taken into account.

When preprocessing it looks like this.

(.venv) [ai@training-pc python]$ python3 -m piper_train.preprocess \
  --language sk \
  --input-dir /home/ai/piper/dataset/ondro \
  --output-dir /home/ai/piper/training/ondro \
  --dataset-format ljspeech \
  --single-speaker \
  --sample-rate 22050
WARNING:preprocess:Missing id
INFO:preprocess:Single speaker dataset
INFO:preprocess:Wrote dataset config
INFO:preprocess:Processing 3630 utterance(s) with 16 worker(s)
WARNING:preprocess:Missing id
WARNING:preprocess:Missing ̝ (223)
WARNING:preprocess:Missing ̊ (72)
WARNING:preprocess:Missing 2 phoneme(s)

Looking to the espeak dictsource/sk_list I can see entries such as:

_3    t'@-*i
_4    St'i*i
_5    p'&t;
_13   t@-*'ina:st;

When comparing these with numbers written as words, I would change these to:

_3    tR'i
_4    St'iRi
_5    p'et;
_13   tR'ina:st;

I can then compile espeak sk and I will get an sk_dic file.
How do I update the piper_phonemize with these tweaks?
The existing slovak voice named lili does not exhibit this behaviour. Is this the proper way on how to address such an issue?

Also when I determine the correct solution to my issue, can I continue training with the fixed data or do I have to start from scratch?

Please note there are more occurences of these phonemes I have just posted these sk_list entries as an example.

Greetings

Peter

[pvagner]

Dear @synesthesiam can you please try to help me with this?

[pvagner]

Looking more into the piper-phonemize I can see all the languages except of ukrainian are trained with espeak-ng as the phonemizer.
By replacing sk_dic I haven't yet got better pronounciation of 4, 44, 34 and similar numbers.
In the text transcription of the recordings I have no numbers. Numbers are written as words such as [štyridsaťštyri] for 44 and similar.
I'd much appreciate all the hints you can give to me. I think I am really missing something.

[pvagner]

Okay, I think I have finally figured out why I am getting different results than you with the Lili slovak voice.
The source text I'm working with has no numbers. Similar than the original english LJ Speech corpus all the numbers are normalized to their text representation e.g. 1 becomes jeden in slovak and so on.
eSpeak-ng sk_list prints out different phonemes for number 4 as its text equivalent word štyri. And since the data set I'm working with does not include recording of that different sound it's not trained.
In order to improve the situation I do have some local changes for the sk_list file here which I intent to submit to espeak-ng anyways as it contains other improvements to slovak language not just changes to number pronounciation.
When using piper infer and piper with exported onnx model I have to have updated the corresponding espeak-ng-data with my changes, otherwise the improvements are not audible. Thus distributing this as a piper voice once the training finishes is problematic as I need to first make piper TTS updated with the proposed espeak-ng-data changes first.
I've a link to the sample of the voice as it sounds while the training is in progress.
I'd really appreciate any kind of feetback mainly discussing the fact if such a model can be distributed with the existing piper voices once my training is finished and what other requirements there are.

Greetings

Peter

[isolveit-aps]

Wow, Peter,
I just stumbled upon your post, and I have the exact same issue with faroese. Did you solve this? And could you explain the workaround that you got working? It seems to me that if I just could get my onnx to use the phonemes from espeak-ng, then the tts model would be quite perfect! But I don't know how to make the inference use the espeak-ng variant that I added to espeak-ng, to implement faroese language. Any clues would be highly appreciated!

Here was my original issue that I hoped someone would reply to earlier this year: #536

Thanks,
Andras

[pvagner]

Hello @isolveit-aps
I haven't progressed further with this however my list of changes to espeak sk_list file is changing all the time as I am adding more exceptions and fixes into it.
Piper has a dependency on espeak-phonemize which includes espeak-ng along with it's espeak-data-ng folder. I simply generate my sk_dic file with my system wide installation of espeak-ng and simply place the sk_dic file where espeak-phonemize is expecting it and while this is not a solution for end users it allows me to continue training.

Once I'll manage to get it polished I'll consider creating relevant pull requests.

Thanks and greetings

Peter

[isolveit-aps]

Hi Peter @pvagner,
Thank you for this response. It makes sense, however, I still struggle getting the number to speak any numbers correctly. Actually the rules of the Faroese espeak-ng variant, is that we say numbers not like in English (from left to right), but like like in German, Danish, etc. where 1s are spoken before the 10s - e.g. (transliterated): 21 = one-and-twenty (German: ein­-und-­zwanzig, Faroese: ein-og-tjúgu), and my faroese espeak-ng does this correctly, however, when I train Piper TTS voice it keeps saying it like in english form, twenty-and-one instead of one-and-twenty, etc.
Would this be because I need to cover all spoken numbers in the training dataset, for it to even consider how to learn from the phonemizer how it should say these numbers? Or does the exported Piper TTS .onnx model still rely on the Piper phonemizer, when doing "inference" for what to speak, and that phonemizer doesn't have the Faroese rules built-in? I wonder...

Anyways, I will try with a dataset that has more numbers coverage, so that the model will train those spoken numbers, hopefully learning these by triggering the phonemization rules for Faroese numbers. I will also try copying the fo_dic file to my espeak-phonemizer, to see if that helps me with this problem, also. Thanks again for your response - I will keep on trying :)

[EvanMason489]

Dear All

I don't know if this is useful or not, but when I started searching for TTS systems, I saw a very few metadata.csv files that had three columns. One was the names of the .wav file, the next was the text; just like normal. The third column was expanded, with numbers expanded into their word forms.
e.g.
123 | There were 12 boys and 37 girls on the boat. | There were twelve boys and thirty seven girls on the boat.

I have tried to find this again and not got there yet.

[EvanMason489]

I can't find it. Maybe spelling out the numbers in the training metadata.csv will fix it, but the fact that it works partly suggests maybe the answer is something to do with espeak-ng and the preprocessing step using piper-phonemize.