Timeline misalignment in PPG extraction for audio with silence

[taisa1]

I noticed that the PPG extractor sometimes doesn't work properly when there is silence before or after the audio section. I'm using the latest version installed via pip.

Does anyone have insights into what might be causing this issue?

Observations

The input audio waveform:

I processed this audio and got the following ppg output:

However, the PPG output does not align with the actual audio section, which starts from frame 85. It's not only about the beginning, but the whole thing seems out of sync on the timeline.
The Mel spectrogram aligns correctly, so it seems the issue might be with the model:

To investigate, I cut the audio to frame range [65, 293], then got this PPG output:

The output aligns with the input audio and seems more accurate than non-cut version.

Test Data

• Original Audio: https://drive.google.com/file/d/1jW9qkXABDvef39KWWZv8TL3lP7GOzLRm/view?usp=sharing
• Cut version: https://drive.google.com/file/d/1sp6tcLNUdXhj5pk1RiT9kWRoorINNluK/view?usp=sharing

Environment

• Python: 3.10.12
• ppgs: 0.0.8

[taisa1]

@maxrmorrison @CameronChurchwell
Could you take a look when you have a moment? Let me know if you need any further details.

[CameronChurchwell]

Thank you for bringing this to our attention.

This is an issue that we have observed as well. Clips with leading (and possibly trailing) silence are out-of-distribution for the model as the training clips are almost all trimmed close to the start and end of speech content.

It is possible that using a shorter chunk length during inference could mitigate this issue as I have observed that the chunking has improved stability overall with some audio examples. You can try this if you want by changing ppgs.CHUNK_LENGTH to something shorter like 100, maybe this will help.

It is also possible that the model could be re-trained with some kind of augmentation adding silence to the beginning of some clips, or even inserting clips containing only silence. I'm interested in trying this at some point, but I don't have time at the moment. You are, of course, welcome to try this yourself if the former suggestion about chunk lengths does not work.

[taisa1]

Thank you for identifying the cause and providing ideas! Both the cause and the approach seem reasonable to me.

In fact, I previously re-trained this model using Common Voice 6.1 as the dataset, and I generated the alignment data myself with Charsiu (since the original google drive link was broken). I checked the /ppgs/data/cache/commonvoice/ directory and I noticed that both the audio files and the .TextGrid files still contain silence.
The re-trained model seems to work correctly for the problematic audio, but has not been thoroughly investigated.

I would like to know at what stage the silence trimmed. Did it happen inside the library, or externally? I'm curious if re-trained model solves the problem.