I am training fastconformer_hybrid_transducer_ctc_bpe from stt_en_fastconformer_hybrid_large_pc for another language, wer keep to be 1 after 5epoch

[wen1q84]

Hello, I am training fastconformer_hybrid_transducer_ctc_bpe from stt_en_fastconformer_hybrid_large_pc for another language, config is almost default, but wer keep to be 1 after 5 epoch.
I didn't change default config, just start training script with different params.
training dataset: about 1500h
batch_size: 32
gradient_accumulation_steps: 32
my training script:

python ${NEMO_ROOT}/examples/asr/asr_hybrid_transducer_ctc/speech_to_text_hybrid_rnnt_ctc_bpe.py \
--config-path=../conf/fastconformer/hybrid_transducer_ctc/ \
--config-name=fastconformer_hybrid_transducer_ctc_bpe \
+init_from_nemo_model.asr_model.path=${PRETRAINED_DIR}/stt_en_fastconformer_hybrid_large_pc.nemo \
+init_from_nemo_model.asr_model.exclude="['decoder', 'joint']" \
model.train_ds.manifest_filepath=[xxx_train_dataset] \
model.train_ds.batch_size=32 \
model.validation_ds.manifest_filepath=[xxx_test_dataset] \
model.validation_ds.batch_size=128 \
model.test_ds.batch_size=128 \
model.tokenizer.dir=${TOKENIZER} \
model.tokenizer.type=bpe \
model.aux_ctc.ctc_loss_weight=0.3 \
trainer.devices=-1 \
trainer.accumulate_grad_batches=32 \
trainer.accelerator="gpu" \
trainer.strategy="ddp" \
trainer.max_epochs=100 \
trainer.log_every_n_steps=100 \
model.optim.name="adamw" \
model.optim.lr=0.001 \
model.optim.betas=[0.9,0.999] \
model.optim.weight_decay=0.0001 \
model.optim.sched.warmup_steps=2000

[wen1q84]

there are some plots.

[saeedzou]

Did you train a tokenizer for the new language?

[wen1q84]

Did you train a tokenizer for the new language?

Yes,I use this script to prepare tokenizer

python ${NEMO_ROOT}/scripts/tokenizers/process_asr_text_tokenizer.py \
        --manifest=xxx \
        --data_root=${TOKENIZER_DIR} \
        --vocab_size=1024 \
        --tokenizer=spe \
        --no_lower_case \
        --spe_type=unigram \
        --spe_character_coverage=1.0 \
        --log

[AudranBert]

Did you train a tokenizer for the new language?

Yes,I use this script to prepare tokenizer

python ${NEMO_ROOT}/scripts/tokenizers/process_asr_text_tokenizer.py \
        --manifest=xxx \
        --data_root=${TOKENIZER_DIR} \
        --vocab_size=1024 \
        --tokenizer=spe \
        --no_lower_case \
        --spe_type=unigram \
        --spe_character_coverage=1.0 \
        --log

Have you looked at the predictions? Is it improving or always kind of random?
Maybe you can try with a different lr, I have finetuned a hybrid model in french and my lr was 1e-5. I had a bigger batch_size but less gradient accumulation though

[wen1q84]

Did you train a tokenizer for the new language?

Yes,I use this script to prepare tokenizer

python ${NEMO_ROOT}/scripts/tokenizers/process_asr_text_tokenizer.py \
        --manifest=xxx \
        --data_root=${TOKENIZER_DIR} \
        --vocab_size=1024 \
        --tokenizer=spe \
        --no_lower_case \
        --spe_type=unigram \
        --spe_character_coverage=1.0 \
        --log

Have you looked at the predictions? Is it improving or always kind of random? Maybe you can try with a different lr, I have finetuned a hybrid model in french and my lr was 1e-5. I had a bigger batch_size but less gradient accumulation though

In the begin, the predictions are random. After 2600 steps about, the predictions become empty.
Thanks for the suggestion, I will try a different lr.

[AudranBert]

Did you train a tokenizer for the new language?

Yes,I use this script to prepare tokenizer

python ${NEMO_ROOT}/scripts/tokenizers/process_asr_text_tokenizer.py \
        --manifest=xxx \
        --data_root=${TOKENIZER_DIR} \
        --vocab_size=1024 \
        --tokenizer=spe \
        --no_lower_case \
        --spe_type=unigram \
        --spe_character_coverage=1.0 \
        --log

Have you looked at the predictions? Is it improving or always kind of random? Maybe you can try with a different lr, I have finetuned a hybrid model in french and my lr was 1e-5. I had a bigger batch_size but less gradient accumulation though

In the begin, the predictions are random. After 2600 steps about, the predictions become empty. Thanks for the suggestion, I will try a different lr.

How much steps do you have in one epoch? How much time does it last?

[wen1q84]

Did you train a tokenizer for the new language?

Yes,I use this script to prepare tokenizer

python ${NEMO_ROOT}/scripts/tokenizers/process_asr_text_tokenizer.py \
        --manifest=xxx \
        --data_root=${TOKENIZER_DIR} \
        --vocab_size=1024 \
        --tokenizer=spe \
        --no_lower_case \
        --spe_type=unigram \
        --spe_character_coverage=1.0 \
        --log

Have you looked at the predictions? Is it improving or always kind of random? Maybe you can try with a different lr, I have finetuned a hybrid model in french and my lr was 1e-5. I had a bigger batch_size but less gradient accumulation though

In the begin, the predictions are random. After 2600 steps about, the predictions become empty. Thanks for the suggestion, I will try a different lr.

How much steps do you have in one epoch? How much time does it last?

One epoch have about 1300 steps, training time is about 4.6h