Question about embeddings

[xxchauncey]

Hi,

Thank you for your great work! I have some questions about those head layers.:

As image shows in the paper, I notice there is a "shared" LLM embeddings in the 1st. stage. However, in the code you released I found that for the LLM a lm_head is re-defined while for audio part a audio_feature_head is defined, and they seem to have no relation but only the same shape.

I doubt how can we guarantee the CTC loss help the audio modality to align to LLM text modality? In my view, I suppose that audio input should be aligned to the text input embedding space of LLM but not output embedding space so that LLM can easily "recognize" the audio input to some extend.

Hope to know your thoughts!

[xiaozhang521]

Thanks for noticing the detail about the definition of embedding. Actually, the audio_feature_head is a copy of the LLM embeddings.

Another question is: why not just use one head? Well, we found that about 30% of embeddings are activated during CTC training. Therefore, our latest code uses only the English embeddings (about 40K) instead of the full 120K. This can further improve training speed, but we have not implemented it in this release version. Sorry for the confusion.

[xxchauncey]

oh I see. BTW, how did you calculate the text and audio embedding similarity, I notice that you did an average pooling operation, could you provide more details? Thanks!