Prompt Generator for ACE-Step

[laitselec]

TLDR: i made a prototype prompt generator for ACE-Step which take in a song and output some tags and lyrics
recently i trained a music understanding model(audio-text-to-text) called MuFun and try to find some applications. in T2I i notice there're tools that take in a picture and output prompt which can probably generate it back, so i wonder if it's possible to make one for a T2M model, then i collect some synthesis data(that is, generated by ACE-Step) and made a finetune. looking forward to feedbacks
model weights: https://huggingface.co/Yi3852/MuFun-ACEStep
demo: http://47.121.209.64/mufun_demo_acestep
(this is run on my pc so may be very slow, maybe i could make some quant version later)
(update: using bitsandbytes 4bit quant to load model now, generate in 30s for a song)
(update: gradio demo code is at https://github.com/laitselec/MuFun/blob/main/demo/mufun_acestep/gr_app.py
the synthesis data is at https://huggingface.co/datasets/Yi3852/ACEStep-Songs which has 21k samples)

[pkrasicki]

This is really cool, thanks for sharing! I've tested it a little and I'm very impressed with how good it is at detecting lyrics. It does often make small mistakes, but at least it seems to make them consistently throughout the song. Lyrics websites also have errors and sometimes this model interprets some fragment of the song more correctly than what is listed online. It even helped me with interpreting a song for which I couldn't find lyrics anywhere. The model doesn't do that well with non-English languages though and it doesn't seem to use lyrics tags like [verse], [chorus], etc.

When it comes to describing the song, I don't think it does that well. It feels very basic. It outputs only a few tags, including the genre. It likes to use the word Exurberant a lot, which I think might be misspelled? It doesn't always describe the feeling of the music much or the instruments used. It might mention that there are vocals, but doesn't say if they are male or female (in my samples it's easy to tell) and doesn't say anything about them. And just like ACE-Step, it's not good at recognizing electronic music genres. I haven't actually tried using the generated prompts, though.

Still, it's a pretty cool project and I'm looking forward to future improvements!

[laitselec]

This is really cool, thanks for sharing! I've tested it a little and I'm very impressed with how good it is at detecting lyrics. It does often make small mistakes, but at least it seems to make them consistently throughout the song. Lyrics websites also have errors and sometimes this model interprets some fragment of the song more correctly than what is listed online. It even helped me with interpreting a song for which I couldn't find lyrics anywhere. The model doesn't do that well with non-English languages though and it doesn't seem to use lyrics tags like [verse], [chorus], etc.

When it comes to describing the song, I don't think it does that well. It feels very basic. It outputs only a few tags, including the genre. It likes to use the word Exurberant a lot, which I think might be misspelled? It doesn't always describe the feeling of the music much or the instruments used. It might mention that there are vocals, but doesn't say if they are male or female (in my samples it's easy to tell) and doesn't say anything about them. And just like ACE-Step, it's not good at recognizing electronic music genres. I haven't actually tried using the generated prompts, though.

Still, it's a pretty cool project and I'm looking forward to future improvements!

hi, thanks for sharing your experiences! i think the finetuning data could explain some phenomenon. for example, they're mainly in en and zh, that's why it may fail on other languages. the dataset comprise of two parts: synthesis songs generated by ACE-Step(prompt tags randomly selected from a pool, lyrics generated by LLMs with structural tags like [verse], [chorus]) and real-world songs(limited tags, lyrics without structural tags) . i trained the first version of this model(hf repo commit 192296e) using only synthesis data (around 20k samples) and found that lyrics extracted are somewhat overfit(since the distribution of lyrics generated by LLMs is quite distinct from real-world ones), so later on i decided to add more real-world data. in the latest version(hf model repo commit 5652286) i scale the finetuning data to around 240k songs(most are real-world). maybe in the future i will find some ways to train this model to extract more meaningful tags

[pkrasicki]

Thanks for explaining! Could something like Qwen Omni be used to provide some tags for the training data? I think that's what ACE-Step authors have used. I'm not sure how well it worked, since ACE-Step can be pretty random (depends on a genre), but maybe it would help.

i trained the first version of this model(hf repo commit 192296e) using only synthesis data (around 20k samples) and found that lyrics extracted are somewhat overfit(since the distribution of lyrics generated by LLMs is quite distinct from real-world ones)

Would it be possible to provide human written lyrics to an LLM to improve what it generates?

I'm also curious about LoRAs. As far as I understand, you don't need as much data to train one compared to finetuning. So if it was possible to train a LoRA for your model, maybe we could do that with a much smaller, manually labeled dataset. Could that improve its responses?

I don't know anything about training models, so I don't know if any of this is helpful.