(Deprecated) Release of ADP notebook beta v0.2

[flavioschneider]

https://colab.research.google.com/gist/flavioschneider/d1f67b07ffcbf6fd09fdd27515ba3701/audio-diffusion-pytorch-v0-2.ipynb

[kjhenner]

Is there an experiment configuration in the training repo that corresponds to the 405M params model?

[kjhenner]

Great, thanks! Though I assume then that I'd be better off training with the new architecture if I'm training a new model? I'm coming from an NLP background, so navigating and understanding the different audio models is still a bit of a leap for me. :)

[flavioschneider]

This one works with the new architecture (didn't change much, just the API) https://github.com/archinetai/audio-diffusion-pytorch-trainer/blob/main/exp/base_l_youtube_0.yaml. Note that it's a larger model still (~700M params) but you can change the architecture config to be the same as the archived experiment and it should work.

[kjhenner]

Perfect. I actually started a run with the larger model last night. You can check the progress at https://wandb.ai/kjhenner/house/runs/z9obd0rp if you're interested!

[flavioschneider]

Amazing! Thanks for sharing.

[kjhenner]

The dataset is ~100 hours of music mostly around french house and nu disco genres.

[StevenSchrembeck]

Fascinating! Keep it up. I produce audio stories with voice actors and sound effects. I'm mulling over diffusion for two different use cases:

1. text to sound effect generation (sound design is a very labor intensive, manual process. this is a very attractive problem)
2. voice to voice transformation (being able to voice multiple characters as a single narrator is also a big time saver)

I'm going to try out this model with some custom data sets for both use cases. I see that commonvoice has 1,600,000 english sentences.

Am I wasting my time trying to construct sound effect datasets in the mere thousands (maybe 10s of thousands)? High level dataset construction advice for audio diffusion appreciated.

I would be happy to share my results and learnings here

[StevenSchrembeck]

I tried this once a few years ago with Wavenet. The output was very grainy. I'm hoping this generation of architectures can unlock the potential I see here for creators. Again, I really appreciate you creating this repo so that run of the mill AI tinkerers can put the models to work on new, cool problems

I couldn't even attempt it without your work so far

[flavioschneider]

Thanks, Steven!

You are definitely not wasting your time with constructing the dataset, the current audio models are trained on ~90h of music, but previously I was training on around 30h and still obtaining good results.
I think that the quality of the current architecture is very good for loudness normalized data, for unnormalized music there's still some work to do (probably not architectural but hyperparameter tuning). Text conditioning for audio generation is there, but largely untested and requires that you add a transformer to encode the text into meaningful embeddings. Voice to voice should also be possible by adding a conditioning channel (I'd expect this to work very well if the voice data is well aligned).

It would be great to have you experimenting as well!

[kjhenner]

@StevenSchrembeck You might also want to follow the progress of LAION's CLAP project if you're not already.

[StevenSchrembeck]

@StevenSchrembeck You might also want to follow the progress of LAION's CLAP project if you're not already.

Thanks for the tip! It's shocking how often someone else is already working on (or done with) a version of the tool you want to build.

[StevenSchrembeck]

Thanks, Steven!

You are definitely not wasting your time with constructing the dataset, the current audio models are trained on ~90h of music, but previously I was training on around 30h and still obtaining good results. I think that the quality of the current architecture is very good for loudness normalized data, for unnormalized music there's still some work to do (probably not architectural but hyperparameter tuning). Text conditioning for audio generation is there, but largely untested and requires that you add a transformer to encode the text into meaningful embeddings. Voice to voice should also be possible by adding a conditioning channel (I'd expect this to work very well if the voice data is well aligned).

It would be great to have you experimenting as well!

That's great to hear! I think I'm going to go for it (text -> audio SFX generation). There's multiple sub-problems here I think. I'm sure you're familiar with DAW workflows, but sound effects are built up from layers of simpler sounds (as well as effects)

I'm going to try to create these base sounds with my first pass. Treating the composition as a separate problem seems wise. (on the premise that "metal clank" is a simpler problem than "car crash followed by police sirens", which has many separate pieces).

Just creating the dataset is going to be a challenge! If it works, even decently well, it would be an order of magnitude shift in audio engineering for storytelling.

Thanks for the support. I'll be back with learnings and questions. Just need to get my hands dirty first I think

[twobob]

How does one specify the path of a waveset please?

[Pwntus]

Hi, I'm really impressed by this work and I'm excited about generative audio using these libraries. However this notebook doesn't seem to be updated or doesn't work anymore. I'm not very familiar with this so please forgive me for not providing much input.

[datacurse]

The model is gone. It is downloading from this link (https://huggingface.co/archinetai/audio-diffusion-pytorch/resolve/main/audio_1136_720k.pt), which contains nothing, since model was removed from hugging face. I was late to download it, wanted to ask the author if there is any change they uploading it back.

[flavioschneider]

Hey all. That model was an old unconditional model that wasn't working so well (didn't think people still used it). I might release a new text conditional model in some time when it's ready

[datacurse]

@flavioschneider oh i see now! Thank you so much for the work! I've emailed you, but you can ignore it, this explains everything!

[datacurse]

@flavioschneider also if you ever need any kind of help or if you're planning on including more people, just type me out! i think i know who could give you gpu racks, help with training, provide datasets etc. it would make developing of music ai much faster! (imho)

[iamisobe]

Hey all. That model was an old unconditional model that wasn't working so well (didn't think people still used it). I might release a new text conditional model in some time when it's ready

Any new model or updates for this? Would pay nicely for some help or info on how to train and use it

[markstent]

Im really trying to use diffusion to create music from my own training set....could you guide me step by step on how to do this?

[iamisobe]

id love to learn how to train it on my music and use this also

[fred-dev]

Any chance you would update the notebook to work with the latest version of ADP?