Welcome to audio-diffusion-pytorch Discussions!

[flavioschneider]

👋 Welcome!

We’re using Discussions as a place to discuss experiments, write here if you find something interesting or have cool improvement ideas!

[yoyolicoris]

Hi @flavioschneider ,

Thanks for opening this repo and exploring different architectures for unconditional audio generation. These are wonderful results! I'm truly impressed. I had tried using DiffWave to do unconditional generation, but the results were far less coherent than yours. Seems like the model size matter the most, I guess.

Some personal thoughts:
Have you tried using the MCG sampling method from https://arxiv.org/abs/2206.00941 to do inpainting?
In my project, I found it can produce more coherent samples and requires fewer steps than RePaint. Maybe it can solve the abrupt change problem when doing infinite audio generation.
In addition, I'm thinking about making a controllable music synthesizer with the trained model. For example, given a musical score that we want to synthesize and a music transcription model, we can perform conditional sampling by incorporating the gradients of the loss between the transcribed score and the given score into the reverse sampling process. There's a high demand for these applications, and the audio quality your models can achieve makes them more likely to happen.

[flavioschneider]

Thanks @yoyololicon!

I think the abrupt change was due to a noise/coding problem, I changed a bit the code from the one proposed on the Repaint paper using the current ADPM2 sampler

audio-diffusion-pytorch/audio_diffusion_pytorch/diffusion.py

Lines 179 to 202 in d51d9a9

	def inpaint(
	self,
	source: Tensor,
	mask: Tensor,
	fn: Callable,
	sigmas: Tensor,
	num_steps: int,
	num_resamples: int,
	) -> Tensor:
	x = sigmas[0] * torch.randn_like(source)
	for i in range(num_steps - 1):
	# Noise source to current noise level
	source_noisy = source + sigmas[i] * torch.randn_like(source)
	for r in range(num_resamples):
	# Merge noisy source and current then denoise
	x = source_noisy * mask + x * ~mask
	x = self.step(x, fn=fn, sigma=sigmas[i], sigma_next=sigmas[i + 1]) # type: ignore # noqa
	# Renoise if not last resample step
	if r < num_resamples - 1:
	sigma = sqrt(sigmas[i] ** 2 - sigmas[i + 1] ** 2)
	x = x + sigma * torch.randn_like(x)
	return source * mask + x * ~mask

and this version seems to be working much better – even with a few resampling steps. I only tested it locally for now, I might link a few examples next week (haven't heard of MCG sampling, I'll have to look into it).

I'm spending most of my time now thinking and trying different ways to do conditioning, with that the possibilities are endless! Once it works properly, I'll merge the code or create a new repo for that as well :)

[yoyolicoris]

Awesome! I look forward to it.

[rlabs-oss]

Hi,
Do you know which parameter can be used to prevent repetitive generation of the same sample?
I thought this would be random seed related similar to latent diffusion in images, yet using the code below it still results in the same sample being generated from a trained model. what is rho=1.0 actually doing?
During training the output media varies greatly, yet during generation (code below) there is very little variation.

seed = 9916020
pytorch_lightning.utilities.seed.seed_everything(seed=seed, workers=False)
random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
np.random.seed(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
#a bad attempt at seeding to prevent repetive generation?
with torch.no_grad():
    samples = model.model.sample(
        noise=torch.randn((num_samples, 2, 2 ** length), device='cuda'),
        num_steps=num_steps,
        sigma_schedule=KarrasSchedule(
            sigma_min=0.001, 
            sigma_max=9.0,
            rho=10.0
        ),
        sampler=ADPM2Sampler(rho=1.0),
    )  
print(samples.shape)  

I hope you can help with this question.
Thank you! :)

[flavioschneider]

What is rho=1.0 actually doing?

In ADPM2Sampler the rho parameter can be used to change the curvature of the sampling process, a higher value will have more curvature (check this https://www.desmos.com/calculator/jbxjlqd9mb?lang=en with where rho is p) i.e. you will remove noise more aggressively towards the start and less towards the end. In my experiments, rho=1 seems to be optimal.

During training the output media varies greatly, yet during generation (code below) there is very little variation.

Note that during training the network is constantly changing, hence there will be more variation in general. Is the variation you get much worse than samples in the same batch during validation (the ones logged on wandb)? Changing the seed should not have a big impact since the noise will change every time you call the randn function, furthermore additional noise is injected during sampling.

[rlabs-oss]

Thank you for clarifying. The samples are improving. It was each step for a given num_steps for a batch that I was confused about. Each step in a batch representing the training variation and thus sound variation. The main issue is not to loose "something that sounds good" since the network will be updated and then it will not be possible to generate based on that particular network (unless a checkpoint is made manually). As an example if the training audio starts with synth sounds, followed by drums in the middle and vocals at the end - the checkpoint maybe made in the drum section, thus the model is unable to generate synth sounds. This is the variation I talk about.

[flavioschneider]

Yeah, this is also something I found in my experiments. Btw, do you mind sharing a few samples or a wandb link?

[rlabs-oss]

They are rather obviously recognizable as copyright content even if mangled in time. I couldn't figure out how to use the pytorch wav file dataset loader or how to configure the experiment .yaml file to load directly from .wav files or a folder of .wav files rather than youtube - if that were possible I could train on my own material and share some results. wanb, I dont use - all offline.
Is the small window of reproduction related to the model size? if a larger model is used and the same trick as used with stable diffusion where noise is spherically interpolated between two arrays (of noise) - the generated content may vary / shift ?