Use ReplayGain (and Opus R128 Gain) tags for loudness normalization

[kepstin]

As far as I can tell, Music Assistant currently uses ffmpeg's loudnorm filter to do loudness normalization. I'm unclear when that calculation is done (on demand? on import?), but it's a relatively cpu-heavy filter to run, and while the loudnorm filter's one-pass support for dynamic normalization (dynamic range compression, effectively) might be useful for radio or podcast streams, the filter is not always appropriate for local music files. (Note that for determining the integrated loudness of local files, ffmpeg's ebur128 filter should run faster.)

But all of my music files contain pre-calculated "integrated" loudness adjustment information, stored either in the form of ReplayGain 2.0 tags or Opus R128 Gain tags. It would be great if Music Assistant could retrieve and use this precalculated loudness normalization information, rather than do its own loudness calculations. (Note that the ReplayGain 2.0 tags use a reference level of -18 LUFS, while the Opus R128 tags use a reference level of -23 LUFS.)

Ideally, the ability to select whether to use "track" or "album" normalization would also be provided - the "album" ReplayGain tags provide a single loudness adjustment for the album as a whole, while preserving dynamic differences between tracks within an album (this can be particularly important for concept albums, and ensures that tracks with gapless playback don't have loudness jumps when switching tracks).

[marcelveldt]

The loudnorm filter is not that heavy, at least other aspects of MA require more powerful hardware than that simple filter so I don't agree on that. Are you running it on a pi3 or even lower perhaps ?

Second of all most tags are just the simple replaygain tags to reduce the loudness of the entire track. Most often those calculations are based by a dumb "check the average loudness" algorithm instead of perreceived average loudness (EBU 128) causing issues with tracks with high dynamic range. It needs to be much, much smarter than that. I suggest you look into the loudnorm filter and the guy who wrote it and then realize that its miles better than just adjusting the gain of an entire track based on a single value. That is at least my opinion after comparing them both (we've had the old method that you propose here before btw).

We use loudnorm in 2 pass mode, which means that the first time you play something it runs in dynamic mode, basically collecting the measurement. It will not be as good at that point as its dynamically adjusting only with a limited look ahead. Now, once playback completes we store the measurement in the db (which is not a single value but a collection of values) and the very next time you play a track, it will then use those measured values to do the loudness leveling.

So I guess what I'm trying to say is that we implemented loudnorm for a reason; its way better than the common/dumb implementations of the replaygain tag. Maybe the Opus R128 tag is a bit better because that is at least the "average perceived loudness" EBU R128 calculation.

More background info about the loudnorm we choose to use in MA:
https://k.ylo.ph/2016/04/04/loudnorm.html

And a pointer to our implementation within the stream controller where you can also see what measurements values we store.
https://github.com/music-assistant/server/blob/main/music_assistant/server/controllers/streams.py#L798

[marcelveldt]

f64 is a bit overkill indeed yeah, lets not so that. The intermediate sample rate is the original sample rate but sometimes upsampled (in case of flow mode streams, everything is resampled to the same). Yes loudnorm in dynamic mode does an intermediate upsample to 192khz and then outputs to our desired sample rate at the end of the chain.

I'll implement some of these ideas to play around with in a beta

[kepstin]

I did a bit of testing, and I found out that the filter chain aresample=sample_rate=192K,alimiter=limit=-2dB:level=false:asc=true works well as a true peak limiter.

On the alimiter filter, the limit param can be adjusted to taste (I used -2dB to match the settings being used with loudnorm right now). Note that when doing true peak limiting (with upsampling), running after EQ/effects, with floating point audio, the limit could be set as high as 0 dBFS - but using −2 dBFS gives extra headroom which might prevent clipping in processing done on low quality playback devices or caused by lossy audio encoding (mp3).

Setting level=false is required so that it does not perform any gain adjustments (by default it applies a gain opposite the limit in order to bring the peaks back up to 0 dbFS).

The asc=true option enables a feature where when detecting multiple peaks in short succession, it averages out the adjustment a bit rather than repeatedly applying the filter attack/release (the loudnorm peak limiter has a similar feature).

[marcelveldt]

Nice! Thanks for testing that. So this true peak limiter needs to be implemented at the end of chain right ? So in stage 2 of our internal processing where the pcm stream will be encoded to the player's preferences.

Only thing I wonder is if we really should upsample all material internally to 192K - I assume that is to correctly detect the true peaks.

[kepstin]

Yeah - if you want the alimiter filter to accurately/precisely limit the "true peak", then you need to upsample the audio.

But I did some additional testing, and the specific alimiter filter I recommend with limit=-2dB works well enough on 44.1 kHz and 48 kHz audio to keep the true peak below 0 dBFS in almost all cases (resulting true peak values around -0.6 dBFS to -1.8 dBFS are typical), which should be good enough to prevent any distortion due to clipping.

Note that "almost all" is not "all". I can find some counter-examples, e.g. one track with a true peak of 2.4 dBFS that only gets true peak reduced to 0.9 dBFS when running alimiter at 44.1kHz. That said, this track was fairly loud at -8.40 LUFS, so if volume normalization is applied the peaks get brought down well below -2 dbFS and the peak limiter doesn't need to do anything.

[marcelveldt]

Great thanks for testing and for being such a valuable source of information once again. BTW: Good to see you here as well, next to Matter/Thread ! I've already started working on something to enhance the volume normalization logic. I'm currently at IFA Berlin on behalf of Home Assistant / Nabu Casa but I'll probably finish this tomorrow in the train back to home and release a beta to play with.

[GreenhouseAU]

So I can't really tell from the replies, but is the 'implemented' tag correct? (if so, how do I enable? :) )

All of my music has replaygain 2.0 tags, and I don't like the idea of listening to thousands of tracks before a 'loudness' value is assigned to a track, when you can do all the processing beforehand and have it stored in a platform-independent manner (ID3 or FLAC metadata).

Also, most of my listening is of albums, so all I need is one 'normalisation' value per album (i.e. album gain tag).

[OzGav]

The original request has been implemented. It works automatically. Look at the audio pipeline to see what volume normalisation is being applied during playback

[GreenhouseAU]

Ah, I have found the issue.

With Plex (what I'm using because local disk doesn't play nicely with my folder structure/album art), the replaygain tags musn't be passed through to MA.

Files on local disk work well with replaygain.

So it appears I can have album art or replaygain, but not both :P

[OzGav]

Don’t forget option c is to have both with an adjustment to your folder structure (not sure what the problem there is).

[galettof]

Question: if I add ReplayGain tags to my music files after they have already been played in Music Assistant, will MA pick up the new tags or will it keep using the normalization values it previously computed? If MA keeps using its own stored values, how can I clear them and force MA to read my ReplayGain tags? Thanks.

[kepstin]

Aha! The difference (z-y) is always 1.03 or 1.04. Aside from rounding, I am guessing this is due to the program I used to create the ReplayGain tags (rsgain) targeting -18 LUFS, while MA targets -17 LUFS.

Indeed, the ReplayGain 2.0 spec uses a reference level of -18 LUFS, and Music Assistant uses that as the basis for calculating the measured loudness of the track from the ReplayGain tags. Note that Music Assistant's default normalization to -17 LUFS is configurable per-player in the player advanced settings; you can change it to -18 to match ReplayGain levels if you prefer, or any other value you like to match other audio source levels if needed.

The method Music Assistant uses for measuring loudness only provides one decimal of precision; other tools may provide two decimal places of precision, so seeing a difference of <0.05 dB is pretty normal, and won't be audible.

[galettof]

Thanks for the explanation! The reason I've been looking into this is that on first pass, when the dynamic normalizer is used, I got the impression that MA may play different tracks on the same album at different loudness levels. This appears to be resolved by the second read when the loudness level is stored, and presumably would be resolved on first read if the existing loudness tags were passed correctly by Jellyfin. At that point -18 or -17 LUFS isn't a big deal for me, but good to know that the value can be configured.

[galettof]

Indeed, the ReplayGain tags are read correctly from local files, so it looks like Jellyfin is not passing that information.

[kepstin]

on first pass, when the dynamic normalizer is used, I got the impression that MA may play different tracks on the same album at different loudness levels.

The dynamic normalizer has to work from scratch on each track. It picks the initial adjustment based on just the first few seconds of the track, then adjusts up or down to try to get the loudness to the expected level as song dynamics change. This means that if you have a track which starts quietly, it will sound "louder" than the previous track (at least for a bit, until the normalization brings it down - which in many cases is done gradually so it doesn't disrupt the music too much)

[galettof]

Thanks, that makes sense.