Audio Normalization and Processing

78 views
Skip to first unread message

Nick Andre

unread,
Feb 26, 2021, 7:05:07 PM2/26/21
to wmfo-ops
Figured I'd fork this off to a separate discussion; this is an extraordinarily technical explanation so hope you brought your signal processing handbook. A gist of what's going on:
  1. rdimport by default sets a normalization setting. The audio all gets unwrapped into a WAV file; these files come in mastered to 0 dBFS (full volume), and then rdimport takes each file and adjusts the volume in the imported version to -12 dBFS.
  2. At some point in the past, I added a few dB of gain on the ASI card so that the audio coming out of Rivendell comes a little closer to the normalized point. DJs have a habit of running audio very low from my experience, and if you play audio quietly enough the final stage AGC won't boost the audio up to hit full scale on the output. For a crappy perceptual codec, you want to utilize the full dynamic range not feed it out to computers 12 dB below. For a bit of history, in the old days we were running an analog compressor on the audio stream and all the webstreams were coming in at -12dB, which we fixed when we switched to the Omnia and the result was that the webstream was very quiet and sounded bad. This is why I authored an extensive section in the DJ handbook indicating that you want to keep the levels on the output of PGM 1 close to the normal mark. I believe that people get "scared" when the line starts to turn a bit red so I emphasized that this is normal and no harm will come to any equipment as a result of a bit of red.  A little red periodically indicates you are near the correct level.
  3. However, the Rivendell audio files are stored on disk at 16 bit, which means that they lose 12 dB of dynamic range when stored. Probably not the end of the world, but not correct. Audio in the Livewire world is 24bit at 48 kHz, so the idea is that with a 24 bit signal you can safely run the audio with additional headroom (e.g. each bit gets you 6 dB of dynamic range; CD quality is 96 dB of dynamic range at 16 bits, 24 bits corresponds to 144 dB of dynamic range so you could maintain identical fidelity with the digital signal at -48 dB in a 24 bit system as CD quality).
  4. The correct way to do this would be to _not_ normalize in Rivendell, but rather to have the ASI card reduce the output volume to -12 dB when the audio is sent to the Livewire network. In this way, you don't discard down to 84 dB of dynamic range on disk.
However, we currently have an entire bushel (15 TB?) of songs imported into Rivendell and stored on disk at -12 dB that would each have to be manually re-normalized to 0 dBFS peak gain. One mistake I made at the beginning of importing the shows into Rivendell was to disable normalization on the shows. This means that the signal levels would clip since the rest of the settings (ASI card gain and fader setting) are optimized for Rivendell source material at -12 dB, resulting in distortion that was audible on the output signal independent of the broadcasting compressor.

So in short, it would be possible to fix this problem but it would require writing a script to modify every single file in Rivendell to re-normalize, as modifying the system settings to handle the correctly normalized audio files would result in audio coming into the final broadcast compressor at around -24 dB which in normal AGC configuration will exceed the correction ability (we want a DJ who fades a song to have it eventually get quieter, not fight them with the AGC all the way down to silence).

It's a project, to say the least. We'd have to have a script touch every single file and I assume it would take weeks; during the interim these files would be half way in between which wouldn't be a good state.

--Nick

---------- Forwarded message ---------
From: Frank Rossano <fr2...@comcast.net>
Date: Fri, Feb 26, 2021 at 2:36 PM
Subject: Re: Getting music remotely from Rivendell?
To: Nick Andre <ni...@wmfo.org>, Bob Donovan <bob.do...@gmail.com>
Cc: Andy Sayler <an...@wmfo.org>, Belinda <bel...@bubblesinthethinktank.com>, Derek Gerry <dge...@gmail.com>, Edward Beuchert <edw...@slidingrock.com>, Joann Keesey <jcke...@gmail.com>, Ray Charbonneau <rcha...@y42k.com>, Ronald LaRussa <Ronald....@tufts.edu>, Savoir Faire <squidp...@gmail.com>, WMFO Operations Director <o...@wmfo.org>


Any way we can stop the automatic wave normalizing going forward? 

Also, anyone notice a difference in sound quality between the local air signal 91.5 FM and the web stream signal on your live shows?  I'm in range locally so I listen to both air and web.

My show never sounds like the original wave file i encoded when it's airing live on the local air signal 91.5 FM. It's not a big difference but it's noticeable to me.  Anyone else?  Are the FM air signal EQ settings the same as the webstream EQ settings?

Frank
Rossano Radio
WMFO Community Member


------ Original message------
From: Nick Andre
Date: Fri, Feb 26, 2021 2:16 PM
To: Bob Donovan;
Cc: Andy Sayler;Belinda;Derek Gerry;Edward Beuchert;Frank Rossano;Joann Keesey;Ray Charbonneau;Ronald LaRussa;Savoir Faire;WMFO Operations Director;
Subject:Re: Getting music remotely from Rivendell?

Spotify explicitly bans the use of their library for broadcast purposes FYI. 

I’ve never entirely understood any of this licensing nonsense to be honest, it all seems thoroughly illogical 😂

I’m not entirely sure where the line would be:

1. Letting DJs remotely manipulate Rivendell 
2. Letting DJs mount a file share and process audio in their computer for broadcast use with the output file never touching their disk 
3. Letting DJs download a file for use with their show if they pinky swear not to share it

My understanding is that with Spotify these days there’s not a huge incentive to actually pirate music, literally the only reason people would even want these files is for making shows like this. 

Being realistic nobody actually keeps FLAC files around anymore on their local disk so I’d imagine approximately 100% of DJs are not following whatever arcane rules are “required” anyways (ie they probably do not have 70 filing cabinets of CDs ready to explode through the floor of Curtis hall). I would assume that our not knowing this for a fact is important to our capacity to pretend people are doing things correctly. 

Side note: we really screwed up with the automatic WAV file normalizing on Rivendell import — we lose two bits of resolution doing that.  It would be quite an endeavor to fix the issue going forward. That should not be the default; it’s easy to drop the levels 12 dB when they enter the livewire network and get converted from 16 to 24 bit on the ASI card. I complained some number of years ago about the “auto-trim” feature being on by default and that was fixed. The normalizing is sadly permanent unlike the auto-trim which I was able to undo.

—Nick

On Fri, Feb 26, 2021 at 11:00 AM Bob Donovan <bob.do...@gmail.com> wrote:
I can't think of a faster way to get WMFO closed permanently for illegal file sharing. There is no other way around it.

On Fri, Feb 26, 2021 at 12:10 PM Ray Charbonneau <rcha...@y42k.com> wrote:
Side note: this library of questionable legality :-) would have higher sound quality than YouTube conversions, which are 128 kbps copies at best. Better to capture a high quality stream from Spotify, assuming you pay and have that option available. 

Still illegal, though. I, of course, would never do either one <snicker>

Ray Charbonneau
Listen to Tardis Jukebox on WMFO!


On Fri, Feb 26, 2021 at 11:50 AM Frank Rossano <fr2...@comcast.net> wrote:
Makes sense Ed.  Instead of the idea (and hassle) from exporting from Rivendell,  a shared drive of songs that have already been ripped into Rivendell is a good idea.  We're WMFO Radio DJs who lost access to their library of music because of a worldwide pandemic that shutdown pretty much everything, including in person radio studios where the music DJs rely on to spin is located.  

I don't think anyone will make a big deal out of it, but always best to get legal advice beforehand imo.  Only students and community members affiliated with WMFO would be allowed to access the shared drive of ripped wmfo songs, with the sole purpose of airing them on their show,  if the idea Ed mentioned ever materializes.

Frank


------ Original message------
From: Edward Beuchert
Date: Fri, Feb 26, 2021 10:41 AM
To: WMFO Operations Director;
Cc: Belinda;Derek Gerry;nick Andre;Frank Rossano;Bob Donovan;Joann Keesey;Andy Sayler;Ray Charbonneau;Ronald LaRussa;Savoir Faire;
Subject:Re: Getting music remotely from Rivendell?

Excellent questions Jordan!

The PSAs you mentioned I posted on the Google drive were actually in their original MP3 distribution format as I had downloaded them from the Ad Council, PETA.org etc. web sites. I still have a task to *upload* the last PSA batch I posted on the drive *into* Rivendell, which I will do using the ssh and VNC utility as Nick Andre described.

Yes, you are absolutely right, it would be wonderful if there was a way for DJs to remotely access Rivendell and download songs from our WMFO music library onto their own computers and mix them into their shows! It's a topic that's been discussed before on [wmfo-ops], most recently on May 13, 2020 in an email thread called "Downloading Songs From Rivendell" started by Frank Rossano. Here is the most relevant quote in that discussion from Derek Gerry:

We looked into this several years ago and it was a huge copyright issue. Belinda may be helpful in untangling the laws in play here, but as I recall it broke down to this: The digital copy of a song on Rivendell is the only one that we have a license to use. If anything is copied onto another hard drive, which would need to happen to do what Frank is asking, that's piracy.

But beyond the legal issues Derek described, as I briefly mentioned during your very informative remote broadcast tutorial, "The grapes are sour anyway!" With my PSA account privileges, I can export tracks out from Rivendell in .WAV format and then transfer them to my computer over the internet, but the problem is that each individual track gets exported to a blank file name, normalized to - 17dB, without any meta-data. Sure you can fix that up afterwards typing or cutting and pasting from the dialog box text, but is that any more convenient than hijacking Youtube audio using a converter utility? Since it's all computer data in the end, somebody could no doubt write scripts to add the Rivendell meta-data programmatically through an API, but that's a project, and not the way Rivendell export works for us now (unless I missed an option to do that ;-o)...

I loved the pre-pandemic WMFO policy of DJs being allowed to bring in laptops with CD drives, and rip any of the music CDs around the station into their personal music libraries. Songs I obtained that way form a significant part of my own collection, especially in the world, electronic and rock genres. Conceivably we could implement something sort of like that "system" for DJs w

Frank Rossano

unread,
Feb 27, 2021, 12:31:17 AM2/27/21
to wmfo-ops
Yeah once you normalize, that’s it, you can’t undo it because it’s destructive editing, unless you hit undo right after the file is normalized.  It’s definitely a “huge” project for sure, to have to write a script to modify every file in Rivendell (15 TB) to re-normalize, and in the meantime while the job is in progress, files would be in between the task, which wouldn’t be good. 
 
Quote:  “We really screwed up with the automatic WAV file normalizing on Rivendell import — we lose two bits of resolution doing that.”
 
It’s such a big job to fix as you mention.  Is it worth fixing and disrupting the song file system for several weeks, or can we make adjustments and get by without re-normalizing?  Losing two bits of resolution is not good though the way it is now.
 
On another note,  when recording shows, make sure the output volume is at the correct loudness level. Some red is good.  Get the signal as loud as you can without distorting so the AGC (Automatic Gain Control) can do it’s job properly.  I’m going to print this for when I brush up on my audio books and in case I ever have to refer to it... : )  Thanks for the informative response.
 
Frank
--
You received this message because you are subscribed to the Google Groups "wmfo-ops" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wmfo-ops+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wmfo-ops/CABiK%3Ddab3GW-oDAh9r-Ypngx10DnNAqp97vqGsdwtQ8RR3Eqxg%40mail.gmail.com.

Nick Andre

unread,
Feb 27, 2021, 12:34:01 AM2/27/21
to wmfo...@googlegroups.com
We won’t get those bits back but we can at least fix the problem going forward. It wouldn’t be too terribly difficult we’d just need a strategy. I’m sure there are sufficient scriptable editors for doing a simple thing like normalizing properly. 

It might actually be an opportune time since we are not using a whole lot of Rivendell right now for automation. 

The other snafu is with the disks starting to fail I don’t really want to push them by rewriting the entire library until we get the bad drive replaced. 

—Nick

Ray Charbonneau

unread,
Feb 27, 2021, 8:09:08 AM2/27/21
to wmfo-ops
In an ideal world, do it. But in the real world, it seems like a lot of work for minimal improvement given AGC, the low quality of most FM listening situations, the fact that our webstream defaults to low quality, and the compressed nature of most modern music. Not worth it, especially given the current situation with the drives.

Semi-related: If you think maximum quality is the be-all, maybe give this a try: http://abx.digitalfeed.net/


Ray Charbonneau
Listen to Tardis Jukebox on WMFO!

To view this discussion on the web visit https://groups.google.com/d/msgid/wmfo-ops/CABiK%3DdYPWRV%3DTmo%2Bhk76FYtyK1RWwV3Hqg3mLbVGHPjdwprJwA%40mail.gmail.com.

Edward Beuchert

unread,
Feb 27, 2021, 12:18:57 PM2/27/21
to wmfo...@googlegroups.com
Just wanted to say Nick, outstanding job on that detailed technical description of the current process and the issues!

It corrected a number of misconceptions I had, and it's great to have such a write up like that now for future understanding too 8-O

Ha ha, another nice example of why [wmfo-ops] is the place for all the musical techie cool kids (and geezers too *8^!)...

-- Edward

Frank Rossano

unread,
Feb 27, 2021, 7:37:07 PM2/27/21
to wmfo...@googlegroups.com
I second that Ed, absolutely great job Nick on the detailed info.  We appreciate all you do for WMFO.
 
I’m thinking maybe you should hold off on re-normalizing until we get the bad drive replaced Nick.  But once we get it replaced, sounds like a good idea to correct everything going forward. 
 
Also shout out to Bob Donovan for all the help he contributes as well, and to Rachel and the exec board for all you guys do behind the scenes.  Kudos to all.
 
I’m happy to be part of this great team of people here at WMFO.  Hope you all have a great week!  Stay well & safe.

Ray Charbonneau

unread,
Mar 2, 2021, 4:15:14 PM3/2/21
to wmfo-ops
Sidetracking... you know what would improve the station's quality more than renormalizing? Trimming the dead air off of the ends of the tracks so there isn't a significant pause between songs when Rick Deckard is playing.

Ray Charbonneau
Listen to Tardis Jukebox on WMFO!

To view this discussion on the web visit https://groups.google.com/d/msgid/wmfo-ops/A560F684DA5E47ACB69B20BD0E867015%40ME3000HP.

Nick Andre

unread,
Mar 2, 2021, 7:56:39 PM3/2/21
to wmfo...@googlegroups.com
There’s an auto trim feature that has a proclivity to remove the last syllable of songs and break gapless playback. Some songs are specifically mastered to the end and others with giant gaps on the CD.

Frank Rossano

unread,
Mar 4, 2021, 12:23:02 AM3/4/21
to wmfo...@googlegroups.com, Nick Andre

Switch Audio File Converter Software

Convert, encode and automatically normalize audio quickly...

Nick Andre

unread,
Mar 5, 2021, 12:31:19 PM3/5/21
to Frank Rossano, wmfo-ops
There are plenty of options; we just have to be careful since we would be doing destructive editing of all these files. We don't have enough space to store an entire copy of the library at present.

Frank Rossano

unread,
Mar 5, 2021, 1:32:40 PM3/5/21
to Nick Andre, wmfo-ops
Is there a way you can set it up so just new audio files coming into Rivendell gets normalized correctly going forward, and not re-normalize what’s already in Rivendell?

Nick Andre

unread,
Mar 6, 2021, 3:28:40 PM3/6/21
to Frank Rossano, wmfo-ops
That can be done but it will result in old songs playing unacceptably quiet until they are renormalized and the process will take quite some time due to the number of songs. 

Bob Donovan

unread,
Mar 6, 2021, 3:33:16 PM3/6/21
to wmfo-ops, Frank Rossano
don't bother, it is not worth it.

To view this discussion on the web visit https://groups.google.com/d/msgid/wmfo-ops/CABiK%3DdYjpT3AxbjLbBuN9t4MaBWtsDyKz1-8aZ%2BA9pT1FgfCsw%40mail.gmail.com.


--
Bob Donovan
WMFO Community Member
Email: Bob.Do...@gmail.com

Frank Rossano

unread,
Mar 6, 2021, 11:15:23 PM3/6/21
to Nick Andre, wmfo-ops
Got it. I'm thinking maybe you should hold off as well. Big job, plus the destructive editing (re-normalizing) will probably degrade the sound quality a little. Thanks for responding back.  Hope you have a great week.

------ Original message------
From: Nick Andre
Date: Sat, Mar 6, 2021 3:28 PM
To: Frank Rossano;
Cc: wmfo-ops;
Subject:Re: [wmfo-ops] Audio Normalization and Processing

That can be done but it will result in old songs playing unacceptably quiet
until they are renormalized and the process will take quite some time due
to the number of songs.

On Fri, Mar 5, 2021 at 1:32 PM Frank Rossano  wrote:

> Is there a way you can set it up so just new audio files coming into
> Rivendell gets normalized correctly going forward, and not re-normalize
> what’s already in Rivendell?
>
> Frank
>
>
> *From:* Nick Andre
> *Sent:* Friday, March 5, 2021 12:31 PM
> *To:* Frank Rossano
> *Cc:* wmfo-ops
> *Subject:* Re: [wmfo-ops] Audio Normalization and Processing
>
> There are plenty of options; we just have to be careful since we would be
> doing destructive editing of all these files. We don't have enough space to
> store an entire copy of the library at present.
>
> On Thu, Mar 4, 2021 at 12:23 AM Frank Rossano 
> wrote:
>
>> Switch Audio File Converter Software Convert, encode and automatically
>> normalize audio quickly...
>>
>> https://www.nch.com.au/switch/index.html?kw=normalize%20mp3%20volume&gclid=EAIaIQobChMItLWjne6V7wIVhObjBx1oYwDMEAAYASAAEgJv7vD_BwE
>>
>>
>> Frank
>>
>>
>>
>> *From:* Nick Andre
>> *Sent:* Saturday, February 27, 2021 12:33 AM
>> *To:* wmfo...@googlegroups.com
>> *Subject:* Re: [wmfo-ops] Audio Normalization and Processing
>>
>> We won’t get those bits back but we can at least fix the problem going
>> forward. It wouldn’t be too terribly difficult we’d just need a strategy.
>> I’m sure there are sufficient scriptable editors for doing a simple thing
>> like normalizing properly.
>>
>> It might actually be an opportune time since we are not using a whole lot
>> of Rivendell right now for automation.
>>
>> The other snafu is with the disks starting to fail I don’t really want to
>> push them by rewriting the entire library until we get the bad drive
>> replaced.
>>
>> —Nick
>>
>> On Fri, Feb 26, 2021 at 9:31 PM Frank Rossano 
>> wrote:
>>
>>> Yeah once you normalize, that’s it, you can’t undo it because it’s
>>> destructive editing, unless you hit undo right after the file is
>>> normalized.  It’s definitely a “huge” project for sure, to have to write a
>>> script to modify every file in Rivendell (15 TB) to re-normalize, and in
>>> the meantime while the job is in progress, files would be in between the
>>> task, which wouldn’t be good.
>>>
>>> *Quote:  “We really screwed up with the automatic WAV file normalizing
>>> on Rivendell import — we lose two bits of resolution doing that”*
>>>
>>> It’s such a big job to fix as you mention.  Is it worth fixing and
>>> disrupting the song file system for several weeks, or can we make
>>> adjustments and get by without re-normalizing?  Losing two bits of
>>> resolution is not good though the way it is now.
>>>
>>> On another note,  when recording shows, make sure the output volume is
>>> at the correct loudness level. Some red is good.  Get the signal as loud as
>>> you can without distorting so the AGC (Automatic Gain Control) can do it’s
>>> job properly.  I’m going to print this for when I brush up on my audio
>>> books and in case I ever have to refer to it... : )  Thanks for the
>>> informative response.
>>>
>>> Frank
>>>
>>>
>>> *From:* Nick Andre
>>> *Sent:* Friday, February 26, 2021 7:04 PM
>>> *To:* wmfo-ops
>>> *Subject:* [wmfo-ops] Audio Normalization and Processing
>>>
>>> Figured I'd fork this off to a separate discussion; this is an
>>> extraordinarily technical explanation so hope you brought your signal
>>> processing handbook. A gist of what's going on:
>>>
>>>    1. rdimport by default sets a normalization setting. The audio all
>>>    gets unwrapped into a WAV file; these files come in mastered to 0 dBFS
>>>    (full volume), and then rdimport takes each file and adjusts the volume in
>>>    the imported version to -12 dBFS.
>>>    2. At some point in the past, I added a few dB of gain on the ASI
>>>    card so that the audio coming out of Rivendell comes a little closer to the
>>>    normalized point. DJs have a habit of running audio very low from my
>>>    experience, and if you play audio quietly enough the final stage AGC won't
>>>    boost the audio up to hit full scale on the output. For a crappy perceptual
>>>    codec, you want to utilize the full dynamic range not feed it out to
>>>    computers 12 dB below. For a bit of history, in the old days we were
>>>    running an analog compressor on the audio stream and all the webstreams
>>>    were coming in at -12dB, which we fixed when we switched to the Omnia and
>>>    the result was that the webstream was very quiet and sounded bad. Th

Nick Andre

unread,
Mar 7, 2021, 3:45:15 PM3/7/21
to Frank Rossano, wmfo-ops
It shouldn’t degrade the quality since the audio is uncompressed. Any loss in quality already happened. 

I can do a little research and a few tests with one of the files to see what’s feasible. It’s just one of those things where you have to triple check to make sure the script is working properly and there’s not any data loss possibilities. 

I have a similar script that I run to compress old archives for long term storage. 

—Nick
Reply all
Reply to author
Forward
0 new messages