Bittorrent of Open Source Bridge 2009 conference audio with Ed's changes
Igal Koshevoy
the raw audio files and gave to Reid on a DVD (2.63GB):
http://opensourcebridge.org/torrents/Open_Source_Bridge_2009_conference_audio,_znmeb_v1.torrent
I had to rename two files because they contained invalid characters,
they're now called:
1.
St_Johns_204_Thu_20090618/05_Paul_Frields-M_is_for_Manual-_Creating_Documentation_for_your_Project.mp3
2.
Marquam_119_Wed_20090617/02_J-P_Voilleque,_Paula_Holm_Jensen-The_Scylla_and_Charybdis_of_Open_Source_Legalese.mp3
This new torrent contains 4 fewer files and 100MB less data than the
original raw audio files, but I don't know which were removed. The
original raws (2.73GB) are at:
http://opensourcebridge.org/torrents/Open_Source_Bridge_2009_conference_audio,_raw.torrent
-igal
John Prohodsky
I would like to start post-processing the audio files, but what would I
do with an audio file after the post processing is done?
I can commit processing 10 audio files a week. If other people help with
post processing, how will we know who does what?
John
Reid Beels
> Hi all,
>
> I would like to start post-processing the audio files, but what would I
> do with an audio file after the post processing is done?
We'll be working at the saturday work session this week to get out site set up to start pushing out podcasts. Once that's up, we'll upload audio and schedule it to go out.
> I can commit processing 10 audio files a week. If other people help with
> post processing, how will we know who does what?
I'm not sure how much post-processing will really be required here, beyond some trimming of whitespace and possibly some level normalization. Ideally, I'd like to record some kind of 30ish second boilerplate intro to stick on the front of all of the audio we push out, telling people about the 2010 event, but that would need to be written and produced.
Reid
M. Edward (Ed) Borasky
them, make sure the tags are correct, trim the dead air off the back.
I *think* I already normalized them.
--
M. Edward (Ed) Borasky
borasky-research.net/m-edward-ed-borasky/
"A mathematician is a device for turning coffee into theorems." ~ Paul Erd?s
Igal Koshevoy
> If other people help with post processing, how will we know who does what?
>
to record who is working on what file.
-igal
Ed Finkler
I am grabbing the BT now, and would like to process stuff as needed.
--
Ed Finkler
Audrey Eschright
On Jun 6, 2010, at 4:11 PM, Ed Finkler wrote:
> Has work been done on this?
>
> I am grabbing the BT now, and would like to process stuff as needed.
If it has, I don't see a page on the wiki or anything to document what happened. So I started one: http://opensourcebridge.org/2009/wiki/Session_Audio_Recordings
If there's other info from work in progress elsewhere, let me know and I'll merge it in. Is any other information needed to get started? I don't know if "where to post the files when they're done" has been sorted out yet, but people can hold onto them for now as long as we have a record of what's done.
Audrey
Selena Deckelmann
Hi!
On Jun 9, 2010, at 6:21 PM, Audrey Eschright <spin...@gmail.com>
wrote:
Thanks, Audrey!
OSUOSL has offered to host files. I'll circle back with Lance tonight
and set up SSH keys for those that have processed audio files to share.
-Selena
Igal Koshevoy
links and information.
Once we get a bit of a breather, I'd like to do a couple meetups to
figure out how to process this data because there's lots of good
content there and this would be excellent practice for the 2010 audio.
The key challenge is that this isn't something you can quickly run a
program on, it requires manual effort to produce a good audio file and
I want to collaborate with others on figuring out how to do this
efficiently.
-igal
Ed Finkler
engineering off and on for the last 15 years or so. I can probably
plow through those files if they just involve trimming in not *too*
much time. I do have some commercial tools laying around to help,
though.
The fact that the files are already in mp3 format will complicate
things a bit and maybe degrade sound quality, but fidelity
requirements aren't terribly high for this kind of thing.
--
Ed Finkler
http://funkatron.com
@funkatron
AIM: funka7ron / ICQ: 3922133 / XMPP:funk...@gmail.com
Igal Koshevoy
A while back you uploaded a bunch of files which we've been publishing
as the "znmeb1" torrent. However, I have a number of questions about
those files that I didn't get answers for earlier. I'd like to get
these answered so we know if we should be using these as the basis of
our work or at least be able to resume it:
1. How did you rename and tag these files? Is there a data file that
explains the mappings between the old and new files? Is there a
program? If so, can you please share these?
2. Why are files missing? All I can tell is that there are fewer files
in your data set than in the original. Did you remove those files
deliberately, and if so, why? Because I don't have a mapping between
the new and old files, I can't figure out which ones are missing.
3. The audio quality seems worse. I recall that you re-encoded these,
but how and why?
-igal
Igal Koshevoy
> set up SSH keys for those that have processed audio files to share.
I'd very much prefer to host the files on the production servers we
already have. We've got plenty of bandwidth and adequate
storage.However, I do not want to hand out more accounts to the
production servers because this causes all sorts of headaches.
I've sent out a formal request to OSUOSL for 5-10GB of shared disk
space so we can use a more restricted account there to exchange
intermediate copies of files as we go through the processing work.
This partitioning approach will make it easier for us to hand out
access to those working on audio processing without impacting the
production servers.
I'll report back as I have news.
-igal
Igal Koshevoy
> I can't do meetups being this far away, but I have been doing audio
> engineering off and on for the last 15 years or so. I can probably
> plow through those files if they just involve trimming in not *too*
> much time. I do have some commercial tools laying around to help,
> though.
>
> The fact that the files are already in mp3 format will complicate
> things a bit and maybe degrade sound quality, but fidelity
> requirements aren't terribly high for this kind of thing.
Ed,
Thanks for the interest in processing the audio.
A major challenge with these audio files is that they're filled with
many extremely loud pops (e.g., 20x louder than the base audio),
fluctuation in level (e.g., presenters on panel often have very
different levels, or a speaker gets much closer/further from the mic),
and unpredictable levels that can't be fixed with simple normalization
or compression across the entire file. Do you have any suggestions on
how to deal with this, other than elbow grease?
The original files are decent quality mp3s and we plan to downsample
them for final distribution. So we're okay with losing quality -- but
I'm concerned with our intermediate files degrading to unacceptable
levels if we have to resave a file multiple times as we work on it. Do
you have any suggestions for file formats that we could use that are
(1) either lossless or close enough that we can safely resave files,
(2) compact enough that uploading them doesn't take forever, and (3)
use a quick enough encoding algorithm that it doesn't take forever to
encode them?
I'd much prefer it if we could use open source tools because that'd
help us get more people involved in the process. However, I definitely
appreciate that commercial tools might make save us time and would be
glad to hear your suggestions.
Thanks!
-igal
Ed Finkler
> On Sun, Jun 13, 2010 at 5:46 PM, Ed Finkler <c...@funkatron.com> wrote:
>> I can't do meetups being this far away, but I have been doing audio
>> engineering off and on for the last 15 years or so. I can probably
>> plow through those files if they just involve trimming in not *too*
>> much time. I do have some commercial tools laying around to help,
>> though.
>>
>> The fact that the files are already in mp3 format will complicate
>> things a bit and maybe degrade sound quality, but fidelity
>> requirements aren't terribly high for this kind of thing.
>
> Ed,
>
> Thanks for the interest in processing the audio.
>
> A major challenge with these audio files is that they're filled with
> many extremely loud pops (e.g., 20x louder than the base audio),
> fluctuation in level (e.g., presenters on panel often have very
> different levels, or a speaker gets much closer/further from the mic),
> and unpredictable levels that can't be fixed with simple normalization
> or compression across the entire file. Do you have any suggestions on
> how to deal with this, other than elbow grease?
Not really. You can mitigate it sometimes with tools to reduce pops
and clicks pulled off vinyl, but with the stuff you describe I
typically have to zoom in on the waveform and delete the pop.
>
> The original files are decent quality mp3s and we plan to downsample
> them for final distribution. So we're okay with losing quality -- but
> I'm concerned with our intermediate files degrading to unacceptable
> levels if we have to resave a file multiple times as we work on it. Do
> you have any suggestions for file formats that we could use that are
> (1) either lossless or close enough that we can safely resave files,
> (2) compact enough that uploading them doesn't take forever, and (3)
> use a quick enough encoding algorithm that it doesn't take forever to
> encode them?
So the work really needs to be done in a lossless format like AIFF or
WAV in my experience, and generally that's what a waveform editor will
do even if you open an mp3 up -- it will convert to a raw/PCM format
like that internally. The best you can do in terms of keeping the size
down for transferring while not losing data is to convert the files to
a lossless, compressed format like FLAC to move them around. You can
then decompress the files to WAV or AIFF to do your local editing.
If you're doing a single round of edits on the mp3s, I don't think
you'll lose much. If you do a bunch of re-compressions, that could
really hose up the fidelity, so you'd want to avoid that.
> I'd much prefer it if we could use open source tools because that'd
> help us get more people involved in the process. However, I definitely
> appreciate that commercial tools might make save us time and would be
> glad to hear your suggestions.
Last I checked Audacity was very capable for editing waveforms, so I
think you'd be fine doing most work with that. The quality of open
source/freebie plugins is, in my experience, not comparable to
commercial products, so stuff like strong noise reduction/click and
pop removal processors might not be up to snuff. I do have tools like
that though, and I don't mind putting the time in to batch process
stuff with those kinds of things.
You may also be able to find some audio-oriented folks locally who
would donate some of their time and/or lend gear. I know a couple
people around PDX I might poke about it.
M. Edward (Ed) Borasky
> Ed,
>
> A while back you uploaded a bunch of files which we've been publishing
> as the "znmeb1" torrent. However, I have a number of questions about
> those files that I didn't get answers for earlier. I'd like to get
> these answered so we know if we should be using these as the basis of
> our work or at least be able to resume it:
>
> 1. How did you rename and tag these files? Is there a data file that
> explains the mappings between the old and new files? Is there a
> program? If so, can you please share these?
All of the intermediate processing artifacts are stored on Github in
http://github.com/znmeb/OSBridge-2010/tree/master/Audio2009/
It will take me a couple of hours to reconstruct everything, but it
was a combination of data extraction from the 2009 schedule page and
manual listening to some files to match up room names and speakers
with file names. When I posted the resulting files, I said we should
crowdsource the validation of individual files with their tags. That
never happened, and I got tied up in preparations for Chirp. I'm
guessing there are only a few errors but I'd be extremely surprised if
the error count was non-zero.
The only audio processing I did was a normalization. I'll have to go
look at the scripts to see which tool I used, but the documentation
said that normalization of MP3s was done by adjusting a "master
volume" in the file somewhere, rather than changing any of the bits in
the file itself.
>
> 2. Why are files missing? All I can tell is that there are fewer files
> in your data set than in the original. Did you remove those files
> deliberately, and if so, why? Because I don't have a mapping between
> the new and old files, I can't figure out which ones are missing.
Some of the files had no audio in them. I can reconstruct a mapping
from the old file names to the new one - it may actually be in the
scripts already.
>
> 3. The audio quality seems worse. I recall that you re-encoded these,
> but how and why?
The only re-encoding was normalization. Let me go back to the Google
Group and resurrect the email chain from when I did the processing.
Igal Koshevoy
<zn...@borasky-research.net> wrote:
> Quoting Igal Koshevoy <ig...@pragmaticraft.com>:
>
>> Ed,
>>
>> A while back you uploaded a bunch of files which we've been publishing
>> as the "znmeb1" torrent. However, I have a number of questions about
>> those files that I didn't get answers for earlier. I'd like to get
>> these answered so we know if we should be using these as the basis of
>> our work or at least be able to resume it:
>>
>> 1. How did you rename and tag these files? Is there a data file that
>> explains the mappings between the old and new files? Is there a
>> program? If so, can you please share these?
>
> All of the intermediate processing artifacts are stored on Github in
>
> http://github.com/znmeb/OSBridge-2010/tree/master/Audio2009/
> It will take me a couple of hours to reconstruct everything, but it was a
> combination of data extraction from the 2009 schedule page and manual
> listening to some files to match up room names and speakers with file names.
> When I posted the resulting files, I said we should crowdsource the
> validation of individual files with their tags. That never happened, and I
> got tied up in preparations for Chirp. I'm guessing there are only a few
> errors but I'd be extremely surprised if the error count was non-zero.
Got it. I think we can ask volunteers to check the filename and tags
for the file they process.
>> 2. Why are files missing? All I can tell is that there are fewer files
>> in your data set than in the original. Did you remove those files
>> deliberately, and if so, why? Because I don't have a mapping between
>> the new and old files, I can't figure out which ones are missing.
>
> Some of the files had no audio in them. I can reconstruct a mapping from the
> old file names to the new one - it may actually be in the scripts already.
Ah. If it's not too hard to double-check this, that'd be great.
>> 3. The audio quality seems worse. I recall that you re-encoded these,
>> but how and why?
[...]
> The only re-encoding was normalization. Let me go back to the Google Group
> and resurrect the email chain from when I did the processing.
> The only audio processing I did was a normalization. I'll have to go look at
> the scripts to see which tool I used, but the documentation said that
> normalization of MP3s was done by adjusting a "master volume" in the file
> somewhere, rather than changing any of the bits in the file itself.
Sorry, I'm now not sure what the sound degradation I encountered was.
I spot-checked the waveforms and samples in a few of the new and old
files using Audacity and, other than levels, the data seems the same.
I seem to recall comparing entirely based on sound before and heard
something different -- maybe I was just fooled because your
normalization boosted noise in along with the vocals, which gave me
the illusion of sound degradation.
Anyway, I've edited the wiki page and indicated that znmeb1 is the
best version of the files to use as a starting point.
-igal
Igal Koshevoy
> On Sun, Jun 13, 2010 at 9:20 PM, Igal Koshevoy <ig...@pragmaticraft.com> wrote:
>> On Sun, Jun 13, 2010 at 5:46 PM, Ed Finkler <c...@funkatron.com> wrote:
>>> I can't do meetups being this far away, but I have been doing audio
>>> engineering off and on for the last 15 years or so. I can probably
>>> plow through those files if they just involve trimming in not *too*
>>> much time. I do have some commercial tools laying around to help,
>>> though.
>>>
>>> The fact that the files are already in mp3 format will complicate
>>> things a bit and maybe degrade sound quality, but fidelity
>>> requirements aren't terribly high for this kind of thing.
>>
>> Ed,
>>
>> Thanks for the interest in processing the audio.
>>
>> A major challenge with these audio files is that they're filled with
>> many extremely loud pops (e.g., 20x louder than the base audio),
>> fluctuation in level (e.g., presenters on panel often have very
>> different levels, or a speaker gets much closer/further from the mic),
>> and unpredictable levels that can't be fixed with simple normalization
>> or compression across the entire file. Do you have any suggestions on
>> how to deal with this, other than elbow grease?
>
> Not really. You can mitigate it sometimes with tools to reduce pops
> and clicks pulled off vinyl, but with the stuff you describe I
> typically have to zoom in on the waveform and delete the pop.
much. Maybe I couldn't figure out their settings. Deleting individual
pops works, although it's tedious.
>> The original files are decent quality mp3s and we plan to downsample
>> them for final distribution. So we're okay with losing quality -- but
>> I'm concerned with our intermediate files degrading to unacceptable
>> levels if we have to resave a file multiple times as we work on it. Do
>> you have any suggestions for file formats that we could use that are
>> (1) either lossless or close enough that we can safely resave files,
>> (2) compact enough that uploading them doesn't take forever, and (3)
>> use a quick enough encoding algorithm that it doesn't take forever to
>> encode them?
>
> So the work really needs to be done in a lossless format like AIFF or
> WAV in my experience, and generally that's what a waveform editor will
> do even if you open an mp3 up -- it will convert to a raw/PCM format
> like that internally. The best you can do in terms of keeping the size
> down for transferring while not losing data is to convert the files to
> a lossless, compressed format like FLAC to move them around. You can
> then decompress the files to WAV or AIFF to do your local editing.
The lossless FLAC format seems like an ideal format for sharing the
intermediate files.
>> I'd much prefer it if we could use open source tools because that'd
>> help us get more people involved in the process. However, I definitely
>> appreciate that commercial tools might make save us time and would be
>> glad to hear your suggestions.
>
> Last I checked Audacity was very capable for editing waveforms, so I
> think you'd be fine doing most work with that. The quality of open
> source/freebie plugins is, in my experience, not comparable to
> commercial products, so stuff like strong noise reduction/click and
> pop removal processors might not be up to snuff. I do have tools like
> that though, and I don't mind putting the time in to batch process
> stuff with those kinds of things.
Anything you can recommend would be great.
> You may also be able to find some audio-oriented folks locally who
> would donate some of their time and/or lend gear. I know a couple
> people around PDX I might poke about it.
That'd be fantastic. This audio processing stuff is very new to all of
us and we could really use a hand.
-igal
Igal Koshevoy
I posted my initial instructions for how to do this with Audacity:
http://opensourcebridge.org/2009/wiki/Session_Audio_Recordings#Processing_instructions
Later, I'll try to record a screencast or add explanatory screencaps,
because these would make it MUCH easier to explain what to look for in
a waveform and how to do things like Envelop and Amplify, which are
tricky to explain in text.
The resulting files and their sizes:
* MP3 original: 50 MB -- http://bit.ly/cA4vM3
* MP3 edited, encoded at 128Kbps: 45 MB -- http://bit.ly/dC6amf (very
nice quality)
* MP3 edited, encoded at q7: 23 MB -- http://bit.ly/9RgW4e (decent
quality for final files)
* FLAC at level 5 compression: 158 MB (excellent quality, lossless)
* Audacity project, storing chunks of raw PCM data: 2.5 GB (OMFG!)
Total intermediate file storage required if we use:
* FLAC: ~8.5 GB
* MP3 128Kbps: ~2.5 GB
Time estimates:
It took me about two hours, although much of this was spent learning
and fussing. I figure that with some practice I could process a file
in as little as 15-30 minutes if I'm less fussy about fixing minor
issues and can find quiet speech (e.g., audience members asking
questions) more quickly because it's almost indistinguishable from
background noise unless you're really zoomed in. I suspect that
sessions with lots of audience participation, multiple speakers, and
those whose speakers vary their volume more will be the most time
consuming to process.
The tasks that were most time-consuming were:
* Removing the many very loud noises, like taps on the mic, coughs and such.
* Quieting loud speech, like when the speaker was
talking/laughing/yelling directly into the mic.
* Boosting quiet speech, where speaker was very quiet or an audience
member was asking a question. These sections were difficult to find
and harder to boost because I had to amplify their associated
background noise with them. I couldn't figure out how to reduce noise
without destroying the quiet words. For an example, the volunteer
introducing Brian's talk in my processed file is about 1/5th the
volume of Brian's speech, and boosting it so it's audible added
unpleasant noise.
* Cleaning out "um's", "er's", tongue-clicks, long pauses, etc. Fixing
them was excessive, but they had very distinct visual shapes, so it
was easy to find and clean them up while I worked through the waveform
looking for more important issues. In retrospect, fixing the long
pauses would have been enough.
Anyway, I'd be glad to hear any suggestions or comments, especially on
the instructions or how to do those tasks more easily.
-igal
Eric Drechsel
I downloaded a session and experimented with it in Audacity, not sure
when I'll have time to do processing for a whole session.
> * Boosting quiet speech, where speaker was very quiet or an audience
> member was asking a question. These sections were difficult to find
> and harder to boost because I had to amplify their associated
> background noise with them. I couldn't figure out how to reduce noise
> without destroying the quiet words. For an example, the volunteer
> introducing Brian's talk in my processed file is about 1/5th the
> volume of Brian's speech, and boosting it so it's audible added
> unpleasant noise.
Did you try just using the noise reduction filter? I found that by
using a 20 second noise sample to create a profile, then applying that
globally before amplifying quiet speech, the speech still comes
through (but maybe if the speech is too quiet the noise reduction
destroys it?)