MediaRecorder does not resample Opus input audio

[rayc...@google.com]

Hi, I'm attempting to record audio via MediaRecorder at 48khz regardless of my microphone's sample rate.

I'm on firefox 59. My microphone records audio at a sample rate of 44.1khz. However, MediaRecorder outputs OGG Opus audio at 48khz.

Here's how I'm using MediaRecorder https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder#Example.

I would have expected 48khz output per OpusTrackEncoder https://github.com/mozilla/gecko-dev/blob/a341b32d064bafac5422f72fef6afe740045105c/dom/media/encoder/OpusTrackEncoder.cpp#L162 and this bug https://bugzilla.mozilla.org/show_bug.cgi?id=922247.

Am I wrong in undrestanding that MediaRecorder should resample to 48khz if the microphone's sample rate is non-standard?

[Timothy B. Terriberry]

raycarino--- via dev-media wrote:
> Hi, I'm attempting to record audio via MediaRecorder at 48khz regardless of my microphone's sample rate.

Okay.

> I'm on firefox 59. My microphone records audio at a sample rate of 44.1khz. However, MediaRecorder outputs OGG Opus audio at 48khz.

This is what you wanted, right?

> I would have expected 48khz output

What did you get that you didn't expect?

[rayc...@google.com]

On Tuesday, July 17, 2018 at 12:09:04 PM UTC-7, Timothy B. Terriberry wrote:
> raycarino--- via dev-media wrote:
> > Hi, I'm attempting to record audio via MediaRecorder at 48khz regardless of my microphone's sample rate.
>
> Okay.
>
> > I'm on firefox 59. My microphone records audio at a sample rate of 44.1khz. However, MediaRecorder outputs OGG Opus audio at 48khz.
>
> This is what you wanted, right?

Thank you for the quick reply!

Sorry, for the typo. Expected: 48kHz output. Actual is 44.1kHz. It seems that the resampling logic either failed or wasn't reached.

[Andreas Pehrson]

Thanks for the report.

I followed the example you linked to the webapp it originated from and
tested with that: https://mdn.github.io/web-dictaphone/

However, I was not able to reproduce your findings on linux with Firefox
Nightly (63). With both a 48k source and a 44.1k source I get ogg/opus
files of 48k. I checked the files with ffprobe and Audacity.

Could you share more details on your setup and how you are checking the
output files?

Also, if you still see 44.1k files, do you mind filing a bug with all
information that could help us reproduce and debug this at
https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Audio%2FVideo%3A%20Recording
?


Thanks,

Andreas

On Thu, Jul 19, 2018 at 12:21 AM raycarino--- via dev-media <
dev-...@lists.mozilla.org> wrote:

> On Tuesday, July 17, 2018 at 12:09:04 PM UTC-7, Timothy B. Terriberry
> wrote:

> > raycarino--- via dev-media wrote:
> > > Hi, I'm attempting to record audio via MediaRecorder at 48khz
> regardless of my microphone's sample rate.
> >
> > Okay.
> >
> > > I'm on firefox 59. My microphone records audio at a sample rate of
> 44.1khz. However, MediaRecorder outputs OGG Opus audio at 48khz.
> >
> > This is what you wanted, right?

> Thank you for the quick reply!
>
> Sorry, for the typo. Expected: 48kHz output. Actual is 44.1kHz. It seems
> that the resampling logic either failed or wasn't reached.
>
> >

> > > I would have expected 48khz output
> >
> > What did you get that you didn't expect?
>

> _______________________________________________
> dev-media mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-media
>

[Timothy B. Terriberry]

raycarino--- via dev-media wrote:
>>> I'm on firefox 59. My microphone records audio at a sample rate of 44.1khz. However, MediaRecorder outputs OGG Opus audio at 48khz.
>>
>> This is what you wanted, right?

> Thank you for the quick reply!
>
> Sorry, for the typo. Expected: 48kHz output. Actual is 44.1kHz. It seems that the resampling logic either failed or wasn't reached.

No worries. I figured there was a typo somewhere, just trying to figure
out exactly where.

The Ogg Opus file format does not actually support encoding 44.1 kHz
audio directly, so unless you can hear some sort of weird pitch shifting
going on, that means it hit the resampling logic correctly.

What it does do is store the original sampling rate in the header as
metadata. That lets other tools resample back to the original sampling
rate if that is appropriate. E.g., if the input was originally 44.1 kHz,
it's normal to resample back to 44.1 kHz when decoding to a file to
avoid surprises for users who don't know that Opus internally only
supports sample rates that evenly divide 48 kHz. But if you were just
playing the file back, you'd normally skip that resampling as most
soundcards run at 48 kHz natively.

I hope that helps!

[rayc...@google.com]

It does. Thank you so much Timothy and Andreas!

Re: Andreas' question about how I'm checking the output files.
I was only checking the input sample rate section of the OGG header. I expected it be the encoded sample rate, but that assumption was wrong and clarified by Timothy.

[rayc...@google.com]

I have a follow on question. What's the best way to determine an Ogg Opus file's encoded sample rate?

I've tried ffprobe as suggested by Andreas like so:
$: ffprobe -show_streams dicto_44.1_label.ogg
>>>
...
[STREAM]
...
sample_rate=48000
...
[/STREAM]

Does this ffprobe command prove the file is encoded at 48kHz? In my mind, it's also possible that the [STREAM] section's sample rate could also be the decoding sample rate per the RFC's spec "1. If the hardware supports 48 kHz playback, decode at 48 kHz." https://tools.ietf.org/html/rfc7845#section-5.1.

[Timothy B. Terriberry]

raycarino--- via dev-media wrote:
> I have a follow on question. What's the best way to determine an Ogg Opus file's encoded sample rate?

Opus is a lossy codec. The sample rate is something that is lost as part
of the lossy encoding. The original sample rate in the header gives you
a hint as to the best sample rate to run the decoder at to get all of
the information that was encoded, but it is perfectly possible to run it
any supported rate (8 kHz, 12 kHz, 16 kHz, 24 kHz, and 48 kHz),
regardless of how it was encoded.

You _could_ get some idea of the content of the audio by looking at the
TOC byte in each of the packets (c.f. RFC 6716 Section 3.1). NB implies
8 kHz, MB implies 12 kHz, WB implies 16 kHz, SWB implies 24 kHz, and FB
implies 48 kHz. But this is allowed to change on a packet-by-packet
basis, and the encoder may in fact change it based on the content of the
audio that was fed to it and the requested bitrate (regardless of what
sampling rate the encoder was running at). I'm not aware of a tool that
actually inspects and reports on this information, either.

[rayc...@google.com]

Thank you again Timothy! That all makes sense.

Hopefully these are my last follow-up questions. In general, I'm wondering whether there will be audio artifacts present when decoding at a different sample rate than what was encoded. Questions:
1. How does a high sample rate decoded at a lower sample rate behave? Example: will decode(48kHz.ogg 16kHz) === decode(16kHz.ogg, 16kHz), or can audio artifacts be expected?
2. Similarly, how does a low sample rate decoded at a higher sample rate behave? Example: will decode(16kHz.ogg, 48kHz) === decode(16kHz.ogg, 16kHz), or can audio artifacts be expected?

I'm hopeful that decoding with a different sample rate either has no distortion, or minimal distortion because:
(a) Firefox always seems to decode at 48kHz https://github.com/mozilla/gecko-dev/blob/f822a0b61631cbb38901569e69b4967176314aa8/dom/media/ogg/OpusParser.cpp#L46
(b) Chrome also seems to always decode at 48kHz https://github.com/chromium/chromium/blob/d196d28f53d37bad65feac0c0fb87a3b2c9480e9/remoting/codec/audio_decoder_opus.cc#L25

[Timothy B. Terriberry]

raycarino--- via dev-media wrote:
> 1. How does a high sample rate decoded at a lower sample rate behave? Example: will decode(48kHz.ogg 16kHz) === decode(16kHz.ogg, 16kHz), or can audio artifacts be expected?

They won't give bit-identical results, but with some hand-waving about
the version encoded at 48 kHz having enough extra bits to make sure it
maintains the same quality for frequencies in the 0...8 kHz range, they
should sound the same. You don't have to worry about aliasing or
anything like that. The high-frequency content, if any, in 48kHz.ogg
will be removed properly.

> 2. Similarly, how does a low sample rate decoded at a higher sample rate behave? Example: will decode(16kHz.ogg, 48kHz) === decode(16kHz.ogg, 16kHz), or can audio artifacts be expected?

These should sound the same without any hand-waving.

In both cases the quality loss from using the decoder API at a different
sample rate than used by the encoder API should be smaller than the loss
introduced by using lossy compression to begin with. In the first case
you will obviously also lose any HF content, but this shouldn't
introduce artifacts in what remains.

> I'm hopeful that decoding with a different sample rate either has no distortion, or minimal distortion because:
> (a) Firefox always seems to decode at 48kHz https://github.com/mozilla/gecko-dev/blob/f822a0b61631cbb38901569e69b4967176314aa8/dom/media/ogg/OpusParser.cpp#L46
> (b) Chrome also seems to always decode at 48kHz https://github.com/chromium/chromium/blob/d196d28f53d37bad65feac0c0fb87a3b2c9480e9/remoting/codec/audio_decoder_opus.cc#L25

"Always use 48 kHz" is a fine approach. The real reason to use lower
sampling rates is if you want to run your whole audio pipeline at the
lower rate. Sometimes for real-time conferencing that can reduce the
processing required for other parts of the pipeline, like the echo
canceler, and improve their reliability. On small embedded devices it
can avoid wasting memory and cycles and save a separate resampling step
if your ADC or DAC aren't going to be run at the higher rate anyway.
None of those are of much concern for media playback in a browser.

[rayc...@google.com]

Thank you so much for all of your help! This all makes sense. All of my outstanding questions have been answered.