MediaStreamGraph latency issues

Robert O'Callahan

unread,

Nov 13, 2012, 8:46:02 PM11/13/12

to dev-...@lists.mozilla.org, Randell Jesup, Ehsan Akhgari, Paul Adenot, Maire Reavy

Randell, Maire and I talked about how to minimize latency for WebRTC and
Web Audio with the MediaStreamGraph. I then talked about it some more with
Ehsan and Paul.

Here's a wiki page summarizing my understanding of where we should or could
be with the current design (modulo bugs and scheduling uncertainty). It
also has some suggestions for improving the design.

Feedback appreciated.

Rob
--
Jesus called them together and said, “You know that the rulers of the
Gentiles lord it over them, and their high officials exercise authority
over them. Not so with you. Instead, whoever wants to become great among
you must be your servant, and whoever wants to be first must be your
slave — just
as the Son of Man did not come to be served, but to serve, and to give his
life as a ransom for many.” [Matthew 20:25-28]

Eric Rescorla

unread,

Nov 13, 2012, 8:49:18 PM11/13/12

to rob...@ocallahan.org, dev-...@lists.mozilla.org, Ehsan Akhgari, Maire Reavy, Randell Jesup, Paul Adenot

There does not appear to be a link in this page.

-Ekr

> _______________________________________________
> dev-media mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-media
>

Robert O'Callahan

unread,

Nov 13, 2012, 8:51:13 PM11/13/12

to Eric Rescorla, dev-...@lists.mozilla.org, Ehsan Akhgari, Maire Reavy, Randell Jesup, Paul Adenot

On Tue, Nov 13, 2012 at 5:49 PM, Eric Rescorla <e...@rtfm.com> wrote:

> There does not appear to be a link in this page.

Indeed! Sorry.
https://wiki.mozilla.org/Gecko:MediaStreamLatency

Randell Jesup

unread,

Nov 14, 2012, 1:52:07 PM11/14/12

to rob...@ocallahan.org, Eric Rescorla, dev-...@lists.mozilla.org, Ralph Giles, Jan-Ivar Bruaroey, Ehsan Akhgari, Maire Reavy, Tim Terriberry, Paul Adenot

Adding Tim, Ralph and Jan-Ivar

On 11/13/2012 8:51 PM, Robert O'Callahan wrote:
> Randell, Maire and I talked about how to minimize latency for WebRTC
> and Web Audio with the MediaStreamGraph. I then talked about it some
> more with Ehsan and Paul.
>
> Here's a wiki page summarizing my understanding of where we should or
> could be with the current design (modulo bugs and scheduling
> uncertainty). It also has some suggestions for improving the design.
>
> Feedback appreciated.
>
> Rob

> On Tue, Nov 13, 2012 at 5:49 PM, Eric Rescorla <e...@rtfm.com
> <mailto:e...@rtfm.com>> wrote:
>
> There does not appear to be a link in this page.
>
>
> Indeed! Sorry.
> https://wiki.mozilla.org/Gecko:MediaStreamLatency

FYI (from dev.media): A December 2009 description of the Chromium
audio/video pipeline (for output), which is all pull-based: (I also sent
this back in July to mozilla.dev.media)

https://sites.google.com/a/chromium.org/dev/developers/design-documents/video

I'm investigating the *actual* code Chromium uses; while not the same
code as in webrtc.org, it's tied into the webrtc.org code at a low
level. See also bug 811695. Their direct input code at least through
the capture and channel stages is all push-based from the driver
callbacks (as expected).

See
https://src.chromium.org/viewvc/chrome/trunk/src/content/renderer/media/webrtc_audio_device_impl.h?view=markup
for part of their code and some discussion of how it works. Note in
particular this:

// Then WebRtcAudioDeviceImpl::SetSessionId() needs to be called to set the
// session id that tells which device to use. The user can either get the
// session id from the MediaStream or use a value of 1 (AudioInputDeviceManager
// ::kFakeOpenSessionId), the later will open the default device without going
// through the MediaStream.

So their code can either take input from a mediastream directly, or can
open the default audio device directly, and then feed it into the
processing code (AGC, etc). I'm still looking for their
audio_device->MediaStream code.

Randell