WebRTC Audio playoutDelayHint

[Ian Sweeney]

Hi there.

I'm working for a company that processes audio and turns it into graphics. We are trying to have instant control of audio delay by trying to use playoutDelayHint set to the time it takes to process audio.

Using Chrome 80.0.3987.87

I've added a delay to the Peer Connection demo https://webrtc.github.io/samples/src/content/peerconnection/pc1/. 

function gotRemoteStream(e) {

  if (remoteVideo.srcObject !== e.streams[0]) {

    remoteVideo.srcObject = e.streams[0];

    console.log('pc2 received remote stream');

  }

  const [audioReceiver, videoReceiver] = pc2.getReceivers();

  // Add additional 2 seconds of buffering.

  audioReceiver.playoutDelayHint = 2;

  videoReceiver.playoutDelayHint = 2;

}

The pc2 video shows instant 2 second buffering, and it takes the audio about 30 seconds to get in sync with the video.

I tried to add a delay to the audio to the Peer Connection: audio only demo https://webrtc.github.io/samples/src/content/peerconnection/audio/

function gotRemoteStream(e) {

  if (audio2.srcObject !== e.streams[0]) {

    audio2.srcObject = e.streams[0];

    console.log('Received remote stream');

  }

  const [audioReceiver] = pc2.getReceivers();

  // Add additional 2 seconds of buffering.

  audioReceiver.playoutDelayHint = 2;

}

There does not appear to be any delay while the audio sounds tinny.

I'm expecting the audio buffer to increase and behave like it was a video stream in the first example.

Am I missing something?

Is there a webrtc example with instant control of audio only delay? 

Help appreciated.

Regards

Ian 

[Sergio Garcia Murillo]

I assume that you are registered to the use the playoutDelayHint on the origin trial experiment, right?

From my tests the neteq jitter buffer does not instantly adds delay to the jitter buffer to match the hint delay, but lowers the playout so it starts to ramp up slowly. I am trying to modify that behavior, because I also think that adding the audio should match the hint as fast as possible (adding silence at the beginning).

Also take note that the jitterBufferDelay stat is inaccurate, so you can't rely on it for double checking if it is working or not.

Best regards

Sergio

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/9796e6ab-a261-477b-90e8-022a453e8f94%40googlegroups.com.

[Ian Sweeney]

Hi Sergio. 

I'm trying to get a handle on playoutDelayHint. It is not predictable in my audio applications.   

playoutDelayHint seems to work as expected for video, but not for audio.   

Keeping things as simple as possible, I've tried adding playoutDelayHint to pc2 in the original WebRTC sample demos, once the remote stream is received.

I'm relying on my own hearing, not stats. With playoutDelayHint=2, making noises in my microphone should be heard in my headset roughly 2 seconds later, but I hear it immediately. No ramping occurs when only audio is transmitted. 

When there is video and audio, the 'slow ramp' is about 30 seconds for audio to sync with delayed video.   

It doesnt seem like desired behaviour when you say you are trying to modify for instant match.

We trying to avoid sending audio through data channels to have control of audio timing.

Can you please inform me on any playoutDelayHint updates.

Thanks and Regards

Ian

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/d395cb2a-41c4-d0e7-0d8f-cdaa3634cd9e%40gmail.com.

--

Ian Sweeney

ian.s...@speech-graphics.com

+447543629522

[Sergio Garcia Murillo]

Just to double check, have you register your domain for the origin trial?

https://developers.chrome.com/origintrials/#/register_trial/1463669878895411201

Best regards

Sergio

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAJLUFi%3D0U67OwM35DUWoP8S0HHfZN6k0_J_ikxeqYwx0CH%2B0Dg%40mail.gmail.com.

[Ian Sweeney]

Thanks Sergio, 

I will register with my experiment. 

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/2361d335-7eaa-f9d3-bbd5-e0d52625d43b%40gmail.com.

[Philipp Hancke]

Am Do., 6. Feb. 2020 um 14:49 Uhr schrieb Sergio Garcia Murillo <sergio.gar...@gmail.com>:

I assume that you are registered to the use the playoutDelayHint on the origin trial experiment, right?

At least for local development that isn't required when one sets

  chrome://flags/#enable-experimental-web-platform-features

no? 'playoutDelayHint' in window.RTCRtpReceiver.prototype should be true in that case.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/d395cb2a-41c4-d0e7-0d8f-cdaa3634cd9e%40gmail.com.

[Sergio Garcia Murillo]

On 06/02/2020 15:45, 'Philipp Hancke' via discuss-webrtc wrote:

At least for local development that isn't required when one sets

  chrome://flags/#enable-experimental-web-platform-features

no? 'playoutDelayHint' in window.RTCRtpReceiver.prototype should be true in that case.

Yes, that works too, thanks for the tip.

Best regards

Sergio

[Ian Sweeney]

thanks. 

I have been using playoutDelayHint and had some success.  'playoutDelayHint' in window.RTCRtpReceiver.prototype returns true. I enabled  enable-experimental-web-platform-features with no changes in behavior.

I have been trying to add audio delay to the conversation between two peers in two different browsers while using a signalling server. While making a call, one of my peers has the expected 2 second audio delay while the other does not. I cant figure out why.

It would be nice to see a sample peer connection audio only working demo with playoutDelayHint set to 2.

They ramp we have been speaking of is also undesirable.

Regards

Ian

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/111c32f5-b86d-0381-b73a-131edb63ac14%40gmail.com.

[Ian Sweeney]

Hi Sergio, 

I've noticed that videoReceiver.playoutDelayHint delays both video and audio.

while audioReceiver.playoutDelayHint has no effect on audio delay.

const [audioReceiver, videoReceiver] = pc2.getReceivers();

  // Add additional 2 seconds of buffering.

  // audioReceiver.playoutDelayHint = 2; // Has no effect on audio delay

  videoReceiver.playoutDelayHint = 2; // Delays both audio and video

Is this a design choice?

Are you only trying to modify the ramp behavior so video/audio gets in sync quicker?

Is there plans to be able to control video delay and audio delay independent from each other? Ie, giving developers the ability to make asynchronous video/audio

Thanks and Regards

Ian

[Sergio Garcia Murillo]

On 07/02/2020 10:46, Ian Sweeney wrote:

Hi Sergio, 

I've noticed that videoReceiver.playoutDelayHint delays both video and audio.

while audioReceiver.playoutDelayHint has no effect on audio delay.

const [audioReceiver, videoReceiver] = pc2.getReceivers();

  // Add additional 2 seconds of buffering.

  // audioReceiver.playoutDelayHint = 2; // Has no effect on audio delay

  videoReceiver.playoutDelayHint = 2; // Delays both audio and video

Is this a design choice?

Probably you are also triggering lip sync which may affect latency of both flows.

Are you only trying to modify the ramp behavior so video/audio gets in sync quicker?

My use case is audio only.

Is there plans to be able to control video delay and audio delay independent from each other? Ie, giving developers the ability to make asynchronous video/audio

Try disabling lip sync (by sending them in two different streams and RTCP RR).

Best regards

Sergio

[Ian Sweeney]

Hi Sergio, 

The big problem we are trying to solve is having audio stream with graphics data coming through datachannel. We have no video stream. Data channel graphics comes from processed audio that includes a delay. We have lip sync issues and need to delay our only stream which is audio. We want to be able use audioReceiver.playoutDelayHint like you would use videoReceiver.playoutDelayHint

adding audioReceiver.playoutDelayHint to audio in this audio only demo https://webrtc.github.io/samples/src/content/peerconnection/audio/ shows no delay in audio. Is this a problem for you and are you trying to fix it?

what do you mean by triggering lip sync and disabling lip sync? Is this applicable to audio only applications?

Thanks and Regards

Ian

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/41734222-b53a-e10a-31ef-7be4c685cf31%40gmail.com.

[Sergio Garcia Murillo]

On 07/02/2020 11:38, Ian Sweeney wrote:
> adding audioReceiver.playoutDelayHint to audio in this audio only demo
> https://webrtc.github.io/samples/src/content/peerconnection/audio/ shows
> no delay in audio. Is this a problem for you and are you trying to fix it?

All I can say is that it is working for me, check this demo:
https://murillo128.github.io/web-audio-rtc-test/ right channel is
delayed via web audio and left channel is delayed via playoutDelayHint,
you can play until you are able to sync them both.. :)


Best regards

Sergio

[Ian Sweeney]

Thanks. 

I'll have a look/listen.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/e1dd2216-e35a-0f85-48e6-895cacc6dc77%40gmail.com.

[Ian Sweeney]

Hi Sergio. 

A bit more work than delaying video which is 1 line of code. But this is exactly what I need.

Many thanks

Ian

[Ian Sweeney]

Hi Sergio,

For the audio demo https://webrtc.github.io/samples/src/content/peerconnection/audio/ 
 

if you want to delay your microphone audio by 2 seconds, the workaround for me is to periodically update playoutDelayHint and jitterBufferDelayHint to 2 seconds. This is the solution I've been looking for. 

function gotRemoteStream(e) {

  if (audio2.srcObject !== e.streams[0]) {

    audio2.srcObject = e.streams[0];

    console.log('Received remote stream');

  }

  // Add additional 2 seconds of buffering

  const [audioReceiver] = pc2.getReceivers();

  setInterval(async () => {

    audioReceiver.playoutDelayHint = audioReceiver.jitterBufferDelayHint = 2;

  }, 1);

}

Not as intuitive as delaying video.

Thanks

Ian