Why would Chrome raise googCurrentDelayMs linearly?

[Juan Navarro]

Hi,

I'm studying some lip sync delays that grow linearly, which is a problem that happens when the source stream is originated by a GStreamer pipeline using a pitch-modifying filter. googCurrentDelayMs grows linearly in Chrome when it receives such stream, and both googMinPlayoutDelayMs and googTargetDelayMs grow very rapidly up until the maximum value of 10000 ms.

This means that Chrome is receiving a video+audio stream and, for some reason, decides that the audio is lagging behind the video, so the video should be delayed in order to synchronize both. The symptoms are practically the same as described in this bug: https://bugs.chromium.org/p/webrtc/issues/detail?id=5456

This is my capture:

However the actual reason for the issue must be different that in the linked bug report; I'm using here a GStreamer plugin that changes the audio pitch, but whenever any other plugin is used, this issue doesn't happen. My question is, what is the reason for such a delay in Chrome? Is it related to RTP sequence numbers? Clock rate? Bad PTS/DTS? Or in other words: from where does Chrome take the information that is later used for lip sync, that could affect in this way if it's wrong?

I've been trying to follow WebRTC's code, but still unsure about the exact point in which the decision is taken to raise the delay amount. I'd like to have as much information as possible to hand it out to the GStreamer community and make it easier to help.

I have captured all information I could think of: diagnostic audio recordings (Chrome's fake audio), diag. packet and event recording, webrtc-internals dump, chrome debug log.

Using Chromium (68.0.3440.106) and Chrome (69.0.3497.81).

[Philipp Hancke]

do a pcap, check the rtp timestamps for both audio and video packets and see if they increase at different rates.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/6537d54d-6417-4fdd-be39-02dc94e7624e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Juan Navarro]

Sorry, here is the webrtc-internals dump capture (from fippo's importer)

On 06/09/18 19:35, Juan Navarro wrote:

Hi,

I'm studying some lip sync delays that grow linearly, which is a problem that happens when the source stream is originated by a GStreamer pipeline using a pitch-modifying filter. googCurrentDelayMs grows linearly in Chrome when it receives such stream, and both googMinPlayoutDelayMs and googTargetDelayMs grow very rapidly up until the maximum value of 10000 ms.

This means that Chrome is receiving a video+audio stream and, for some reason, decides that the audio is lagging behind the video, so the video should be delayed in order to synchronize both. The symptoms are practically the same as described in this bug: https://bugs.chromium.org/p/webrtc/issues/detail?id=5456

This is my capture:

However the actual reason for the issue must be different that in the linked bug report; I'm using here a GStreamer plugin that changes the audio pitch, but whenever any other plugin is used, this issue doesn't happen. My question is, what is the reason for such a delay in Chrome? Is it related to RTP sequence numbers? Clock rate? Bad PTS/DTS? Or in other words: from where does Chrome take the information that is later used for lip sync, that could affect in this way if it's wrong?

I've been trying to follow WebRTC's code, but still unsure about the exact point in which the decision is taken to raise the delay amount. I'd like to have as much information as possible to hand it out to the GStreamer community and make it easier to help.

I have captured all information I could think of: diagnostic audio recordings (Chrome's fake audio), diag. packet and event recording, webrtc-internals dump, chrome debug log.

Using Chromium (68.0.3440.106) and Chrome (69.0.3497.81).

--

---
You received this message because you are subscribed to a topic in the Google Groups "discuss-webrtc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/discuss-webrtc/k9so0_12pqM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss-webrt...@googlegroups.com.

[Juan Navarro]

Thanks for the tip; that's actually the case. Video packets alternate the Marker bit, and for those packets that have Marker = 1, the timestamp doesn't change from the precious one. For packets with Marker = 0, timestamp raises by 4500.

On the other hand, audio packets don't have Marker = 1 and always increase their timestamp by 960.

A table can be clearer:

Video:

Marker | Seq N | Timestamp  | Ts increment
0      | 32787 | 3783431126 | -
1      | 32788 | 3783431126 | 0
0      | 32789 | 3783435626 | +4500
1      | 32790 | 3783435626 | 0
0      | 32791 | 3783440126 | +4500
1      | 32792 | 3783440126 | 0
0      | 32793 | 3783444626 | +4500
1      | 32794 | 3783444626 | 0
0      | 32795 | 3783449126 | +4500


Audio:

Marker | Seq N | Timestamp | Ts increment
0      | 18669 | 238837622 | -
0      | 18670 | 238838582 | +960
0      | 18671 | 238839542 | +960
0      | 18672 | 238840502 | +960
0      | 18673 | 238841462 | +960
0      | 18674 | 238842422 | +960
0      | 18675 | 238843382 | +960
0      | 18676 | 238844342 | +960
0      | 18677 | 238845302 | +960

Should the timestamp rates be the same?

I'm right now going to repeat the capture with a well-working pipeline, and check how the timestamp values evolve over time.

On Thursday, September 6, 2018 at 7:41:57 PM UTC+2, Philipp Hancke wrote:

do a pcap, check the rtp timestamps for both audio and video packets and see if they increase at different rates.

2018-09-06 19:35 GMT+02:00 Juan Navarro <juan.n...@gmx.es>:

Hi,

I'm studying some lip sync delays that grow linearly, which is a problem that happens when the source stream is originated by a GStreamer pipeline using a pitch-modifying filter. googCurrentDelayMs grows linearly in Chrome when it receives such stream, and both googMinPlayoutDelayMs and googTargetDelayMs grow very rapidly up until the maximum value of 10000 ms.

This means that Chrome is receiving a video+audio stream and, for some reason, decides that the audio is lagging behind the video, so the video should be delayed in order to synchronize both. The symptoms are practically the same as described in this bug: https://bugs.chromium.org/p/webrtc/issues/detail?id=5456

This is my capture:

However the actual reason for the issue must be different that in the linked bug report; I'm using here a GStreamer plugin that changes the audio pitch, but whenever any other plugin is used, this issue doesn't happen. My question is, what is the reason for such a delay in Chrome? Is it related to RTP sequence numbers? Clock rate? Bad PTS/DTS? Or in other words: from where does Chrome take the information that is later used for lip sync, that could affect in this way if it's wrong?

I've been trying to follow WebRTC's code, but still unsure about the exact point in which the decision is taken to raise the delay amount. I'd like to have as much information as possible to hand it out to the GStreamer community and make it easier to help.

I have captured all information I could think of: diagnostic audio recordings (Chrome's fake audio), diag. packet and event recording, webrtc-internals dump, chrome debug log.

Using Chromium (68.0.3440.106) and Chrome (69.0.3497.81).

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

[Philipp Hancke]

compare this to the sampling rate. For audio, 960usec per packet times 50 packets per second is 48000 which is the opus sampling rate.

For video, it is a bit more complicated since there are multiple packets per frame but it should come out at an increase of 90000 per second. If you frequently get more that would explain the drift.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/0149cfcd-3848-4c02-802f-449e8899b218%40googlegroups.com.