RTP synchronization and PTS assignment issues

[mparis]

Hello,
I am dealing with synchronization and PTS assignment issues interoperating with Chrome in our implementation.
The interstream synchronization (typically known as lypsync) and the PTS assignment are based on:

• Timestamp of the RTP packets received [1]
  
• RTP timestamp of the RTP SR (Sender Report) packets received [2]
• NTP timestamp of the RTP SR (Sender Report) packets received [2]

Specifically, I have problems in the video PTS assignment due to:

• Huge gaps in the timestamp of the RTP packets sent by Chrome.
  
• The intrinsic problem of receiving RTP and RTCP-SR packets out of sync (not serialized).
• Even, the RTCP-SR can be lost
  

To understand the problem I am exposing a real example. The example shows part of a video RTP stream where the clockrate is 90000:
The RTCP-SR-M is received

SSRC: 100020, rtp_time: 1874314573, ntp_time: 15816590577043708928, ntp_ns_time: 1022940:49:24.462815999

The RTP packets of the VideoFrame-N are received

seqnum 8016, rtptime 1874315833, pts 0:00:22.727008174
seqnum 8017, rtptime 1874315833, pts 0:00:22.727008174
seqnum 8018, rtptime 1874315833, pts 0:00:22.727008174
seqnum 8019, rtptime 1874315833, pts 0:00:22.727008174

[5 frames more received seqnums: 8020-8040]
The RTP packets of the VideoFrame-(N+6) are received

seqnum 8041, rtptime 1874342833, pts 0:00:23.027008174
seqnum 8042, rtptime 1874342833, pts 0:00:23.027008174
seqnum 8043, rtptime 1874342833, pts 0:00:23.027008174
seqnum 8044, rtptime 1874342833, pts 0:00:23.027008174

An RTCP-SR-(M+1) is received

SSRC: 100020, rtp_time: 1874345983, ntp_time: 15816590579105293230, ntp_ns_time: 1022940:49:24.942815999

The RTP packets of the VideoFrame-(N+7) are received
Here we can notice a huge increment gap in the RTP timestamps: 3.687 seconds [(1874674663 - 1874342833) / 90000]

seqnum 8045, rtptime 1874674663, pts 0:00:26.845008174
seqnum 8046, rtptime 1874674663, pts 0:00:26.845008174
seqnum 8047, rtptime 1874674663, pts 0:00:26.845008174
seqnum 8048, rtptime 1874674663, pts 0:00:26.845008174
seqnum 8049, rtptime 1874674663, pts 0:00:26.845008174
seqnum 8050, rtptime 1874674663, pts 0:00:26.845008174
seqnum 8051, rtptime 1874674663, pts 0:00:26.845008174
seqnum 8052, rtptime 1874674663, pts 0:00:26.845008174

[3 frames more received seqnums: 8053-8068]
The RTP packets of the VideoFrame-(N+11) are received

seqnum 8069, rtptime 1874699503, pts 0:00:27.121008174
seqnum 8070, rtptime 1874699503, pts 0:00:27.121008174
seqnum 8071, rtptime 1874699503, pts 0:00:27.121008174

the RTCP-SR-(M+2) is received
Here we can notice a huge increment gap between the RTP timestamp of this RTCP SR packet and the last one: 3.948 seconds [(1874701303 - 1874345983) / 90000]
but a small gap between their NTP timestamps: 0.368 seconds [25.310815999 - 24.942815999]

SSRC: 100020, rtp_time: 1874701303, ntp_time: 15816590580685841195, ntp_ns_time: 1022940:49:25.310815999

The RTP packets of the VideoFrame-(N+12) are received
Here we can notice a huge decrement gap in the resulting PTS due to use the last RTCP SR info

seqnum 8072, rtptime 1874704003, pts 0:00:23.591008174
seqnum 8073, rtptime 1874704003, pts 0:00:23.591008174
seqnum 8074, rtptime 1874704003, pts 0:00:23.591008174
seqnum 8075, rtptime 1874704003, pts 0:00:23.591008174

From this example, I would say that the PTS of the VideoFrame(N+7) to VideoFrame(N+10) are not valid and should have been based on the RTCP-SR-(M+2), but we cannot know it.
So, if I am not wrong, with the currently implemented protocols there is not a solution for that. To solve the problem we should have the NTP time in each RTP packet as proposed in "Rapid Synchronisation of RTP" [3] (discussed in [4]).

Anyway, I would like to know:

• How does Chrome deal with this problem?
• How does Chrome assign the PTS for the video frames and audio samples received?
• Could please anybody give me a pointer to the source code? (I didn't find it)

Thanks in advance!!

Versions
Chrome 53.0.2785.101 m

Refs
[1] https://tools.ietf.org/html/rfc3550#section-5.1
[2] https://tools.ietf.org/html/rfc3550#section-6.4.1
[3] https://tools.ietf.org/html/rfc6051
[3] https://groups.google.com/forum/#!searchin/discuss-webrtc/from$3Ame%7Csort:relevance/discuss-webrtc/XRd5WEWD5VM/1_4FaUpVBQAJ

[Simone Mazzoni]

mparis did you found a solution for this?

I experienced a similar problem.

Basically I'm trying to implement an algorithm to synchronize webrtc RTP audio and video stream using RTP timestamp, RTCP timestamp and RTCP ntptimestamp.

Have you already implemented an algorithm for this?

[Stefan Holmer]

FYI, the corresponding issue has been resolved: https://bugs.chromium.org/p/chromium/issues/detail?id=651338

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/1b58ca3c-59ba-4b68-a471-11628d58b16d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Simone Mazzoni]

Hi Stefan,

Thanks for the report.

I ask information not because I experienced the same bug in chromium, but because I'm trying to develop an algorithm to deal with synchronization between two RTP streams using RTCP SR (sender reports).

Long story short, I'm trying to implement the mechanism used by chromium when it receives Webrtc Audio and Video RTP streams, to synchronize them to achieve Lip-synchronization.

Maybe I should open a dedicated post?