Sync streams from multiple participants

[pablo platt]

I'm capturing RTP and RTCP packets from audio and video streams and merge them to a video with a post-processing script.

I've read about the RTP timestamp and NTP timestamp in RTCP SR and how to sync audio and video streams from the same source.

How can I synchronize streams from multiple participants?

I also need to synchronize the streams with other events in the recording like whiteboard annotations and presentation slides.

rfc3550 says:

An implementation is not required to run the Network Time Protocol in
order to use RTP.  Other time sources, or none at all, may be used
(see the description of the NTP timestamp field in Section 6.4.1).
However, running NTP may be useful for synchronizing streams
transmitted from separate hosts.

Do WebRTC clients (Chrome, Firefox, OpenWebRTC) sync their clocks with NTP servers?

How does Chrome handles clock drift between the sender and the receiver?

Can I compensate the clock drift with the following?
- Save the receiver local timestamp for each SR packet
- Get a polynomial fit that translates SR NTP timestamp to receiver local timestamp.

- Use the polynomial fit with setpts/asetpts in ffmpeg.

https://ffmpeg.org/ffmpeg-filters.html#setpts_002c-asetpts

Thanks

[Randell Jesup]

On 8/1/2015 11:12 PM, pablo platt wrote:
> I'm capturing RTP and RTCP packets from audio and video streams and
> merge them to a video with a post-processing script.
>
> I've read about the RTP timestamp and NTP timestamp in RTCP SR and how
> to sync audio and video streams from the same source.
>
> How can I synchronize streams from multiple participants?
> I also need to synchronize the streams with other events in the
> recording like whiteboard annotations and presentation slides.

You can synchronize the streams coming *from* a participant (by putting
them in the same MediaStream). Synchronizing *multiple* participants
means basically recording the view as seen by either the server or one
participant.

>
> rfc3550 says:
> An implementation is not required to run the Network Time Protocol in
> order to use RTP. Other time sources, or none at all, may be used
> (see the description of the NTP timestamp field in Section 6.4.1).
> However, running NTP may be useful for synchronizing streams
> transmitted from separate hosts.
> Do WebRTC clients (Chrome, Firefox, OpenWebRTC) sync their clocks with
> NTP servers?

You should ask "do the computers users run WebRTC on sync NTP?" Likely
yes, but not always. And even synced NTP isn't perfect *between*
machines. Some machines don't sync and the time may be off by minutes,
hours, days or more. You can figure out the delta (including one-way
transit time, which will vary from packet to packet!) between sender and
receiver (given access to the RTCP, note below).

>
> How does Chrome handles clock drift between the sender and the receiver?

This is handled by the jitter buffers and associated time-warping done
to avoid jitter buffer buildup. Basically, the mechanisms for an
adaptive jitter buffer as a side-effect will compensate for clock drift
between sender and receiver.

>
> Can I compensate the clock drift with the following?
> - Save the receiver local timestamp for each SR packet
> - Get a polynomial fit that translates SR NTP timestamp to receiver
> local timestamp.
> - Use the polynomial fit with setpts/asetpts in ffmpeg.
> https://ffmpeg.org/ffmpeg-filters.html#setpts_002c-asetpts

Maybe...

Note: unless you are a true WebRTC endpoint (and are decrypting the
packets), you have no access to the RTP/RTCP-level of media. A JS
client can manipulate the decoded streams (or feed them to
MediaRecorders, which will reencode them into media files), but can't
access the packets themselves. Recording the packets will be worthless
as you don't have access to the keys to decrypt them.

--
Randell Jesup -- rjesup a t mozilla d o t com
Please please please don't email randel...@jesup.org! Way too much spam

[pablo platt]

On Sun, Aug 2, 2015 at 9:12 AM, Randell Jesup <randel...@jesup.org> wrote:

On 8/1/2015 11:12 PM, pablo platt wrote:

I'm capturing RTP and RTCP packets from audio and video streams and merge them to a video with a post-processing script.

I've read about the RTP timestamp and NTP timestamp in RTCP SR and how to sync audio and video streams from the same source.

How can I synchronize streams from multiple participants?
I also need to synchronize the streams with other events in the recording like whiteboard annotations and presentation slides.

You can synchronize the streams coming *from* a participant (by putting them in the same MediaStream).  Synchronizing *multiple* participants means basically recording the view as seen by either the server or one participant.

I'm capturing raw RTP and RTCP packets (un-encrypted) and trying to convert them to a video with a post-processing script.

MediaStream can't help me here, unless I'll find a way to use it from the webrtc sdk.

 

rfc3550 says:
An implementation is not required to run the Network Time Protocol in
order to use RTP.  Other time sources, or none at all, may be used
(see the description of the NTP timestamp field in Section 6.4.1).
However, running NTP may be useful for synchronizing streams
transmitted from separate hosts.
Do WebRTC clients (Chrome, Firefox, OpenWebRTC) sync their clocks with NTP servers?

You should ask "do the computers users run WebRTC on sync NTP?" Likely yes, but not always.  And even synced NTP isn't perfect *between* machines.  Some machines don't sync and the time may be off by minutes, hours, days or more.  You can figure out the delta (including one-way transit time, which will vary from packet to packet!) between sender and receiver (given access to the RTCP, note below).

How does Chrome handles clock drift between the sender and the receiver?

This is handled by the jitter buffers and associated time-warping done to avoid jitter buffer buildup.  Basically, the mechanisms for an adaptive jitter buffer as a side-effect will compensate for clock drift between sender and receiver.

Is a jitter buffer relevant in offline mode when I have a file with all the RTP and RTCP packets captured?

Do I really need to broadcast the packets in the same time intervals as in the live mode and feed a jitterbuffer just to convert to a video?

Converting a 4 hours recording will take me 4 hours?

 

Can I compensate the clock drift with the following?
- Save the receiver local timestamp for each SR packet
- Get a polynomial fit that translates SR NTP timestamp to receiver local timestamp.
- Use the polynomial fit with setpts/asetpts in ffmpeg.
https://ffmpeg.org/ffmpeg-filters.html#setpts_002c-asetpts

Maybe...

Note: unless you are a true WebRTC endpoint (and are decrypting the packets), you have no access to the RTP/RTCP-level of media.  A JS client can manipulate the decoded streams (or feed them to MediaRecorders, which will reencode them into media files), but can't access the packets themselves.   Recording the packets will be worthless as you don't have access to the keys to decrypt them.

As said above, I'm capturing the RTP/RTCP packets on the MCU un-encrypted.

MediaRecorder will do double encoding and will force me to automate a real browser and stream the packets with the same intervals as the original session.

 

--
Randell Jesup -- rjesup a t mozilla d o t com
Please please please don't email randel...@jesup.org!  Way too much spam

--

--- You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/55BDB4D6.8070406%40jesup.org.
For more options, visit https://groups.google.com/d/optout.

[Randell Jesup]

On 8/2/2015 2:28 AM, pablo platt wrote:

On Sun, Aug 2, 2015 at 9:12 AM, Randell Jesup <randel...@jesup.org> wrote:

You can synchronize the streams coming *from* a participant (by putting them in the same MediaStream).  Synchronizing *multiple* participants means basically recording the view as seen by either the server or one participant.

I'm capturing raw RTP and RTCP packets (un-encrypted) and trying to convert them to a video with a post-processing script.

MediaStream can't help me here, unless I'll find a way to use it from the webrtc sdk.

Ok, then you need to deal with a) packet loss concealment, b) a/v sync (as well as sync between different streams).  Remember that timestamps won't increment at the 'nominal' rate NTP advances, and audio and video will increment at different rates vs NTP.   So sync will require compensating timestamps to match NTP, and then adjusting the streams (from a given should-be-synced source) to match up in NTP time.  Then to do overall sync, you just need to figure out the deltas for NTP between sources, and that can be done by looking at the arrival times for each stream.  In almost all reasonable cases, you can ignore NTP clock drift between sources, though for long encodings you may need to do that as well.

In general, the only data you'll need other then the RTP/RTCP packets would be time-of-arrival for each packet.  (You probably could do it with just time-of-arrival for RTCP.)

Note: this won't perfectly match what any participant sees.  It will match what a theoretical participant might have seen, more or less.

How does Chrome handles clock drift between the sender and the receiver?

This is handled by the jitter buffers and associated time-warping done to avoid jitter buffer buildup.  Basically, the mechanisms for an adaptive jitter buffer as a side-effect will compensate for clock drift between sender and receiver.

Is a jitter buffer relevant in offline mode when I have a file with all the RTP and RTCP packets captured?

Do I really need to broadcast the packets in the same time intervals as in the live mode and feed a jitterbuffer just to convert to a video?

Converting a 4 hours recording will take me 4 hours?

That's an option, and will more closely match what a participant will have seen.  However, you don't need to do so.  You *will* need to reorder packets in sequence-number order, and conceal lost (audio) packets (which is often handled along with the jitter buffer code) during decoding so it sounds right and the stream isn't missing bits of time.

I implemented a video answering machine once by recording the RTP/RTCP, and then feeding them back to the in-call code by sending them to myself at the right times (based on timestamp, not arrival time) into the jitter buffers, then let all the normal code work (jitter buffer, concealment, etc).  But you're doing something a bit different.

You could use a fixed-depth jitter buffer+PLC and feed them into that, and run "time" much faster (jitter buffers in general don't look at 'time', they look at "what sample am I at in the stream".  NetEQ for example is clocked by the calls to pull data out of it.  Thus it's possible to run the entire thing basically "as fast as everything can work".

[pablo platt]

So the needed steps are:

1. Reorder packets and drop duplicates

2. Conceal lost packets. Can use something like the audiorate element from gstreamer:

http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-audiorate.html

3. Sync audio and video streams against the sender NTP time.

Do I need to run some average or can I use the NTP-Timestamp to RTP-Timestamp conversion from SR to all the packets until the next SR?

Can I compensate by adding the offset to the timestamp in RTP packets or do I need to handle it on the decoder level?

4. Sync all streams against a master receiver time.

As you said, the drift might be small compared to A/V sync but on long recording it might be noticeable.

I can't just compensate using the last arrival_time - last NTP SR because of network jitter.

What kind of average or moving average can I use?

Can I again apply the compensation by modifying the RTP timestamp in the packets?

 

-- 
Randell Jesup -- rjesup a t mozilla d o t com
Please please please don't email randel...@jesup.org!  Way too much spam

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/55BE08E6.90208%40jesup.org.

[Randell Jesup]

On 8/2/2015 12:48 PM, pablo platt wrote:
>
>
> So the needed steps are:
> 1. Reorder packets and drop duplicates
> 2. Conceal lost packets. Can use something like the audiorate element
> from gstreamer:
> http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-audiorate.html

I doubt that will work. However this is a common procedure for anything
that decodes RTP audio. Worst case you can insert silence for lost
packets, but that will suck - repeating the last 10ms (with care to
avoid a glitch at entry/exit!) is better, though not great.

> 3. Sync audio and video streams against the sender NTP time.
> Do I need to run some average or can I use the NTP-Timestamp to
> RTP-Timestamp conversion from SR to all the packets until the next SR?

I'd try to avoid jumps, but honestly that's probably fine as the SR's
aren't far apart.

> Can I compensate by adding the offset to the timestamp in RTP packets
> or do I need to handle it on the decoder level?

Decoders don't know or care about timestamps.

> 4. Sync all streams against a master receiver time.
> As you said, the drift might be small compared to A/V sync but on long
> recording it might be noticeable.
> I can't just compensate using the last arrival_time - last NTP SR
> because of network jitter.
> What kind of average or moving average can I use?

Up to you. There are many ways to approach this.

> Can I again apply the compensation by modifying the RTP timestamp in
> the packets?

Depends on how you process all this. Sorry, you're in the area where
you need to experiment or look at your code.

[pablo platt]

Thanks

--

--- You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/55C03B81.7010004%40jesup.org.