Intent to Ship: Adding captureTimestamp and senderCaptureTimeOffset to RTCRtpContributingSource.

230 views
Skip to first unread message

Minyue Li

unread,
Mar 10, 2020, 6:22:32 PM3/10/20
to blink-dev

Intent to Ship: Adding captureTimestamp and senderCaptureTimeOffset to RTCRtpContributingSource.


Contact emails

min...@chromium.org,ch...@google.com,hb...@chromium.org


Explainer

https://w3c.github.io/webrtc-extensions/#rtcrtpcontributingsource-dictionary



Design docs/spec

Specification: https://w3c.github.io/webrtc-extensions/#rtcrtpcontributingsource-dictionary


https://w3c.github.io/webrtc-extensions/#rtcrtpcontributingsource-dictionary


TAG review



Summary

Two new data, captureTimestamp and senderCaptureTimeOffset, have been proposed to WebRTC Extensions. See https://w3c.github.io/webrtc-extensions/#rtcrtpcontributingsource-dictionary. They can be used to measure A/V sync and end-to-end delay in real-time communication (RTC) systems.


Motivation

CaptureTimestamp and senderCaptureTimeOffset are introduced to overcome the difficulty of measuring A/V sync and end-to-end delay in an RTC system is that one or more RTCP-terminating sub-system are involved. In such a system, the round trip time measurement on RTCRemoteInboundRtpStreamStats only accounts for the "last hop": the connection between the receiver and the nearest RTCP-terminating sub-system that it receives RTCP packets from. This makes it hard to estimate the offset between the receiver's clock and the capturer's clock, and thus end-to-end delay.


CaptureTimestamp and senderCaptureTimeOffset, as originated from RTP Header Extension for Absolute Capture Time, see https://github.com/webrtc/webrtc-org/blob/gh-pages/experiments/rtp-hdrext/abs-capture-time/index.md can mitigate the problem.


Risks

Interoperability and Compatibility

The header extension that the two new data are based on is negotiated through SDP, and thus adding the two new data won't break Interoperability.


The two new data are mainly used for measuring the quality of an RTC system, and the risk of breaking a web application is small.


The proposal has been presented to W3C WG and browser implementers. While there is no signal from other implementers to implement this yet, the proposal has been reviewed without concerns raised.


Will this feature be supported on all six Blink platforms (Windows, Mac, Linux,

Chrome OS, Android, and Android WebView)?

Yes


Is this feature fully tested by web-platform-tests?

Yes, tests will be added as part of implementation efforts, can be reviewed on https://chromium-review.googlesource.com/c/chromium/src/+/2072161


Link to entry on the Chrome Platform Status

https://www.chromestatus.com/feature/5728533701722112


Shipping plan

It is planned that captureTimestamp and gets shipped in Chrome M82 and senderCaptureTimeOffset gets shipped slightly later in Chrome M83.


Philip Jägenstedt

unread,
Mar 11, 2020, 10:12:52 AM3/11/20
to Minyue Li, blink-dev
A few questions inline:

On Tue, Mar 10, 2020 at 11:22 PM 'Minyue Li' via blink-dev <blin...@chromium.org> wrote:

Intent to Ship: Adding captureTimestamp and senderCaptureTimeOffset to RTCRtpContributingSource.


Contact emails

min...@chromium.org,ch...@google.com,hb...@chromium.org


Explainer

https://w3c.github.io/webrtc-extensions/#rtcrtpcontributingsource-dictionary

Should there be something in https://github.com/w3c/webrtc-extensions/blob/master/explainer.md for this whole spec, and these additions in particular?

An explanation of the use case and examples of use would be great. From reading the spec itself, my best guess is that this is intended for some broadcasting scenario with a bunch of equipment ("RTCP-terminating intermediate systems") but I'm also pretty sure that's not a correct guess :)
Did this spec as a whole go through TAG review in the past? I searched https://github.com/w3ctag/design-reviews/issues a bit but couldn't find anything.

Summary

Two new data, captureTimestamp and senderCaptureTimeOffset, have been proposed to WebRTC Extensions. See https://w3c.github.io/webrtc-extensions/#rtcrtpcontributingsource-dictionary. They can be used to measure A/V sync and end-to-end delay in real-time communication (RTC) systems.


Motivation

CaptureTimestamp and senderCaptureTimeOffset are introduced to overcome the difficulty of measuring A/V sync and end-to-end delay in an RTC system is that one or more RTCP-terminating sub-system are involved. In such a system, the round trip time measurement on RTCRemoteInboundRtpStreamStats only accounts for the "last hop": the connection between the receiver and the nearest RTCP-terminating sub-system that it receives RTCP packets from. This makes it hard to estimate the offset between the receiver's clock and the capturer's clock, and thus end-to-end delay.


CaptureTimestamp and senderCaptureTimeOffset, as originated from RTP Header Extension for Absolute Capture Time, see https://github.com/webrtc/webrtc-org/blob/gh-pages/experiments/rtp-hdrext/abs-capture-time/index.md can mitigate the problem.

Has that RTP header extension already been shipped, or is it part of this intent? 

Risks

Interoperability and Compatibility

The header extension that the two new data are based on is negotiated through SDP, and thus adding the two new data won't break Interoperability.


The two new data are mainly used for measuring the quality of an RTC system, and the risk of breaking a web application is small.


Indeed the compat risk (for web applications) is effectively zero because this is adding members to an existing dictionary. Compat risk is more of a consideration for deprecations/removal.

The interop risk, however, is the risk that this won't be implemented by most/all browsers in the long run, and so has mostly to do with buy-in from other browser vendors. See below.

(The term interoperability is also used to mean two implementations of a protocol successfully communicating, and the way we use it on the web platform is arguably weird.)

The proposal has been presented to W3C WG and browser implementers. While there is no signal from other implementers to implement this yet, the proposal has been reviewed without concerns raised.

Can you link to the PR or issue where that happened? Is your expectation that others will in fact implement this?

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux,

Chrome OS, Android, and Android WebView)?

Yes


Is this feature fully tested by web-platform-tests?

Yes, tests will be added as part of implementation efforts, can be reviewed on https://chromium-review.googlesource.com/c/chromium/src/+/2072161

Looks like you have detailed test review there :)

Aside with API owners hat off: it looks like there's no testing for when captureTimestamp would be missing, somewhat related to https://github.com/w3c/webrtc-extensions/issues/34. Making sure that case is covered would be good because it should match what happens when the remote is any browser (including current Chrome) that doesn't have the RTP extension yet.

Link to entry on the Chrome Platform Status

https://www.chromestatus.com/feature/5728533701722112


Shipping plan

It is planned that captureTimestamp and gets shipped in Chrome M82 and senderCaptureTimeOffset gets shipped slightly later in Chrome M83.


--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/e4a19d48-9c68-4c96-b9ed-efdd055e34c4%40chromium.org.

Henrik Boström

unread,
Mar 11, 2020, 1:26:41 PM3/11/20
to blink-dev, Philip Jägenstedt, blink-dev, Minyue Li
On Wednesday, March 11, 2020 at 3:12:52 PM UTC+1 Philip Jägenstedt wrote:
A few questions inline:

On Tue, Mar 10, 2020 at 11:22 PM 'Minyue Li' via blink-dev <blin...@chromium.org> wrote:

Intent to Ship: Adding captureTimestamp and senderCaptureTimeOffset to RTCRtpContributingSource.


Contact emails

Should there be something in https://github.com/w3c/webrtc-extensions/blob/master/explainer.md for this whole spec, and these additions in particular?

See the Introduction section of the spec explaining that these are small additions to the webrtc-pc spec. We should add an explainer.md file too saying something similar!


An explanation of the use case and examples of use would be great. From reading the spec itself, my best guess is that this is intended for some broadcasting scenario with a bunch of equipment ("RTCP-terminating intermediate systems") but I'm also pretty sure that's not a correct guess :)
Did this spec as a whole go through TAG review in the past? I searched https://github.com/w3ctag/design-reviews/issues a bit but couldn't find anything.

The main spec (webrtc-pc) has gone through TAG review in the past, and these are just additions to existing API surfaces. If they were added to the main spec I don't think that would have warranted re-requesting TAG review for each new addition? Or is this something that should be done on a per-feature basis?

That said this is a new document has not gone through TAG review as a whole. There was a TAG started for playoutDelay here but that was not completed after the engineer moved on to other things.

Let me know if we should TAG the whole doc or this feature in particular, or if relying on webrtc-pc's TAG review (from years ago) is fine.

Summary

Two new data, captureTimestamp and senderCaptureTimeOffset, have been proposed to WebRTC Extensions. See https://w3c.github.io/webrtc-extensions/#rtcrtpcontributingsource-dictionary. They can be used to measure A/V sync and end-to-end delay in real-time communication (RTC) systems.


Motivation

CaptureTimestamp and senderCaptureTimeOffset are introduced to overcome the difficulty of measuring A/V sync and end-to-end delay in an RTC system is that one or more RTCP-terminating sub-system are involved. In such a system, the round trip time measurement on RTCRemoteInboundRtpStreamStats only accounts for the "last hop": the connection between the receiver and the nearest RTCP-terminating sub-system that it receives RTCP packets from. This makes it hard to estimate the offset between the receiver's clock and the capturer's clock, and thus end-to-end delay.


CaptureTimestamp and senderCaptureTimeOffset, as originated from RTP Header Extension for Absolute Capture Time, see https://github.com/webrtc/webrtc-org/blob/gh-pages/experiments/rtp-hdrext/abs-capture-time/index.md can mitigate the problem.

Has that RTP header extension already been shipped, or is it part of this intent?

RTP Header extensions have historically been shipped by modifying the SDP protocol parser in WebRTC, which has automatically been imported into Chromium, without Chromium changes needed.
If I were to guess, this is probably why RTP header extensions have - historically speaking - not gone through the blink intent-to-ship process.

If header extension needs intent-to-ship approval, this intent thread is as good as any to cover the new header extension. The header extension code is new, but it may already have landed?

Risks

Interoperability and Compatibility

The header extension that the two new data are based on is negotiated through SDP, and thus adding the two new data won't break Interoperability.


The two new data are mainly used for measuring the quality of an RTC system, and the risk of breaking a web application is small.


Indeed the compat risk (for web applications) is effectively zero because this is adding members to an existing dictionary. Compat risk is more of a consideration for deprecations/removal.

The interop risk, however, is the risk that this won't be implemented by most/all browsers in the long run, and so has mostly to do with buy-in from other browser vendors. See below.

While we hope that it will be implemented by other browsers (see below), with regards to interop: if it wasn't implemented the result would be that you would have less metrics available measuring quality - the web application would still be fully functional without it.
 

(The term interoperability is also used to mean two implementations of a protocol successfully communicating, and the way we use it on the web platform is arguably weird.)

The proposal has been presented to W3C WG and browser implementers. While there is no signal from other implementers to implement this yet, the proposal has been reviewed without concerns raised.

Can you link to the PR or issue where that happened? Is your expectation that others will in fact implement this?

The webrtc-extensions spec has been adopted by the W3C WebRTC Working group and changes to it are reviewed. The idea is that features will mature and after "WebRTC 1.0" is in Proposed Recommendation status, we will start migrating features from webrtc-extensions into webrtc-pc. In the meantime, commitment of individual features are evaluated on a case-by-case basis. Currently resources are mostly spent on implementing webrtc-pc, but we need to add new features as well as polishing old ones.

This feature was discussed at an editor's meeting with the W3C WebRTC Working Group. The PR is here and it got "editors can integrate" label based on discussions from that meeting, but there are no comments in text. This was also later presented at W3C WebRTC Virtual Interim with Chrome, Firefox and Edge representatives presents, as well as participants working with the WebRTC platform. The slides are here. The *hope* is that this will get implemented by other browsers but we do not have expressed commitments from anybody at this time.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux,

Chrome OS, Android, and Android WebView)?

Yes


Is this feature fully tested by web-platform-tests?

Yes, tests will be added as part of implementation efforts, can be reviewed on https://chromium-review.googlesource.com/c/chromium/src/+/2072161

Looks like you have detailed test review there :)

Aside with API owners hat off: it looks like there's no testing for when captureTimestamp would be missing, somewhat related to https://github.com/w3c/webrtc-extensions/issues/34. Making sure that case is covered would be good because it should match what happens when the remote is any browser (including current Chrome) that doesn't have the RTP extension yet.

Good idea - will be done. 

Link to entry on the Chrome Platform Status

https://www.chromestatus.com/feature/5728533701722112


Shipping plan

It is planned that captureTimestamp and gets shipped in Chrome M82 and senderCaptureTimeOffset gets shipped slightly later in Chrome M83.


--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

sligh...@chromium.org

unread,
Mar 12, 2020, 3:42:33 PM3/12/20
to blink-dev, foo...@chromium.org, min...@google.com
Hey all,

OWNERS met today. There seem to be some issues with this intent:
  • No TAG review; we ask for these to be filed because even small changes can have large impacts
  • No Intent-to-Implement cited. I can't find a previous thread on this topic and it isn't listed in the mails above.
  • The link to an "explainer" is actually a link to spec text. We expect Explainers to be separate documents that use (or start with) the TAG template.
  • Per Philip's note about interop risk, would like to understand better if there are counter-proposals? What is the developer interest level? The API OWNERS are generally happy to let things move forward absent other vendor support so long as we can show that we've done the homework (e.g. TAG review), are being responsible (tests, spec language), and have a strong developer need that we can demonstrate our additions resolve. Outlining the need is something an Explainer is often helpful for.
Perhaps a standalone explainer can help resolve a lot of this?

Regards

Minyue Li

unread,
Mar 12, 2020, 4:14:59 PM3/12/20
to blink-dev, sligh...@chromium.org, Philip Jägenstedt, Minyue Li
Thanks a lot for looking into this!

We'll address these issues shortly.

Minyue Li

unread,
Mar 31, 2020, 5:55:36 PM3/31/20
to blink-dev, Minyue Li, sligh...@chromium.org, Philip Jägenstedt
Hi,


Please also find answers to previous questions inline.


On Thursday, March 12, 2020 at 9:14:59 PM UTC+1 Minyue Li wrote:
Thanks a lot for looking into this!

We'll address these issues shortly.

On Thursday, March 12, 2020 at 8:42:33 PM UTC+1 sligh...@chromium.org wrote:
Hey all,

OWNERS met today. There seem to be some issues with this intent:
  • No TAG review; we ask for these to be filed because even small changes can have large impacts
  • No Intent-to-Implement cited. I can't find a previous thread on this topic and it isn't listed in the mails above.
 Actually, we want this intent to be an intent-to-implement-and-ship.

  • The link to an "explainer" is actually a link to spec text. We expect Explainers to be separate documents that use (or start with) the TAG template.
  • Per Philip's note about interop risk, would like to understand better if there are counter-proposals? What is the developer interest level? The API OWNERS are generally happy to let things move forward absent other vendor support so long as we can show that we've done the homework (e.g. TAG review), are being responsible (tests, spec language), and have a strong developer need that we can demonstrate our additions resolve. Outlining the need is something an Explainer is often helpful for.
Real applications, Google Hangouts and Meet, for example, have been asking for reliable audio video synchronization and end-to-end delay measurements for several years, which we can back up with discussions from 2017, see w3c/webrtc-stats#158
 
The lack of signals from other implementors is strongly related to the lack of resources. When presented to the working group, the feedback is basically "ok, that makes sense" and no opposition to us implementing it. But no commitments are given by other vendors to implement the feature. Not because it doesn't make sense but because they are busy with WebRTC 1.0 and other higher priority issues, so no promises are given and they are basically remaining neutral. With all extensions, the tentative plan is to revisit all of the extension features in the future and see which ones we want to move over to the main spec, but this is not happening any time soon. We still need to make progress, though. We can't wait for years for other browsers to maybe have resources.

Is there a problem worth solving? Yes.
Does this adequately address the problem? Yes.
Has it been reviewed by experts in this field? Yes.
Is it ready to be implemented? Yes.
Do other vendors have the resources to implement it too? Not now, maybe in the future, maybe not.
Will other browsers not having this cause significant interoperability problems? No.
Would having this reviewed by more experts help with the lack of resources problem? No. 
 
Perhaps a standalone explainer can help resolve a lot of this?

Regards

On Wednesday, March 11, 2020 at 10:26:41 AM UTC-7, Henrik Boström wrote:
On Wednesday, March 11, 2020 at 3:12:52 PM UTC+1 Philip Jägenstedt wrote:
A few questions inline:

On Tue, Mar 10, 2020 at 11:22 PM 'Minyue Li' via blink-dev <blin...@chromium.org> wrote:

Intent to Ship: Adding captureTimestamp and senderCaptureTimeOffset to RTCRtpContributingSource.


Contact emails

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

Chris Harrelson

unread,
Apr 2, 2020, 3:19:42 PM4/2/20
to Minyue Li, blink-dev, sligh...@chromium.org, Philip Jägenstedt
Thanks for this additional work, TAG review and explainer!

LGTM1

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/b1e5305f-379f-4a58-938e-b9bc0c1fbd4c%40chromium.org.

sligh...@chromium.org

unread,
Apr 5, 2020, 9:40:04 PM4/5/20
to blink-dev, min...@google.com, sligh...@chromium.org, foo...@chromium.org
LGTM2.

I've asked in private mail for some clarifications, particularly around the potential impact of the additional API for partners and users, but am grateful for the quick work on a well-written Explainer. 

Regards
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Mike West

unread,
Apr 6, 2020, 2:53:01 AM4/6/20
to Alex Russell, blink-dev, min...@google.com, Philip Jägenstedt
LGTM3.

-mike


To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/087b78e1-cb83-41cb-827b-5c490779fe78%40chromium.org.
Reply all
Reply to author
Forward
0 new messages