Intent to Ship: WebRTC Scalable Video Coding extensions

[Harald Alvestrand]

DRFT

Contact emails

h...@chromium.org, sergio.gar...@gmail.com

Explainer

https://w3c.github.io/webrtc-svc/

Specification

https://w3c.github.io/webrtc-svc/

Design docs

https://w3c.github.io/webrtc-svc/

Summary

This extension defines a standard method for picking between possible Scalable Video Coding (SVC)  configurations on an outgoing WebRTC video track.

Blink component

Blink>WebRTC>Video

Search tags

video, svc

TAG review

https://github.com/w3ctag/design-reviews/issues/396

TAG review status

Issues addressed

Risks

None identified

Interoperability and Compatibility

This is the first implementation of the spec.

Gecko: Positive - https://github.com/mozilla/standards-positions/issues/544 

WebKit: Positive, with change request  - https://lists.webkit.org/pipermail/webkit-dev/2021-October/032008.html

https://github.com/w3c/webrtc-svc/issues/49 

Web developers: Positive (sample)

Ergonomics

This is an extension to the WebRTC 1.0 API (webrtc-pc).
Support can be detected by checking the presence of scalabilityModes values in the RTCRtpCodecCapability dictionary.

Security

None identified above what's already part of video in WebRTC in general.

Debuggability

No special considerations.

Is this feature fully tested by web-platform-tests?

Yes (as far as the API surface goes; effect on bitstream content is not tested).

Flag name

RTCSvcScalabilityMode

Requires code in //chrome?

False

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=986069

Estimated milestones

M97

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5769626174619648

This intent message was generated by Chrome Platform Status.

[Harald Alvestrand]

Note - the change request from WebKit (https://github.com/w3c/webrtc-svc/issues/49) involves removing the list of scalability modes from the codec information in WebRTC and instead using the MediaCapabilities API with the "webrtc" usage to detect whether a given scalability mode is available or not.

This would require gating this API behind the updated MediaCapabilities API, and would not give exactly the same programming paradigm (see bug).

[Fernando Serboncini]

It seems they are asking for a delay on Chrome launching this until the WebRTC WG makes a decision on it.

It's not clear from the issue that there's a consensus on the right approach there.

Did you consider launching things separately and delaying the scalability modes? Or does the whole launch make no sense without it?

[Harald Alvestrand]

The scalability modes (being able to set them) are the point of the launch.

Figuring out which of the desired ones are available seems like it would be a requirement.

[Yoav Weiss]

How long of a delay are we talking about here? Weeks? Months? Years?

[Mike West]

Friendly ping on Yoav's question about timelines.

-mike

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/1197505b-23e6-491a-8fc6-4b386cce0bcen%40chromium.org.

[Harald Alvestrand]

It seems like we don't have a strong push towards either the MediaCapabilities or the Codec Capabilities solution in the issue on the sender side (https://github.com/w3c/webrtc-svc/issues/49). This is bad for quick resolution.

In the interest of getting the uncontroversial parts shipped - what would people think of letting the Codec Capabilities list of modes remain behind the flag, and push the rest of the API to public?
Many usages of the function would work quite well with only probing for supported modes by trying to set the ones they want; we could then let the issue sort itself out in peace.

(On the receiver side, there seems to be consensus on abandoning the mode list for other reasons.)

[Philip Jägenstedt]

Hi Harald,

Can you spell out what the uncontroversial parts of this would be? Looking at the IDL in https://w3c.github.io/webrtc-svc/ it looks like it's all about modes.

My guess is that it's RTCRtpEncodingParameters's scalabilityMode, but is that right?

Best regards,

Philip

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOqqYVHmEvq6MANGA078Fa9TqQe63b3QS5icAFaLbjH34ETfmw%40mail.gmail.com.

[Philipp Hancke]

Am Mi., 8. Dez. 2021 um 17:52 Uhr schrieb Philip Jägenstedt <foo...@chromium.org>:

Hi Harald,

Can you spell out what the uncontroversial parts of this would be? Looking at the IDL in https://w3c.github.io/webrtc-svc/ it looks like it's all about modes.

My guess is that it's RTCRtpEncodingParameters's scalabilityMode, but is that right?

yeah

https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/peerconnection/rtc_rtp_encoding_parameters.idl;l=24

which is currently behind a flag.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYdU%2BkKr%3DqETzu8fBD6VmqGDQJwEuiXtn%2BKO-EtbDnquvg%40mail.gmail.com.

[Bernard Aboba]

Harald said: 

"It seems like we don't have a strong push towards either the MediaCapabilities or the Codec Capabilities solution in the issue on the sender side (https://github.com/w3c/webrtc-svc/issues/49). This is bad for quick resolution."

[BA] On the receiver/decoder side (for WebRTC-SVC, Media Capabilities and WebCodecs), we have a path forward  which involves using a referenceScaling boolean and removing scalabiltyMode advertisement and configuration.  The consensus is  reflected in the current editor's draft of WebRTC-SVC (see:  https://w3c.github.io/webrtc-svc/ ) and compatible PRs are under development for MediaCapabilities and WebCodecs. 

On the sender/encoder side, we have added the "L1T1" scalability mode and specified its use in both advertisement and encoder configuration. 

Chris can provide more details with respect to the moving parts in Media Capabilities and WebCodecs. 

Here are links to the (now resolved) WebRTC-SVC issues:

https://github.com/w3c/webrtc-svc/issues/48

https://github.com/w3c/webrtc-svc/issues/52

Here are links to related WebCodecs issues: 

https://github.com/w3c/webcodecs/issues/399

Here are links to the related Media Capabilities issues:  

https://github.com/w3c/media-capabilities/issues/182

https://github.com/w3c/media-capabilities/issues/183

https://github.com/w3c/media-capabilities/issues/185

[Philip Jägenstedt]

Hi Bernard,

Can you clarify what the consensus is on RTCRtpEncodingParameters's scalabilityMode member? That remains in https://w3c.github.io/webrtc-svc/, but will it be removed? Meanwhile, referenceScaling is only partly spec'd, there's no IDL for it but a link to https://github.com/w3c/media-capabilities/issues/182.

Harald, if you could confirm the precise API surface that you'd like to ship, that would be great.

Best regards,

Philip

[Bernard Aboba]

There seems to be agreement to add support for referenceScaling in Media Capabilities (https://github.com/w3c/media-capabilities/issues/182) so I'm assuming that a PR will follow.  

Chris - please jump in if I'm misunderstanding the consensus on Issue 182. 

RTCRtpEncodingParameters.scalabilityMode (Section 5.1) is  required in order to set the scalability mode in RTCRtpSender.setParameters() as well as addTransceiver().   There has been no discussion of removing it. 

[Harald Alvestrand]

At the moment, I think we can safely ship:

- RTCRtpEncodingParameters extension scalabilityMode (https://w3c.github.io/webrtc-svc/#dom-rtcrtpencodingparameters-scalabilitymode)

We have an open discussion on whether or not to ship this part on senders (we've decided not to ship it on receivers):

- RTCRtpCodecCapability extension scalabilityModes (https://w3c.github.io/webrtc-svc/#dom-rtcrtpcodeccapability-scalabilitymodes)

There are no mandatory-to-implement scalability modes except for L1T1 (which we need to add support for).

I think that as currently specified, feature detection can be done in the absence of the RTCRtpCodecCapability extension by setting the mode to L1T1, reading back the encoding parameters, and seeing if the mode is set.

[Chris Cunningham]

Sorry I'm late. Lots of family stuff this month. I'm about to be OOO for the holidays. 

> There seems to be agreement to add support for referenceScaling in Media Capabilities (https://github.com/w3c/media-capabilities/issues/182) so I'm assuming that a PR will follow.  

I can confirm this agreement for MediaCapabilities. I expect +Johannes Kron will send a PR to amend the MC spec. 

[Chris Harrelson]

LGTM1 to ship scalabilityMode, but not the pluralized name or referenceScaling.

Please open a new intent if you wish to ship one of the others (otherwise this intent-to-ship would be too confusing).

Thanks, and happy new year.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/f9e9006d-22be-4686-add7-1dcefe09a603n%40chromium.org.

[Daniel Bratell]

LGTM2

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw_AJUughhscpGnOQcnJBWQB0NrY-hsqd0Tag%2Bbhn0_9Cg%40mail.gmail.com.

[Yoav Weiss]

LGTM3

[Joe Medley]

When are you hoping to ship?

[Florent Castelli]

After a change of priorities last year, the launch unfortunately didn't happen.

But we are now ready to ship and are targeting M111.

The API that would be launched is:

RTCRtpEncodingParameters extension scalabilityMode (https://w3c.github.io/webrtc-svc/#dom-rtcrtpencodingparameters-scalabilitymode)

The previously mentioned change in RTCRtpCodecCapability is no longer in scope and isn't required as the functionality is also covered by the MediaCapabilities specification and has already been launched in Chrome. 

WPT coverage has been expanded and covers a lot more of the spec now: https://wpt.fyi/results/webrtc-svc?label=experimental&label=master&aligned

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/dcb36afb-827a-452b-a04c-8f946bd2fa2bn%40chromium.org.