field trial with VP8 simulcast temporal layer behaviour

559 views
Skip to first unread message

Philipp Hancke

unread,
Feb 17, 2020, 6:30:49 AM2/17/20
to discuss...@googlegroups.com, Òscar Divorra Escoda

Hi,


I think a field trial that is being conducted in Chrome is affecting interoperability of media servers.


After Oscar Divorra told me that the VP8 simulcast/svc behaviour seems to have changed in ways affecting the number of temporal layers sent I've just bisected and came up with this.


You are probably looking for a change made after 694936 (known good), but no later than 694939 (first known bad).


The changelog for that is here:

https://chromium.googlesource.com/chromium/src/+log/a515bf3d443e188e92d500321b4c8c6803e41c0c..08ccf9cf53cbe091f447e39369360735503f7152


Sadly the issue is only accessible to Googlers so the rationale is unknown.

  https://bugs.chromium.org/p/chromium/issues/detail?id=1000023


While this commit seems to have been only affecting the tests, it pointed to the right field trial flag:

  --force-fieldtrials=WebRTC-VP8ConferenceTemporalLayers/3

restores the previous behaviour with three temporal layers that I was expecting.


chrome://version/?show-variations-cmd shows that this is currently field-trialed as

  WebRTC-VP8ConferenceTemporalLayers/2_V1


Can this trial be stopped?


Are there any plans to run similar field trials in the near future? It makes sense to announce it as PSAs in discuss-webrtc so that the media server vendors here can be prepared and/or evaluate the impact.


cheers


Philipp

Harald Alvestrand

unread,
Feb 17, 2020, 7:14:49 AM2/17/20
to discuss...@googlegroups.com, Òscar Divorra Escoda
Can you file a bug describing the nature of the interoperability issue?

The experimentation seems to have indicated that the resultant video (after turning on 2-layer) was of better quality than the 3-layer version.
It shouldn't affect the ability to receive 3-layer from other participants.


--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CADxkKiJsojwSFV%2BSk1eXx75Y7D3bGTsZrf63RuLQ0eoPfH-_Gg%40mail.gmail.com.

Sergio Garcia Murillo

unread,
Feb 17, 2020, 7:38:23 AM2/17/20
to discuss...@googlegroups.com

Harald, is your work on scalabilityModes functional now? it should be possible to set "L1T3" to restore previous behavior.


RTCRtpSender.getCapabilities("video").codecs[0]
  1. {clockRate: 90000, mimeType: "video/VP8", scalabilityModes: Array(2)}
    1. clockRate: 90000
    2. mimeType: "video/VP8"

Harald Alvestrand

unread,
Feb 17, 2020, 7:44:54 AM2/17/20
to discuss...@googlegroups.com
It's still on experimental status, so you have to start Chrome with --enableExperimentalFeatures (or whatever the flag is called this week) to use it.


To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/ffaf6e2b-2c23-5a6e-2dab-174cb67d99a4%40gmail.com.

Sergio Garcia Murillo

unread,
Feb 17, 2020, 8:08:02 AM2/17/20
to discuss...@googlegroups.com
Maybe it is a good time for pushing it forward (again), is there anything that we can do for help?

Best regards
Sergio

Philipp Hancke

unread,
Feb 17, 2020, 8:31:45 AM2/17/20
to discuss...@googlegroups.com, Òscar Divorra Escoda
Harald,

Am Mo., 17. Feb. 2020 um 13:14 Uhr schrieb 'Harald Alvestrand' via discuss-webrtc <discuss...@googlegroups.com>:
Can you file a bug describing the nature of the interoperability issue?

Can do once the bug is actionable. This is more a process issue, rolling out such a change without notification is not very open.

The experimentation seems to have indicated that the resultant video (after turning on 2-layer) was of better quality than the 3-layer version.

Simulcast with temporal layers is always happening in a system with a SFU. I am not sure how you can an evaluate a field trial against many different SFUs and get results that are good.
Temporal layers are used to fine-tune to the receivers available bandwidth, somewhat described in https://webrtchacks.com/sfu-simulcast/ or https://webrtchacks.com/chrome-vp9-svc/ (vp9 is a bit different but some principles still apply).

In my local environment I can see quite a change in the behaviour. With three streams things work as expected:
image.png
(top: sending bitrate and frame width; bottom: video element, receiving bitrate, receiving framerate)
I get a low spatial layer with three resulting streams that have three different bitrates (the limits are a bit off but that is due to the exclusion of header bytes i think)

The same scenario with just two layers breaks this ability:
image.png
As you can see, i only get a single output stream with the same server code.
Also note that Chrome will behave different from Edgium and Firefox +
Safari which makes server-side adaption harder.

This is in the lab.In the field it would take quite some time to evaluate the impact. For a single SFU deployment.
I'm open to the possibility that two layers are enough and result in better quality.
The path toward that is however not secret experimentation but shipping the API Sergio mentioned and then careful evaluation. Happy to help.

It shouldn't affect the ability to receive 3-layer from other participants.

Only one layer can be received in simulcast :-)

cheers

Philipp

Oscar

unread,
Feb 17, 2020, 8:35:53 AM2/17/20
to discuss-webrtc
Indeed, something of the kind scalabilityModes capability is something really missing, as different usecases may have different layering needs.
Alternatively, I personally would not even mind at all something of the kind numTemporalLayers in RTCRtpEncodingParameters .

Thanks for the insight, Harald.
It is clear that, for video encoding, the closer the reference frames are to the picture to encode, the more likely typically to improve the Rate-Distortion relation, particularly in conferencing.

However, at SFU level, different usecases may leverage the availability of more or less temporal layers to control the Frames-per-second delivered to destination. One example is to control cpu usage if needed, besides just maximizing quality. Also, quality impact of more or less layers may depend on the type of content (and thus usecase). 

What happens here is that the temporal layer full framerate factors changed from 1, 1/2, 1/4   to  1, 1/2 without further notice, and that may modify the behavior of some usecases.

This can be followed-up in the bug you propose.

Best,

  Òscar
To unsubscribe from this group and stop receiving emails from it, send an email to discuss...@googlegroups.com.
--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss...@googlegroups.com.
--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss...@googlegroups.com.
--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss...@googlegroups.com.

Oscar

unread,
Feb 17, 2020, 9:26:55 AM2/17/20
to discuss-webrtc

Tommi

unread,
Feb 18, 2020, 5:28:06 AM2/18/20
to discuss...@googlegroups.com
Hey Oscar, Philipp and Sergio,

Thanks for raising this. We're looking into it, what happened to wrt a PSA, what contributors should expect or do to prepare etc. We'll report back later today with an update on the next steps and details wrt the field trial itself.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/612cc92e-99b6-4ebf-a81a-10e34c219c00%40googlegroups.com.

Stefan Holmer

unread,
Feb 18, 2020, 10:37:15 AM2/18/20
to discuss-webrtc

Hi,


In the fall we experimented with a change in temporal layer structure for VP8 where we switched from three temporal layers to two.


Our goal was to improve video quality:

  • Two layers makes it possible to allocate more bandwidth to the base layer, improving quality.

  • A lost packet in any temporal layer with VP8 forces the receiver to fall back on the base layer, substantially reducing the framerate momentarily. Using fewer temporal layers reduces the effect of this, resulting in smoother video.


Our measurements showed an improvement and we launched this to 100% in Chrome Stable on Dec 4. We unfortunately did not send out a PSA on this in advance, sorry for this. The experiment is still around and we can roll it back if we get confirmation that this change causes problems in SFUs. It would also be good to understand what type of problems you are seeing before we decide on the next steps.


As a general suggestion, we recommend that SFUs be written in a way to not rely on a specific number of temporal layers. There are platforms where Chrome uses HW encoders that don't have temporal layer support. In which case Chrome will only send a single temporal layer.


/Stefan



To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CANDN_yU%3DoFYGBgw4_z-d%2BUoN_qxf%2BrGMeHukwuZ1cY2GdG-ZEQ%40mail.gmail.com.

Oscar

unread,
Feb 19, 2020, 6:24:21 AM2/19/20
to discuss-webrtc
Thanks for the feedback Stefan.

Given that no-one seemed to come rushing on Dec 4th with things broken, makes me suspect SFU writers know what they are doing, particularly when it comes to expectations on unannounced changes.
That doesn't mean that the change could not have an impact on use-cases and services, as one temporal layer across resolutions, i.e. a previously available and selectable FPS (and associated network use, and associated cpu use, and associated battery consumption point per decoded video) disappeared.

According to the analysis you comment from the experiment, the conclusion could easily also lead to just have 1 temporal layer to improve further quality for the bitrate sent.
But going into that direction basically defeats the purpose of scalability, as scalable coding is about trading-off coding rate-distortion performance with decoding flexibility in a number of cost/performance dimensions.

For your experiment, could you please elaborate on how was heterogeneity of usecases and decoding flexibility needs taken into account ?
Seems there is an assumption that only full-frame-rate decoding, is the target to optimize in any case, but that doesn't necessarily serve all use-cases as we know. 
What usecases and content were considered ? What applications and multiparty use heterogeneity ? 
Could you comment on how was the impact assessed of the subsequent reduction of control granularity capacity on cpu and battery consumption?

Without knowing, some applications modes of operation, at the endpoint, may now be forced to use a lower resolution for a given bitrate/BW budget available, or are forced to use way more cpu and battery when they'd need less in a given situation, etc... 

Thank you in advance.
Best,

    Òscar

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss...@googlegroups.com.

Philipp Hancke

unread,
Feb 19, 2020, 7:55:16 AM2/19/20
to discuss...@googlegroups.com
Hi Stefan,

Am Di., 18. Feb. 2020 um 16:37 Uhr schrieb 'Stefan Holmer' via discuss-webrtc <discuss...@googlegroups.com>:

Hi,


In the fall we experimented with a change in temporal layer structure for VP8 where we switched from three temporal layers to two.


Our goal was to improve video quality:

  • Two layers makes it possible to allocate more bandwidth to the base layer, improving quality.

  • A lost packet in any temporal layer with VP8 forces the receiver to fall back on the base layer, substantially reducing the framerate momentarily. Using fewer temporal layers reduces the effect of this, resulting in smoother video.


That helps in scenarios with packet loss but not in scenarios with limited bandwidth?

Our measurements showed an improvement and we launched this to 100% in Chrome Stable on Dec 4.


Sadly I ran another experiment that week so its hard to attribute changes. I am looking for an improved average framerate in cases with sporadic packet loss?

We unfortunately did not send out a PSA on this in advance, sorry for this.


This isn't the first time there is no PSA. I'm sure you remember the new-jitter-buffer crashes from early 2017 (nobody misses one-byte picture ids hopefully).
Sometimes the changes are easy to notice, like ICE failures because of mdns or the attempt to deprecate DTLS 1.0.
At other times, they result in a week long search for the root cause, like AEC3 not dealing well with certain high-end bluetooth headsets.
Switching to MediaFoundation for webcams on Windows? The device blacklist keeps growing...

What is missing is clear communication, like in the case of Unified Plan (but then the new iceConnectionState...).
I'm not saying there should be a PSA for every commit or that the Blink Launch Process should be used all the time (which would be a denial of service).
This will always be a case-by-case decision.

But changing the default number of temporal layers seems like something that is clearly covered by
  All non-trivial breaking changes should have a clear and accurate chromestatus.com entry where developers can find information on the justification for the change and the advice for accommodating to it
(from the Blink principles of web compatibility document)

Rolling it out via a finch experiment (I assume) has the disadvantage that e2e tests may not pick this up, see https://bugs.chromium.org/p/webrtc/issues/detail?id=11366#c5 for details.
 

The experiment is still around and we can roll it back if we get confirmation that this change causes problems in SFUs. It would also be good to understand what type of problems you are seeing before we decide on the next steps.


We're in a very dark area which isn't covered by the specification.
From what I see this happens both when using SDP munging to enable simulcast (which has always included temporal layers) as well as using addTransceiver.
This change comes at a time when the API to control SVC is not too far away. Wouldn't it be better to wait? (OTOH i can see the flag was added ~2 years ago so there was probably some waiting involved already)
If not: is it possible to control this via an origin trial until the API is ready?


As a general suggestion, we recommend that SFUs be written in a way to not rely on a specific number of temporal layers. There are platforms where Chrome uses HW encoders that don't have temporal layer support. In which case Chrome will only send a single temporal layer.


That suggests the number of layers should be signalled in the SDP (or alongside; i.e. queryable from sender.getCapabilities()) somehow *shudder*

Philipp

Oscar

unread,
Feb 27, 2020, 10:33:57 AM2/27/20
to discuss-webrtc
Stefan, Tommy,

    After these discussion, what next steps have you considered ?

    Looks no further comments have followed here or in the ticket since last week. 
    From those here and in https://bugs.chromium.org/p/webrtc/issues/detail?id=11366 looks to me the logical step would be to roll-back to the default behavior and follow-up making the necessary so that the proper API is exposed for everyone to configure layers at will to fit needs before changing default behavior again.
    
    Please comment.
    Best Regards,

        Òscar

/Stefan



--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss...@googlegroups.com.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss...@googlegroups.com.

Stefan Holmer

unread,
Mar 6, 2020, 3:22:01 AM3/6/20
to discuss-webrtc
Since the experiment is showing benefits in our internal testing (higher/more stable framerate especially when experiencing packet loss, less time spent cpu adapted, higher quality on higher resolutions), we would like to keep this the default on. We haven't received any issue reports up until now, so we believe this is a good thing to keep.

Again, apologies for not communicating this upfront.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/e3044264-297c-4c8a-a3ed-53d3aac95a8d%40googlegroups.com.

Philipp Hancke

unread,
Mar 6, 2020, 8:41:26 AM3/6/20
to discuss...@googlegroups.com
Am Fr., 6. März 2020 um 09:21 Uhr schrieb 'Stefan Holmer' via discuss-webrtc <discuss...@googlegroups.com>:
Since the experiment is showing benefits in our internal testing (higher/more stable framerate especially when experiencing packet loss, less time spent cpu adapted, higher quality on higher resolutions),
we would like to keep this the default on. We haven't received any issue reports up until now, so we believe this is a good thing to keep.

This takes a away a capability (see https://docs.google.com/document/d/1RC-pBBvsazYfCNNUSkPqAVpSpNJ96U8trhNkfV0v9fk/edit# in the "ease of adaptation" section) and it does so in a non-open manner.

If you think this is a good idea then please follow the established process Blink has outlined in
Shall we continue on blink-dev? I think that shipping Harald's stuff first is the right way forward here.

Again, apologies for not communicating this upfront.

not faxing the EU... yet ;-)
Reply all
Reply to author
Forward
0 new messages