Intent to Deprecate and Remove: 'stalled' events for HTMLMediaElements using MediaSource

[Chrome Cunningham]

Primary eng

chcunn...@chromium.org

Summary

The 'stalled' event is raised if the media download has failed to progress for ~3 seconds. When using MSE, the web app manages the download (xhr) and the media element is not aware of its progress.

Web authors using MSE have noticed that 'stalled' seems to be emitted at random times.

https://bugs.chromium.org/p/chromium/issues/detail?id=517240

Due to code cruft, Chrome will raise 'stalled' whenever app has not appended new media (SourceBuffer.Append(...)) in the last 3 seconds. Apps may choose to append large chunks of data at a much lower frequency, so this is not a useful signal about buffering health.

Removing the 'stalled' for MSE should clear up confusion and bring us more in line with the spec (https://github.com/w3c/media-source/issues/88#issuecomment-374406928).

Media elements that don't use MSE will continue to raise 'stalled' as they do today.

Interoperability and Compatibility Risk

All browsers bellow appear to have the same behavior as current Chrome.

Edge: Haven't heard back yet

Firefox: Supportive (https://bugzilla.mozilla.org/show_bug.cgi?id=1451149

Safari: Supportive (email)

Alternative implementation suggestion for web developers

MSE users typically fetch the media chunks via XHR. XHR provides a number of events for monitoring the progress of the download.

https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Using_XMLHttpRequest#Monitoring_progress

Usage information

We don't have a good use-counter for this. Folks probably are listening for the event (e.g. unknowingly as part of a video streaming library). But listening != meaningful usage.

I polled the video streaming community at video-dev.slack.com (developers representing many major streaming sites/platforms). None were using 'stalled'.

One nuance is the opposite event: 'progress', which is not being deprecated. For non-MSE media elements, 'progress' indicates the download is moving along. For MSE, 'progress' has come to mean that the range of what's buffered has changed. This meaning is not in the spec, but has been baked into Chrome's implementation since the MSE shipped and other UAs seem to do the same. In a separate video-dev poll, many web authors using MSE indicated they listen for 'progress' as a trigger to update the part of their custom UI that shows how much of the stream is buffered. There are alternative means of achieving this, but breaking these folks is not worth the cost - 'progress' can stay.

Entry on the feature dashboard

None. This is a very small removal. Happy to create one if desired.

Requesting approval to remove too?

Yes, in M68.

[Anne van Kesteren]

On Mon, Apr 23, 2018 at 10:23 PM, Chrome Cunningham
<chcunn...@chromium.org> wrote:
> The 'stalled' event is raised if the media download has failed to progress
> for ~3 seconds. When using MSE, the web app manages the download (xhr) and
> the media element is not aware of its progress.

I checked with the HTML Standard and it seems it's only supposed to
fire for remote resources. Will you also be removing it for File and
MediaStream objects?

Are there web-platform-tests?


--
https://annevankesteren.nl/

[Joe Medley]

Can you please create a status entry for this? 

Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195

If an API's not documented it doesn't exist.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c9c57bdc-e3f3-4cb7-bb34-ea01b58b64fb%40chromium.org.

[Mike West]

Removing without warning seems risky in the absence of any data about usage. Do we have a feel for the origins where MSE sees large-scale usage? Have we verified that the lack of this event won't break those sites?

It also seems a little strange that we'd distinguish between MSE content and non-MSE content for this event. If it isn't useful (or used) for MSE content, it might be possible to remove the event entirely rather than bifurcating our behavior based on (seemingly?) unrelated qualities of the content being rendered. Did y'all consider that possibility?

-mike

[Chrome Cunningham]

+Marijn for "File"

+Miguel for "MediaStream"

> I checked with the HTML Standard and it seems it's only supposed to 
fire for remote resources. Will you also be removing it for File and 
MediaStream objects? 

I agree with your take on the standard. I had planned to only change MediaSource. This deprecation is a low priority technical debt cleanup - IMO, its only worth pursuing because I'm confident that the risk is very small (demonstrated by wide community survey and my inability to create even a hypothetical breakage). I'm not familiar enough to say whether that's true for the other sources.

> Are there web-platform-tests?

There are no tests that expect this event to be fired for MSE. There is one test that explicitly expects it is NOT fired. The ideal post-deprecation test would verify that the event is NEVER fired. This is a bit difficult ;). We could write a test that waited for 3 seconds (ugh) between sourceBuffer appends to ensure that at least the current firing behavior has gone away. Not sure if that's worth doing.

> Can you please create a status entry for this? 

Sure. https://www.chromestatus.com/feature/6338037575319552

> Do we have a feel for the origins where MSE sees large-scale usage? Have we verified that the lack of this event won't break those sites?

We do have this data. I have not systematically verified for the top N sites, but the poll I performed in the video-dev community gave us really good coverage. In no particular order, we heard "won't break" from devs representing:

Netflix, Vimeo, Brighcove, Ingest.io, Soundcloud, Jwplayer, RealEyes, NyTimes.com, Bitmovin, DailyMotion

+strobe@ for YouTube (haven't asked yet)

> Removing without warning

We could add a console warning (e.g. for any MSE <video> element with 'stalled' event listeners) if this would address your concern? IMO this is not needed.

> If it isn't useful (or used) for MSE content, it might be possible to remove the event entirely rather than bifurcating our behavior based on (seemingly?) unrelated qualities of the content being rendered. Did y'all consider that possibility?

MSE vs non-MSE (or in spec-speak: 'local' vs 'remote' resources) is a meaningful distinction and bifurcating in this way is what the spec would have us do. The 'stalled' event has solid meaning/use-case for non-MSE playbacks where the UA manages the download. No desire deprecate for those.  

Chris

[Marijn Kruisselbrink]

On Wed, Apr 25, 2018 at 2:39 PM, Chrome Cunningham <chcunn...@chromium.org> wrote:

+Marijn for "File"

Not sure what the question is? Following the spec better sounds good to me, but I'm not familiar with the media side of things. So I guess that would mean not firing "stalled" events for any srcObject or valid blob: URL?

[Anne van Kesteren]

On Wed, Apr 25, 2018 at 11:50 PM, Marijn Kruisselbrink <m...@chromium.org> wrote:
> Not sure what the question is? Following the spec better sounds good to me,
> but I'm not familiar with the media side of things. So I guess that would
> mean not firing "stalled" events for any srcObject or valid blob: URL?

Yeah, that's what the HTML Standard requires.


--
https://annevankesteren.nl/

[Daniel Bratell]

I support this change, but just a question/request. Is it possible to find stalled event handlers for MSE media in a code search (at github, in Google's archive if you have access to that or somewhere else) to get an idea if they even exist?

Good that you added a status entry for this. Even if it's a small change, this way it will be listed in platform changelogs.

Not sure a deprecation phase is needed. Maybe a code search will tell us.

/Daniel

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/5a950f89-b6c0-45d6-a313-f47aa570fe70%40chromium.org.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

[Chrome Cunningham]

Miguel, friendly ping for thoughts on MediaStream.

Marijn: Are you of aware of any use case for 'stalled' with File? Any objection to not firing it as the spec suggests? 

I can help from the media side. Do you have any examples of media playbacks using srcObject=File? It may be that this event is already not fired in this case. 

> +strobe@ for YouTube (haven't asked yet)

Strobe replied back, but he isn't part of this group. YouTube is not using 'stalled' with MSE. 

> Is it possible to find stalled event handlers for MSE media in a code search (at github, in Google's archive if you have access to that or somewhere else) to get an idea if they even exist?

I did not find any uses of 'stalled' with MSE in google's codebase. Its not trivial to search all of github... need to narrow it to an owner, or repo. 

I suspect it *is* possible to find people who are listening,  but this is a red herring. In the examples I have where folks listen to this event with MSE, they expected the event indicated poor video buffering health and observing the event triggered various metrics reporting. This usage is simply wrong. Eliminating that confusion is the main goal of this cleanup. 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/5a950f89-b6c0-45d6-a313-f47aa570fe70%40chromium.org.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/5a950f89-b6c0-45d6-a313-f47aa570fe70%40chromium.org.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/5a950f89-b6c0-45d6-a313-f47aa570fe70%40chromium.org.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/5a950f89-b6c0-45d6-a313-f47aa570fe70%40chromium.org.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/5a950f89-b6c0-45d6-a313-f47aa570fe70%40chromium.org.

[Harald Alvestrand]

httparchive contains 13 mentions of "onstalled" together with "srcObject".

I think this is small enough to be able to say that "there's no significant usage of onstalled together with MediaStreams".

The most common pattern in this set of 13 was a line saying:

declare var onstalled: (this: Window, ev: Event) => any

which is not comprehensible to me, but "feels" like setting up handlers for "anything else that happens".

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/a2c18fbf-7126-4c32-ac16-0acb15b620ba%40chromium.org.

[Miguel Casas]

On 26 April 2018 at 19:30, Chrome Cunningham <chcunn...@chromium.org> wrote:

Miguel, friendly ping for thoughts on MediaStream.

 I have no strong opinions about it: from the POV of capture-from-element, this event is irrelevant; +guidou@ for a <video> with a MediaStream as feeder.

--

Miguel Casas-Sanchez |

Gatopardo del Software |

ydog / mca...@google.com |

+1 (650) 603 1380

[Steven Robertson]

yeah we don't even listen to it in YT

[Marijn Kruisselbrink]

On Thu, Apr 26, 2018 at 4:30 PM, Chrome Cunningham <chcunn...@chromium.org> wrote:

Miguel, friendly ping for thoughts on MediaStream.

Marijn: Are you of aware of any use case for 'stalled' with File? Any objection to not firing it as the spec suggests? 

I can help from the media side. Do you have any examples of media playbacks using srcObject=File? It may be that this event is already not fired in this case. 

Sorry, no idea. Not firing as the spec suggests sounds good to me. In practice I expect it would indeed never be fired for blobs since unless something really funky is going on reading from memory and/or a file on disk should never stall for multiple seconds anyway...

[Chris Harrelson]

LGTM1

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/5a950f89-b6c0-45d6-a313-f47aa570fe70%40chromium.org.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BOSsVZ9VhsHWw3HtT9qzZeH3J3G8PJCFDGySnPFyK5Tq6OiTw%40mail.gmail.com.

[Chrome Cunningham]

> I checked with the HTML Standard and it seems it's only supposed to 
fire for remote resources. Will you also be removing it for File and 
MediaStream objects? 

Good news - neither File nor MediaStream objects currently fire stalled event.

In the File case, the HTMLMediaElement.networkState transitions to "idle" almost immediately, which cancels the associated timers. 

In the MediaStream case, the underlying media player (WebMediaPlayerMS) hardcodes the DidLoadingProgress() to return true, so the media element will never 

consider loading to have "stalled". 

On Monday, April 30, 2018 at 1:46:12 PM UTC-7, Chris Harrelson wrote:

LGTM1

Woot! Awaiting 2 more. 

LGTM1

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

...

[Daniel Bratell]

LGTM2

/Daniel

LGTM1

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

...

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/1d1d1eeb-2467-4112-a070-1bebd65f95b5%40chromium.org.

[Yoav Weiss]

LGTM3

Please add WPT tests that make sure this event never fires (timeout based tests are not great, but no tests is worse...)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/op.zidqp5twrbppqq%40cicero2.linkoping.osa.

[Chrome Cunningham]

Will do!

https://chromium-review.googlesource.com/c/chromium/src/+/982564/4/third_party/WebKit/LayoutTests/external/wpt/media-source/mediasource-append-buffer.html

LGTM2

/Daniel

LGTM1

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

...

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/1d1d1eeb-2467-4112-a070-1bebd65f95b5%40chromium.org.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[chcunn...@google.com]

FYI - this is landed now, but didn't quite make M68. Will go out with M69.

[mohan...@gmail.com]

Why is this autoplay feature not working??

[gawa...@gmail.com]

.