Intent to Ex[periment: MSE-in-Workers

[Matthew Wolenetz]

Contact emails

wole...@chromium.org

Explainer

https://github.com/wicg/media-source/blob/mse-in-workers-using-handle/mse-in-workers-using-handle-explainer.md

Specification

https://github.com/w3c/media-source/pull/282

Summary

Enable Media Source Extensions (MSE) API usage from DedicatedWorker contexts to enable improved performance of buffering media for playback by an HTMLMediaElement on the main Window context. By creating a MediaSource object on a DedicatedWorker context, an application may then create an ObjectURL for it and postMessage that URL to the main thread for use in attaching to an HTMLMediaElement. The context that created the MediaSource object may then use it to buffer media.

Blink component

Blink>Media

Search tags

MSE, MediaSource, MediaSourceExtensions

TAG review

https://github.com/w3ctag/design-reviews/issues/656

TAG review status

Pending

Risks

Interoperability and Compatibility

Main interoperability risk is that other browsers may not implement it. Compatibility risk is mitigated by unchanged same-thread MSE API support and proactive feature-detectability of MSE-in-Workers with a new canConstructInDedicatedWorker attribute on the MediaSource interface.

Gecko: No signal (https://github.com/mozilla/standards-positions/issues/547)

WebKit: No signal (https://lists.webkit.org/pipermail/webkit-dev/2021-June/031916.html)

Web developers: No signals

Ergonomics

DedicatedWorker, WorkerGlobalScope and postMessage/onmessage availability is integral to using this feature.

Activation

The primary risk is the potential difficulty in refactoring existing MSE web applications to (conditionally, depending on browser support) perform buffering in a different execution context from the Window context that hosts the DOM and the media element. The benefit of potentially improved buffering performance by offloading the MSE API usage to a worker context provides motivation to overcome this challenge.

Security

Unpredictability of racing context destruction of either the main thread hosting the media element (and owning the underlying MSE media demuxer and player) or the dedicated worker thread hosting the interface to the MSE demuxer when using MSE from a worker context motivated careful design of a refcounted, outside-of-Oilpan, attachment abstraction to ensure safety of operations using locks and having a lifetime that exceeds those execution contexts. Preventing use-after-free and similar was a primary complexity of implementation and a focus area of added tests.

Goals for experimentation

* Obtain developer feedback on API ergonomics and implementation stability. * Increase confidence in benefits of feature (exposing and enabling MSE usage from DedicatedWorker contexts) to support its standardization.

Reason this experiment is being extended

Ongoing technical constraints

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

MSE and DedicatedWorker already exist on all six Blink platforms. This feature lets MSE work from DedicatedWorker contexts.

Is this feature fully tested by web-platform-tests?

Yes

DevTrial instructions

https://github.com/w3c/media-source/issues/175#issuecomment-721395481

Flag name

MediaSourceInWorkers

Tracking bug

https://crbug.com/878133

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5177263249162240

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/u/1/a/chromium.org/g/blink-dev/c/CNRywDqgKjY/m/F0nnA4tTAwAJ

This intent message was generated by Chrome Platform Status.

[Matthew Wolenetz]

Changed title (fixed typo) in case filters missed the original transmission.

[Mike West]

It sounds like there may be some potential for creating a high-resolution timer using a cross-thread stream. Is this something y'all have looked into? Is memory shared between the Dedicated Worker and its Window? https://w3c.github.io/media-source/ doesn't have a security considerations section, so it's not clear if this has been considered.

-mike

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAADho6OYdKY_VBN94cDZGKwDrR_D7JSG4EZq%3DvMas%2B5Q%2BXR-xA%40mail.gmail.com.

[Mike West]

Friendly ping on the question below.

-mike

[Matthew Wolenetz]

Thank you for your question. I'm uncertain what is meant by a "cross-thread stream" in the context of MSE-in-Workers, though I believe I understand your overall concern.

The specification makes most of the MSE/HTMLMediaEelement Worker/Window state communication operations rely upon a "cross-context communication mechanism" which allows for implementations to be as slow as an app doing postMessages to communicate state changes across contexts. However, the spec also allows for implementations to perform such communication in more optimized fashion. The current experimental Chromium implementation does some, though not all, operations in this more performant fashion. I'll attempt to list the ops which are/are not optimized here:

• Most operations in the MSE impl that touch the media pipeline/demuxer are asynchronous:
• SourceBuffer appendBuffer(..)/appendEncodedChunk(..)/remove(start,end): apart from initial sanity checks internal to the MSE API, these queue tasks to parse and buffer input (or remove a range of buffered media) with the demuxer asynchronously, with completion or error causing a change to the 'updating' attribute in Worker, newly queued events in Worker, and if the media pipeline deems the newly buffered information merits media element notification (readyState transition, playback state progress/stall/etc), that is done too (though concurrently and not synchronous with the dispatch or 'updating' transition in Worker). Also, at most one of these operations on a specific SourceBuffer can be in flight at a time. I don't think it's possible to construct a high-res timer from these operations. 
• Duration attribute updates in Worker are trampolined via media thread asynchronously to Window.
• HTMLMediaElement.buffered and seekable queries in Window are dependent upon the result of asynchronous operations in Worker (appends/removes, described above), with the exception of:
• The buffered and seekable synchronous calculations are done under a micro-lock on the shared Worker state, and though the buffered ranges are updated asynchronously (per above) in a SourceBuffer, there are other operations on MSE API that are synchronous and could modify the result of the seekable query:
• [*] set/clearLiveSeekableRange is used by .seekable if duration is +Infinity. If duration is not used and nothing is buffered, then seekable contains the duration value as the single range's end time. This is the only information of which I am aware that the implementation might allow a Worker to set timing information using the seekable range and duration.

Considering a successful MSE-in-Worker attachment/connection is allowed only for same-origin workers, if I understand correctly, then would [*] be a cause for concern?

Possible mitigations for [*]:

• Make the seekable and buffered queries done by the media element depend on asynchronously updated live-seekable-range and duration information.

Regarding P&S in the MSE spec, there's an MSE spec bug previously already, separate from MSE-in-Workers, to add the P&S sections per Media WG: https://github.com/w3c/media-source/issues/261. I could update it and the draft MSE-in-Worker spec PR to ensure that the seekable and buffered information is not over-optimized, if either cross-origin MSE attachments (worker scripts) are somehow allowed, or if there is still the high-res timer concern even if same-origin.

Thanks again for your question - please help me further understand the concern.

-Matt

[Mike West]

On Thu, Jul 15, 2021 at 8:54 PM Matthew Wolenetz <wole...@chromium.org> wrote:

Thank you for your question. I'm uncertain what is meant by a "cross-thread stream" in the context of MSE-in-Workers, though I believe I understand your overall concern.

The specification makes most of the MSE/HTMLMediaEelement Worker/Window state communication operations rely upon a "cross-context communication mechanism" which allows for implementations to be as slow as an app doing postMessages to communicate state changes across contexts. However, the spec also allows for implementations to perform such communication in more optimized fashion. The current experimental Chromium implementation does some, though not all, operations in this more performant fashion. I'll attempt to list the ops which are/are not optimized here:

• Most operations in the MSE impl that touch the media pipeline/demuxer are asynchronous:
• SourceBuffer appendBuffer(..)/appendEncodedChunk(..)/remove(start,end): apart from initial sanity checks internal to the MSE API, these queue tasks to parse and buffer input (or remove a range of buffered media) with the demuxer asynchronously, with completion or error causing a change to the 'updating' attribute in Worker, newly queued events in Worker, and if the media pipeline deems the newly buffered information merits media element notification (readyState transition, playback state progress/stall/etc), that is done too (though concurrently and not synchronous with the dispatch or 'updating' transition in Worker). Also, at most one of these operations on a specific SourceBuffer can be in flight at a time. I don't think it's possible to construct a high-res timer from these operations. 
• Duration attribute updates in Worker are trampolined via media thread asynchronously to Window.
• HTMLMediaElement.buffered and seekable queries in Window are dependent upon the result of asynchronous operations in Worker (appends/removes, described above), with the exception of:
• The buffered and seekable synchronous calculations are done under a micro-lock on the shared Worker state, and though the buffered ranges are updated asynchronously (per above) in a SourceBuffer, there are other operations on MSE API that are synchronous and could modify the result of the seekable query:
• [*] set/clearLiveSeekableRange is used by .seekable if duration is +Infinity. If duration is not used and nothing is buffered, then seekable contains the duration value as the single range's end time. This is the only information of which I am aware that the implementation might allow a Worker to set timing information using the seekable range and duration.

Considering a successful MSE-in-Worker attachment/connection is allowed only for same-origin workers, if I understand correctly, then would [*] be a cause for concern?

Possible mitigations for [*]:

• Make the seekable and buffered queries done by the media element depend on asynchronously updated live-seekable-range and duration information.

If one side of the optimized communication channel can set state that's visible to the other side, it seems like it might be a cause for concern, as it sounds like it might allow developers to create a very high-resolution timer in the same way that SharedArrayBuffer does (e.g. by incrementing a value in a tight loop on one side, and reading the value from the other, see https://xsleaks.dev/docs/attacks/timing-attacks/clocks/#implicit-clocks). We've addressed that concern for SharedArrayBuffer by gating it on cross-origin isolation to ensure that attackers using timers for side-channel attacks have a harder time gaining access to sensitive data. Could [CrossOriginIsolated] be a reasonable restriction for this mechanism as well? Is this something you've discussed in the working group? Or am I misunderstanding the capability you refer to above?

[Matthew Wolenetz]

Thank you for the background information. I have not discussed CrossOriginIsolated], though will look and see if it might be a viable option, as well as discuss with the WG.

Matt

[Matthew Wolenetz]

[CrossOriginIsolated]'s isolation seems to be too strict of a requirement, especially given we can instead require (variable) delays in communication to be induced by queuing tasks to transfer the information. I will investigate adding task queueing for the worker-to-window communication of live seekable range and duration, as used by the window thread for seekable, buffered and duration attribute values. Similarly, "recent error" status of the media element may need task queueing at least in the specification, though I think Chromium's implementation may already be somewhat protected from using it to create a high-res timer.

[Alex Russell]

I know lots of folks are out on holiday, but wanted to check to see if the postMessage timing experiment worked out.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Matthew Wolenetz]

I'm lining up the code to do this currently, since the various hooks into the shared state are complex. Hope to have something for review by EOW, but then I'm out next week on holiday.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Matthew Wolenetz]

The mitigation code (using kPostedMessage task runner type for cross-thread MSE API state that an app might otherwise use for a high-res timer if optimized too much) is currently in code review (https://chromium-review.googlesource.com/c/chromium/src/+/3095089).

[Mike West]

LGTM to experiment (assuming a normal length of ~M95 through ~M99 (and my sincere apologies that it took me a while to get back to this). I think the mitigation y'all are putting into place is reasonable, and I'm happy with the CL's direction.

I would like to see this mitigation explained in the specification; the current PR suggests that "implementations MAY choose to communicate in potentially faster ways", and I'd like to see that tempered a bit with considerations that implementers should run through before implementing a faster-than-postMessage mechanism (up to and including cross-origin isolation).

Thanks!

-mike

[Matthew Wolenetz]

Thank you for the response - I fully intend to much more strictly restrict the specification on what kinds of allowances are there, probably just say that it must be like postMessage(), with a P&S section reference explicitly describing why (to mitigate potential for enabling Spectre-like attacks). Cross-origin isolation may be too restrictive for interop unless implementations use different mechanisms (optimized if isolated, unoptimized if not, for example).

[Matthew Wolenetz]

Today, I've requested extension of this experiment.

Please see https://groups.google.com/u/1/a/chromium.org/g/blink-dev/c/WETrrUYLrTM.

Thanks!

Matt