Intent to prototype: MediaStreamTrackProcessor and VideoTrackGenerator with transferable MediaStreamTrack and MediaStreamTrackHandle

[Andreas Pehrson]

Summary:
MediaStreamTrackProcessor is the consuming end of a (video) MediaStreamTrack whereas VideoTrackGenerator is the producing end. Together they let sites process video in a worker by for instance drawing frames to an OffscreenCanvas, applying a transform, and reading them back out again.

Similar functionality is already available through the HTMLCanvasElement.captureStream main thread API but this approach has drawbacks. One is that requestAnimationFrame is throttled in background tabs, forcing sites to use various hacks to keep processing running when in the background.

MediaStreamTrackProcessor and VideoTrackGenerator solves this problem by natively providing an API for direct VideoFrame access, with timestamp information. The new interfaces will be available only on DedicatedWorker and only for video MediaStreamTracks.

MediaStreamTrackHandle is a smaller unit than a MediaStreamTrack that one can transfer to a worker for use with MediaStreamTrackProcessor while control of the track remains on the main thread.

Bug:
Transferable tracks: bug 1991619
MediaStreamTrackHandle: bug 2074270
MediaStreamTrackProcessor: bug 1991617
VideoTrackGenerator: bug 1991618

Specification:
Transferable tracks: w3c/mediacapture-extensions
MediaStreamTrackProcessor/VideoTrackGenerator/MediaStreamTrackHandle: w3c/mediacapture-transform

Platform Coverage:
All platforms.

Preference:
Nothing has landed yet, but let's use:
media.mediastreamtrack.transferable.enabled
media.mediastreamtrackprocessor.enabled
media.videotrackgenerator.enabled

DevTools Bug:
MediaStreamTracks have no special handling in devtools.

Extensions Bug:
MediaStreamTracks have no special handling in extensions.

Use Counter:
TBD

Standards-Positions Discussion:
Transferable tracks: 1044
MediaStreamTrackProcessor/VideoTrackGenerator: 1078

Other Browsers:
- Blink: Shipped an older form of MediaStreamTrackProcessor (exposed on Window instead of DedicatedWorker) in Chromium 94. Per-spec VideoTrackGenerator is tracked in chromium/40058895 and the support on DedicatedWorker in chromium/509846181.
- WebKit: Shipped in Safari 18.

web-platform-tests:
Transferable tracks: in part on medicapture-stream, in part on mediacapture-extensions.
MediaStreamTrackProcessor/VideoTrackGenerator: mediacapture-insertable-streams, and we'll add more as necessary. Not filed yet.

Andreas Pehrson

WebRTC Media Engineer

Slack: @pehrsons

Mozilla