Intent to Implement: MediaStreamTrack Content Hints
Peter Boström
Contact emails
Spec
https://github.com/wicg/mst-content-hint
Summary
This specification extends MediaStreamTrack to provide a media-content hint attribute. This optional hint permits MediaStreamTrack consumers such as PeerConnection or MediaRecorder to encode or process track media with methods more appropriate to the type of content that is being consumed.
Motivation
Several APIs that consume MediaStreamTracks do implicit processing on a MediaStreamTrack based on what type of content they believe it to be. Some of this processing can currently be turned off with goog-prefixed constraints, but none of this is standardized.
For audio, WebRTC for instance does packet loss concealment in a way that messes up rhythm (smears out packets) and noise suppression that removes drum snares by default because WebRTC assumes that the signal is speech.
For video, Blink treats a track as screencast/non-screencast based on how it was opened (webcam or tab/desktop capture), and prioritizes detail for screencast, meaning that it preserves resolution and detail in individual frames at the cost of framerate (when target bitrate is lower than optimal).
This means that screencasting a website preserves detail in text, but conversely means that screencasting a video game or YouTube gives a jarringly low frame rate to preserve individual frame quality.
There are several track consumers that benefit from this signal. Benefiting consumers are PeerConnection, RTCRtpSender and MediaRecorder APIs to name a few. Adding them to consumers instead of the track itself would require updating several specifications (all consumers + future track consumers) to benefit from the signal.
Consumer APIs that want to add implicit processing as part of their implementation can make use of this content hint if the processing is only suitable for one type of content. Otherwise such processing would require explicit exposure in APIs to avoid mis-treating the incoming signal.
Interoperability and Compatibility Risk
Low I believe, using this hint enables better treatment of data when available, but does not specify any specific behavior. An implementation is technically compliant even if all track consumers ignore the hint completely.
Ongoing technical constraints
None
Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?
Yes
OWP launch tracking bug
Link to entry on the feature dashboard
https://www.chromestatus.com/feature/5689466211532800
Requesting approval to ship?
Not right now.