Intent-to-Ship: MediaRecorder keyframe configurability

[Markus Handell]

Contact emails:

hand...@google.com

Explainer:

• https://github.com/w3c/mediacapture-record/pull/216

• crbug.com/1371449

TAG review:

N/A, small incremental change

Summary:

The `MediaRecorder` spec was recently updated to allow users to configure video keyframe generation parameters of the encoder. Previously there was no such support, and no spec requirements related to keyframe generation existed, leading to recorded streams that sometimes contain a single keyframe at the beginning of the stream. When played back those streams are inefficient to seek in, and it’s impossible to excise segments from them without resorting to transcoding. These problems present significant hurdles to use the MR API for some applications.

Link to “Intent to Prototype”

N/A, small incremental change and going directly for shipping.

Risks

Interoperability and Compatibility

This is now in the editors draft https://w3c.github.io/mediacapture-record/#mediarecorder-api

Gecko: No signal

WebKit: No signal

Web developers: No signals

The pull request adding this feature was reviewed and approved by the WebRTC WG containing representatives from Apple, Microsoft, and Mozilla. Feedback was addressed in the spec PR. But we have not reached out for comments on signals.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Is this feature fully tested by web-platform-tests?

Testability requires muxer parsers which MediaRecorder WPTs are currently lacking. https://github.com/web-platform-tests/wpt/issues/39642 has been filed to ensure WPTs are added when this is fixed.

Tracking bug

crbug.com/1371449

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5117302963437568

[Henrik Boström]

I was present when the WG discussed this pull request and can confirm that representatives of Apple, Microsoft, Firefox and Google were present when the PR was merged.

This API solves problems relating to variable frame rate tracks and being able to ensure key frames in the streams. Allowing implementing this seems like a good thing as it plugs an existing hole, so to speak.

[Philip Jägenstedt]

Thanks for filing
https://github.com/web-platform-tests/wpt/issues/39642, I've asked
there for more details on what it would take to test this.

The spec change added two dictionary members to
https://w3c.github.io/mediacapture-record/#mediarecorderoptions-section
which means that it would at least be possible to add a test for
videoKeyFrameIntervalDuration and videoKeyFrameIntervalCount being
present. Can you write such a test?

Also, I think vendor signals here aren't needed given the reviews on
https://github.com/w3c/mediacapture-record/pull/216, but can you file
bugs for Gecko and WebKit?

> --
> You received this message because you are subscribed to the Google Groups "blink-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/e104767c-a31e-4553-ad18-1a101e33d0een%40chromium.org.

[Yoav Weiss]

On Fri, Apr 21, 2023 at 6:48 PM 'Markus Handell' via blink-dev <blin...@chromium.org> wrote:

Contact emails:

hand...@google.com

Explainer:

• https://github.com/w3c/mediacapture-record/pull/216

• crbug.com/1371449

I think that a real explainer that outlines the API shape and how developers would be using it would really help here with reviewing this.

Developer facing documentation could be a good replacement for that, if y'all have any.

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAJjiFfHABg0DrRviiT5HXAidwynN5ti%2BTyE7Go4ZWUViQg-shg%40mail.gmail.com.

[Philip Jägenstedt]

On Wed, Apr 26, 2023 at 12:42 PM Yoav Weiss <yoav...@chromium.org> wrote:

On Fri, Apr 21, 2023 at 6:48 PM 'Markus Handell' via blink-dev <blin...@chromium.org> wrote:

Contact emails:

hand...@google.com

Explainer:

• https://github.com/w3c/mediacapture-record/pull/216

• crbug.com/1371449

I think that a real explainer that outlines the API shape and how developers would be using it would really help here with reviewing this.

I would suggest searching for "keyframeinterval" in https://w3c.github.io/mediacapture-record/, I found that to be enough to understand the API surface and what it does.

[Yoav Weiss]

https://w3c.github.io/mediacapture-record/#mediarecorderoptions-section indeed helps clarify what this is.  

[Markus Handell]

Thanks for filing
https://github.com/web-platform-tests/wpt/issues/39642, I've asked
there for more details on what it would take to test this.

I added some more detail there.

 

The spec change added two dictionary members to
https://w3c.github.io/mediacapture-record/#mediarecorderoptions-section
which means that it would at least be possible to add a test for
videoKeyFrameIntervalDuration and videoKeyFrameIntervalCount being
present. Can you write such a test?

Yes!

 

Also, I think vendor signals here aren't needed given the reviews on
https://github.com/w3c/mediacapture-record/pull/216, but can you file
bugs for Gecko and WebKit?

Bugs filed:

https://bugs.webkit.org/show_bug.cgi?id=256031

https://bugzilla.mozilla.org/show_bug.cgi?id=1830236

[Philip Jägenstedt]

On Thu, Apr 27, 2023 at 9:24 AM Markus Handell <hand...@google.com> wrote:

Thanks for filing
https://github.com/web-platform-tests/wpt/issues/39642, I've asked
there for more details on what it would take to test this.

I added some more detail there.

 

The spec change added two dictionary members to
https://w3c.github.io/mediacapture-record/#mediarecorderoptions-section
which means that it would at least be possible to add a test for
videoKeyFrameIntervalDuration and videoKeyFrameIntervalCount being
present. Can you write such a test?

Yes!

Thanks! It won't test the behavior, but will at least let the tests show if there's some support or not, and catch typos in IDL, which happens.

 

Also, I think vendor signals here aren't needed given the reviews on
https://github.com/w3c/mediacapture-record/pull/216, but can you file
bugs for Gecko and WebKit?

Bugs filed:

https://bugs.webkit.org/show_bug.cgi?id=256031

https://bugzilla.mozilla.org/show_bug.cgi?id=1830236

Thank you!

Skimming https://w3c.github.io/mediacapture-record/#mediarecorderoptions-section I have just one more question about interop here. The spec says "If both videoKeyFrameIntervalDuration and videoKeyFrameIntervalCount are not null, the UA may ignore either." It seems to me an implementation will definitely pick one of them to ignore and it will be deterministic. Is there any reason to leave this up to the UA instead of writing in the spec which takes precedent?

Best regards,

Philip

[Markus Handell]

Yep this topic was discussed on https://github.com/w3c/mediacapture-record/pull/216#discussion_r1159521616 and earlier. In the end I backed out of my proposed semantics due to scepticism so the PR leaves it unspecified.

During the interim that hbos@ speaks to where the PR was merged, it was decided that a separate spec issue be filed on that topic, https://github.com/w3c/mediacapture-record/issues/219.

[Philip Jägenstedt]

Great, with https://github.com/w3c/mediacapture-record/issues/219 filed and the discussion you linked to, I don't think this needs to block.

LGTM1 to ship given a surface-level test in WPT as discussed.

[Chris Harrelson]

LGTM2

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYf5CAXSME%2B4nmBT%2B2upB-2-hGh3MMJUn8j1M5%3DV4EYLxQ%40mail.gmail.com.

[Yoav Weiss]

LGTM3

[Markus Handell]

Thanks all!