Intent to Implement and Ship: Support FLAC in ISO-BMFF with MSE

[Matthew Wolenetz]

Contact emails
wole...@chromium.org

Spec
https://github.com/xiph/flac/blob/master/doc/isoflac.txt

https://www.w3.org/TR/mse-byte-stream-format-isobmff/

Summary
FLAC is a lossless audio coding format, already supported in regular Chrome HTML5 playback. This feature adds support for FLAC in ISO-BMFF to Chrome HTML5 Media Source Extensions (MSE).

Motivation
Chrome HTML5 SRC has supported FLAC since M56 (ChromeOS had FLAC support from launch), and FLAC in ISO-BMFF since M57.  FLAC is a lossless audio codec, and there currently is no other lossless audio codec supported by Chrome MSE.

Firefox developed and implemented (in 51+) support for a FLAC-in-ISOBMFF (MP4) encapsulation spec, and BBC is experimenting with using that with MSE. Support for FLAC with MSE in Chrome is also a prerequisite for Chrome EME support for FLAC.

Since FLAC is lossless, it’s likely premium audio players may eventually require EME support, too. This feature is focused primarily on MSE, but enables more rapid ability to enable support for FLAC+MSE with EME in the future.

Interoperability risk
Though the isoflac encapsulation spec is a draft (version 0.0.4 currently), at least Firefox, FFmpeg and BBC use it, with Firefox and BBC actively supporting and experimenting with MSE FLAC in ISO-BMFF.

The formal registration (with MP4RA) of the encapsulation spec is pending xiph.org action (https://github.com/xiph/flac/issues/38), and I have also raised concern that the ‘fLaC’ fourcc mismatches the “flac” codec string used currently in implementations and experiments (https://github.com/w3c/media-source/issues/188).

Firefox: Shipped (51)
Edge: No public signals
Safari: No public signals
Web developers: Positive

Compatibility risk
None. There is no previous support for FLAC in ISOBMFF in Chrome MSE.

Ongoing technical constraints
None

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux,
Chrome OS, Android, and Android WebView)? Yes or no.
Yes

OWP launch tracking bug
http://crbug.com/666000 is the implementation bug.

Link to entry on the Chrome Platform Status
https://www.chromestatus.com/features/5713014258925568

Requesting approval to ship?
Yes, for M62. Feature “MseFlacInIsobmff” will initially be disabled-by-default, with transition to enabled-by-default on launch and ship approval.

[Chris Harrelson]

LGTM1

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAADho6OeVmeHPbYcLa%3DX%2Buu_AHQCNERpnUBLuMKokNC7GNXwDQ%40mail.gmail.com.

[Matthew Wolenetz]

+blink-api-ow...@chromium.org - I'd like to switch this on by default ASAP in M62 Canary/Dev. Please review (my understanding is that I need your 3 lgtm's to do this, note the feature can be disabled in release branches later if the overall chrome launch review indicates that).

Thank you,

Matt

[Rick Byers]

LGTM2

Pardon my ignorance, but to what extent are codec details like this covered by conformance tests?  If they are, do we know if the Chrome and Firefox implementations pass the relevant tests?  The current ship template has a section for web-platform-tests, but codec work like this is obviously different than the typical web platform API change.

On Fri, Aug 4, 2017 at 2:43 PM, Matthew Wolenetz <wole...@chromium.org> wrote:

+blink-api-owners-discuss@chromium.org - I'd like to switch this on by default ASAP in M62 Canary/Dev. Please review (my understanding is that I need your 3 lgtm's to do this, note the feature can be disabled in release branches later if the overall chrome launch review indicates that).

Web platform API changes require 3 LGTMs before being enabled by default in trunk, yes.  But I personally don't know how codecs fit into the web platform API process.  Let's discuss with a few folks internally.

You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-discuss+unsub...@chromium.org.
To post to this group, send email to blink-api-owners-discuss@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CAADho6Me0V4-67-Ynf%2BfEtywxr8%2BEBb50VTypAM4QQn-4PtEiQ%40mail.gmail.com.

[Matthew Wolenetz]

Pardon my ignorance, but to what extent are codec details like this covered by conformance tests?  If they are, do we know if the Chrome and Firefox implementations pass the relevant tests?  The current ship template has a section for web-platform-tests, but codec work like this is obviously different than the typical web platform API change.

That's a great question. I'm unaware of web-platform codec conformance testing beyond things like obviously valid or invalid media streams (and some basic configuration change MSE tests that don't do much decode verification at all, just await expected events like video resize [a]). There could probably be some improvement done in general there, though it seems out of scope to me for this feature itself. Outside of Chrome and the general set of web-platform-tests, there are probably things like FFmpeg and other hardware decoder conformance testing for various codecs, though I'm much less certain of their consistency or coverage.

That said, the FLAC decoder (FFmpeg) used in this feature is the same as that already used in SRC= playback, and this MSE feature's implementation in Chrome [b] includes basic audio decoder output hashing, comparing the decode output of a test file with a hash. Various transmuxes of that same test media (sfx.flac, sfx-flac.mp4, sfx-frag-flac.mp4) and even a couple other lossless transcodes (sfx_s32le.wav and sfx_f32le.wav) are tested on the waterfall and fail if their decode hash differs. The hash, of course, is constant across these tests [c]. This feature hooks new MSE ISO-BMFF fLaC and dFla parsing into the existing MSE parsers and the rest of the media pipeline; the latter uses the same decoders regardless of SRC= or MSE.

[a] https://github.com/w3c/web-platform-tests/tree/master/media-source (*-config-change* files there, in particular)

[b] https://chromium.googlesource.com/chromium/src/+/cc79060bcce11b0cb6fafa673a2a20dcb12bd077

[c] https://cs.chromium.org/search/?q=ksfxlosslesshash&sq=package:chromium&type=cs

To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-discuss+unsubscr...@chromium.org.

[PhistucK]

As far as I know, the Web Platform Tests project pretty much accepts any test you are willing to contribute that would verify that browsers behave the same. Other browsers would simply skip or not import it if they do not support this (or even better, implement support for it).

So while support for this codec (or any, at least in non-WebRTC specifications) is not specified, either mandatory or optional anywhere, I think the ecosystem would only benefit from having those tests (they are basically tests for <video>, <audio>, Web Audio, Media Source Extensions, later Encrypted Media Extensions and perhaps even Media Recorder...).

While you cannot (or can you?) verify the actual sound that gets out of the speakers, you can test the time it takes for it to finish playing, perhaps the frame/sample count, perhaps the audio bits itself using Web Audio and the success of parsing various container and codecs configurations and combinations that can benefit the rest of the browsers.

It would be really great if you added those tests.

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-discuss+unsub...@chromium.org.

To post to this group, send email to blink-api-owners-discuss@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CAADho6M0%3DD%2B9Ed7POhp_ZESBhucWEEDxS48TTDvEexTqopDcMQ%40mail.gmail.com.

[Rick Byers]

Thanks for the details Matt.  Definitely whatever you can do over time to advance codec conformance testing would be great (prioritized where we're seeing implementations struggle in practice since our goal is pragmatic interop, not just testing for it's own sake of course).  But I agree that is a larger question that shouldn't block this specific intent.  Codecs are definitely different from web platform APIs in multiple ways, including having their own history/process around conformance and collaboration between implementations.

On Mon, Aug 7, 2017 at 1:28 PM, PhistucK <phis...@gmail.com> wrote:

As far as I know, the Web Platform Tests project pretty much accepts any test you are willing to contribute that would verify that browsers behave the same. Other browsers would simply skip or not import it if they do not support this (or even better, implement support for it).

So while support for this codec (or any, at least in non-WebRTC specifications) is not specified, either mandatory or optional anywhere, I think the ecosystem would only benefit from having those tests (they are basically tests for <video>, <audio>, Web Audio, Media Source Extensions, later Encrypted Media Extensions and perhaps even Media Recorder...).

While you cannot (or can you?) verify the actual sound that gets out of the speakers, you can test the time it takes for it to finish playing, perhaps the frame/sample count, perhaps the audio bits itself using Web Audio and the success of parsing various container and codecs configurations and combinations that can benefit the rest of the browsers.

It would be really great if you added those tests.

☆PhistucK

On Mon, Aug 7, 2017 at 8:08 PM, Matthew Wolenetz <wole...@chromium.org> wrote:

Pardon my ignorance, but to what extent are codec details like this covered by conformance tests?  If they are, do we know if the Chrome and Firefox implementations pass the relevant tests?  The current ship template has a section for web-platform-tests, but codec work like this is obviously different than the typical web platform API change.

That's a great question. I'm unaware of web-platform codec conformance testing beyond things like obviously valid or invalid media streams (and some basic configuration change MSE tests that don't do much decode verification at all, just await expected events like video resize [a]). There could probably be some improvement done in general there, though it seems out of scope to me for this feature itself. Outside of Chrome and the general set of web-platform-tests, there are probably things like FFmpeg and other hardware decoder conformance testing for various codecs, though I'm much less certain of their consistency or coverage.

That said, the FLAC decoder (FFmpeg) used in this feature is the same as that already used in SRC= playback, and this MSE feature's implementation in Chrome [b] includes basic audio decoder output hashing, comparing the decode output of a test file with a hash. Various transmuxes of that same test media (sfx.flac, sfx-flac.mp4, sfx-frag-flac.mp4) and even a couple other lossless transcodes (sfx_s32le.wav and sfx_f32le.wav) are tested on the waterfall and fail if their decode hash differs. The hash, of course, is constant across these tests [c]. This feature hooks new MSE ISO-BMFF fLaC and dFla parsing into the existing MSE parsers and the rest of the media pipeline; the latter uses the same decoders regardless of SRC= or MSE.

[a] https://github.com/w3c/web-platform-tests/tree/master/media-source (*-config-change* files there, in particular)

[b] https://chromium.googlesource.com/chromium/src/+/cc79060bcce11b0cb6fafa673a2a20dcb12bd077

[c] https://cs.chromium.org/search/?q=ksfxlosslesshash&sq=package:chromium&type=cs

On Fri, Aug 4, 2017 at 12:58 PM, Rick Byers <rby...@chromium.org> wrote:

LGTM2

Pardon my ignorance, but to what extent are codec details like this covered by conformance tests?  If they are, do we know if the Chrome and Firefox implementations pass the relevant tests?  The current ship template has a section for web-platform-tests, but codec work like this is obviously different than the typical web platform API change.

On Fri, Aug 4, 2017 at 2:43 PM, Matthew Wolenetz <wole...@chromium.org> wrote:

+blink-api-owners-discuss@chromium.org - I'd like to switch this on by default ASAP in M62 Canary/Dev. Please review (my understanding is that I need your 3 lgtm's to do this, note the feature can be disabled in release branches later if the overall chrome launch review indicates that).

Web platform API changes require 3 LGTMs before being enabled by default in trunk, yes.  But I personally don't know how codecs fit into the web platform API process.  Let's discuss with a few folks internally.

It sounds like codec changes are really different from other web platform API changes, but that there's also a bunch of potential value in trying to use the web platform change process for them.  Let's see how it goes.  Please forgive the ignorance of most of us on blink-dev who have very little context in the world of codecs :-)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CAADho6M0%3DD%2B9Ed7POhp_ZESBhucWEEDxS48TTDvEexTqopDcMQ%40mail.gmail.com.

[mk...@chromium.org]

LGTM3. I look forward to more losslessness in my web audio experience. :)

I agree with the general assertion that it would be nice to have conformance tests shared at the WPT-level in order to guarantee that we have interoperably embedded the underlying libraries, and that we're therefore producing an equivalent web-visible API. I also agree that in this case, the underlying unit tests on the codec itself are providing the lion's share of the benefit, and that we shouldn't block this intent on the production of web-facing extensions of those tests.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-discuss+unsub...@chromium.org.

To post to this group, send email to blink-api-owners-discuss@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CAADho6Me0V4-67-Ynf%2BfEtywxr8%2BEBb50VTypAM4QQn-4PtEiQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-discuss+unsub...@chromium.org.

To post to this group, send email to blink-api-owners-discuss@chromium.org.

[Dale Curtis]

+rtoy since WebAudio does have bit-perfect decoding tests for several codecs. Currently they're using some special features of the layout test harness, but they could probably be rewritten using the MediaRecorder framework if we added a lossless encoding option (wav/flac); lossy would do for now too, but doubles the fragility relative to library changes (both encode+decode vs decode only).

Filed https://bugs.chromium.org/p/chromium/issues/detail?id=753412

- dale

To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-discuss+unsubscr...@chromium.org.

To post to this group, send email to blink-api-owners-discuss@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CAADho6Me0V4-67-Ynf%2BfEtywxr8%2BEBb50VTypAM4QQn-4PtEiQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-discuss+unsubscr...@chromium.org.

To post to this group, send email to blink-api-owners-discuss@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CAADho6M0%3DD%2B9Ed7POhp_ZESBhucWEEDxS48TTDvEexTqopDcMQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/643ecae8-32aa-4f18-bdb1-1ffc2422f91e%40chromium.org.