Intent to Implement & Ship: Support Opus in mp4 (ISO-BMFF) with Media Source Extensions (MSE)

[Dale Curtis]

Contact emails dalec...@chromium.org Spec http://opus-codec.org/docs/opus_in_isobmff.html

https://www.w3.org/TR/mse-byte-stream-format-isobmff/

Summary Opus is an audio codec, already supported in normal src=<url> Chrome HTML5 playback within the mp4, ogg, and webm containers as well as within the webm container using MSE. This feature adds support for the Opus codec in the mp4 container to MSE. Motivation Since the AV1 video codec has adopted the MP4 container, it's important to have a modern, royalty-free audio codec available alongside it. Interoperability risk Firefox: Shipped Edge: No public signals (but supports opus in webm) Safari: No public signals (no opus support) Web developers: Positive Compatibility risk The spec is listed as incomplete, despite Chrome already having support in src= and Firefox having support in MSE and src=. As such there's always the risk that the spec changes in some future way. Ongoing technical constraints None. Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? Yes or no. Yes, with a caveat for encrypted opus content. Android versions prior to Lollipop do not support Opus and we must use the platform decoder for encrypted content. For clear content all versions of Android will have support via Chrome's built-in decoder. OWP launch tracking bug https://bugs.chromium.org/p/chromium/issues/detail?id=649438 Link to entry on the Chrome Platform Status https://www.chromestatus.com/features/5100845653819392 Requesting approval to ship? Yes. https://chromium-review.googlesource.com/c/chromium/src/+/1178929

[Yoav Weiss]

Can such support (once added) be feature detected?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPUDrwf-A2cH-x6YSFqN8c_xdAivEakJoSXcrRFiWwD6G37UQw%40mail.gmail.com.

[Dale Curtis]

Yes, through the standard media canPlayType/isTypeSupported mechanisms. E.g., for MSE pages can check MediaSource.isTypeSupported('audio/mp4; codecs="opus"');

- dale

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Yoav Weiss]

On Fri, Aug 17, 2018 at 5:45 PM Dale Curtis <dalec...@chromium.org> wrote:

Yes, through the standard media canPlayType/isTypeSupported mechanisms. E.g., for MSE pages can check MediaSource.isTypeSupported('audio/mp4; codecs="opus"');

Sounds good.

 

- dale

On Friday, August 17, 2018, Yoav Weiss <yo...@yoav.ws> wrote:

Can such support (once added) be feature detected?

On Thu, Aug 16, 2018 at 11:30 PM Dale Curtis <dalec...@chromium.org> wrote:

Contact emails dalec...@chromium.org Spec http://opus-codec.org/docs/opus_in_isobmff.html

https://www.w3.org/TR/mse-byte-stream-format-isobmff/

Summary Opus is an audio codec, already supported in normal src=<url> Chrome HTML5 playback within the mp4, ogg, and webm containers as well as within the webm container using MSE. This feature adds support for the Opus codec in the mp4 container to MSE. Motivation Since the AV1 video codec has adopted the MP4 container, it's important to have a modern, royalty-free audio codec available alongside it. Interoperability risk Firefox: Shipped Edge: No public signals (but supports opus in webm) Safari: No public signals (no opus support) Web developers: Positive Compatibility risk The spec is listed as incomplete, despite Chrome already having support in src= and Firefox having support in MSE and src=. As such there's always the risk that the spec changes in some future way.

That's unfortunate. Are you active in the standardization process? (Is there a process? Not familiar with opus.org)

Can we make sure that future changes will be backwards compatible? If they won't be, what would be our options?

Not sure if this is a blocker, but I would like to better understand the situation. 

 

Ongoing technical constraints None. Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? Yes or no. Yes, with a caveat for encrypted opus content. Android versions prior to Lollipop do not support Opus and we must use the platform decoder for encrypted content. For clear content all versions of Android will have support via Chrome's built-in decoder.

I'm guessing that wherever it's not supported by the OS, isTypeSupported will return false, right?

OWP launch tracking bug https://bugs.chromium.org/p/chromium/issues/detail?id=649438 Link to entry on the Chrome Platform Status https://www.chromestatus.com/features/5100845653819392 Requesting approval to ship? Yes. https://chromium-review.googlesource.com/c/chromium/src/+/1178929

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Dale Curtis]

On Friday, August 17, 2018, Yoav Weiss <yo...@yoav.ws> wrote:

On Fri, Aug 17, 2018 at 5:45 PM Dale Curtis <dalec...@chromium.org> wrote:

Yes, through the standard media canPlayType/isTypeSupported mechanisms. E.g., for MSE pages can check MediaSource.isTypeSupported('audio/mp4; codecs="opus"');

Sounds good.

 

- dale

On Friday, August 17, 2018, Yoav Weiss <yo...@yoav.ws> wrote:

Can such support (once added) be feature detected?

On Thu, Aug 16, 2018 at 11:30 PM Dale Curtis <dalec...@chromium.org> wrote:

Contact emails dalec...@chromium.org Spec http://opus-codec.org/docs/opus_in_isobmff.html

https://www.w3.org/TR/mse-byte-stream-format-isobmff/

Summary Opus is an audio codec, already supported in normal src=<url> Chrome HTML5 playback within the mp4, ogg, and webm containers as well as within the webm container using MSE. This feature adds support for the Opus codec in the mp4 container to MSE. Motivation Since the AV1 video codec has adopted the MP4 container, it's important to have a modern, royalty-free audio codec available alongside it. Interoperability risk Firefox: Shipped Edge: No public signals (but supports opus in webm) Safari: No public signals (no opus support) Web developers: Positive Compatibility risk The spec is listed as incomplete, despite Chrome already having support in src= and Firefox having support in MSE and src=. As such there's always the risk that the spec changes in some future way.

That's unfortunate. Are you active in the standardization process? (Is there a process? Not familiar with opus.org)

Can we make sure that future changes will be backwards compatible? If they won't be, what would be our options?

We've reached out to the spec author to see about providing assistance or at least understanding what's incomplete.

There's really not much that's opus specific, so I don't expect complications. I.e., the implementation CL below doesn't even implement the Opus box reading yet (it will before submission), but still play all Opus content (just without preroll or delay working).

 

Not sure if this is a blocker, but I would like to better understand the situation. 

 

Ongoing technical constraints None. Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? Yes or no. Yes, with a caveat for encrypted opus content. Android versions prior to Lollipop do not support Opus and we must use the platform decoder for encrypted content. For clear content all versions of Android will have support via Chrome's built-in decoder.

I'm guessing that wherever it's not supported by the OS, isTypeSupported will return false, right?

Yes, but that signaling will be through the EME API, MediaKeysSystemAccess.getConfiguration().

 

OWP launch tracking bug https://bugs.chromium.org/p/chromium/issues/detail?id=649438 Link to entry on the Chrome Platform Status https://www.chromestatus.com/features/5100845653819392 Requesting approval to ship? Yes. https://chromium-review.googlesource.com/c/chromium/src/+/1178929

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Alex Russell]

LGTM1

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Chris Harrelson]

LGTM2, but please do try to find out more about the status of this spec and whether it can be advanced.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fe029f97-cd0b-4271-86ec-1b93b9504ab7%40chromium.org.

[Philip Jägenstedt]

LGTM3

The spec is maintained at https://github.com/xiph/opus/blob/master/doc/opus_in_isobmff.html and is mostly recently updated by Xiph veteran Ralph Giles, on CC. Ralph, this is a thread about shipping support for Opus in MP4 in one additional context in Chrome. The spec at http://opus-codec.org/docs/opus_in_isobmff.html claims to be "incomplete", do you have a sense for what it'd take to make it complete?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw-PfUt%2BYa%3DyQbDRKpeZ8NvpnyozLadpLRD_9qK7cL8o8A%40mail.gmail.com.

[Philip Jägenstedt]

Dale, do we count as an implementer of that spec, or are we using a third party MP4 demuxer? If we have implemented ourselves, here's a chance to provide spec feedback :)

On Wed, Aug 22, 2018 at 5:17 AM Ralph Giles <gi...@thaumas.net> wrote:

Great news!

The spec is incomplete in that there's been little feedback from
implementors, and afaik no one's gone to the trouble to officially
register the new boxes.

I don't anticipate any changes, but if you find issues, please let me know!

  -r

+ Matthew Gregan who as been overseeing the Firefox implementation.

On 2018-08-21 11:56 AM, Philip Jägenstedt wrote:
> LGTM3
>
> The spec is maintained at
> https://github.com/xiph/opus/blob/master/doc/opus_in_isobmff.html and is
> mostly recently updated by Xiph veteran Ralph Giles, on CC. Ralph, this
> is a thread about shipping support for Opus in MP4 in one additional
> context in Chrome. The spec at
> http://opus-codec.org/docs/opus_in_isobmff.html claims to be
> "incomplete", do you have a sense for what it'd take to make it complete?
>
> On Tue, Aug 21, 2018 at 8:04 PM Chris Harrelson <chri...@chromium.org
> <mailto:chri...@chromium.org>> wrote:
>
>     LGTM2, but please do try to find out more about the status of this
>     spec and whether it can be advanced.
>
>     On Tue, Aug 21, 2018 at 9:31 AM 'Alex Russell' via blink-dev

>     <blin...@chromium.org <mailto:blin...@chromium.org>> wrote:
>
>         LGTM1
>
>         On Friday, August 17, 2018 at 9:19:17 AM UTC-7, Dale Curtis wrote:
>
>
>             On Friday, August 17, 2018, Yoav Weiss <yo...@yoav.ws> wrote:
>
>
>                 On Fri, Aug 17, 2018 at 5:45 PM Dale Curtis
>                 <dalec...@chromium.org> wrote:
>
>                     Yes, through the standard media
>                     canPlayType/isTypeSupported mechanisms. E.g., for
>                     MSE pages can check
>                     MediaSource.isTypeSupported('audio/mp4; codecs="opus"');
>
>
>                 Sounds good.
>
>
>                     - dale
>
>
>                     On Friday, August 17, 2018, Yoav Weiss
>                     <yo...@yoav.ws> wrote:
>
>                         Can such support (once added) be feature detected?
>
>                         On Thu, Aug 16, 2018 at 11:30 PM Dale Curtis
>                         <dalec...@chromium.org> wrote:
>
>                             Contact
>                             emailsdalec...@chromium.orgSpechttp://opus-codec.org/docs/opus_in_isobmff.html
>                             https://www.w3.org/TR/mse-byte-stream-format-isobmff/

>                             SummaryOpus is an audio codec, already

>                             supported in normal src=<url> Chrome HTML5
>                             playback within the mp4, ogg, and webm
>                             containers as well as within the webm
>                             container using MSE. This feature adds
>                             support for the Opus codec in the mp4

>                             container to MSE.MotivationSince the AV1

>                             video codec has adopted the MP4 container,
>                             it's important to have a modern,
>                             royalty-free audio codec available alongside

>                             it. Interoperability riskFirefox: Shipped

>                             Edge: No public signals (but supports opus
>                             in webm) Safari: No public signals (no opus
>                             support) Web developers:

>                             PositiveCompatibility riskThe spec is listed

>                             as incomplete, despite Chrome already having
>                             support in src= and Firefox having support
>                             in MSE and src=. As such there's always the
>                             risk that the spec changes in some future way.
>
>
>                 That's unfortunate. Are you active in the
>                 standardization process? (Is there a process? Not

>                 familiar with opus.org <http://opus.org>)

>                 Can we make sure that future changes will be backwards
>                 compatible? If they won't be, what would be our options?
>
>
>             We've reached out to the spec author to see about providing
>             assistance or at least understanding what's incomplete.
>
>             There's really not much that's opus specific, so I don't
>             expect complications. I.e., the implementation CL below
>             doesn't even implement the Opus box reading yet (it will
>             before submission), but still play all Opus content (just
>             without preroll or delay working).
>
>
>                 Not sure if this is a blocker, but I would like to
>                 better understand the situation.
>

>                             Ongoing technical constraintsNone.Will this

>                             feature be supported on all six Blink
>                             platforms (Windows, Mac, Linux, Chrome OS,
>                             Android, and Android WebView)? Yes or

>                             no.Yes, with a caveat for encrypted opus

>                             content. Android versions prior to Lollipop
>                             do not support Opus and we must use the
>                             platform decoder for encrypted content. For
>                             clear content all versions of Android will
>                             have support via Chrome's built-in decoder.
>
>
>                 I'm guessing that wherever it's not supported by the OS,
>                 isTypeSupported will return false, right?
>
>
>             Yes, but that signaling will be through the EME
>             API, MediaKeysSystemAccess.getConfiguration().
>
>                             OWP launch tracking
>                             bughttps://bugs.chromium.org/p/chromium/issues/detail?id=649438Link
>                             to entry on the Chrome Platform
>                             Statushttps://www.chromestatus.com/features/5100845653819392Requesting
>                             approval to ship?Yes.
>                             https://chromium-review.googlesource.com/c/chromium/src/+/1178929
>
>                             --
>                             You received this message because you are
>                             subscribed to the Google Groups "blink-dev"
>                             group.
>                             To unsubscribe from this group and stop
>                             receiving emails from it, send an email to
>                             blink-dev+...@chromium.org.
>                             To view this discussion on the web visit
>                             https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPUDrwf-A2cH-x6YSFqN8c_xdAivEakJoSXcrRFiWwD6G37UQw%40mail.gmail.com

>                             <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPUDrwf-A2cH-x6YSFqN8c_xdAivEakJoSXcrRFiWwD6G37UQw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

>
>         --
>         You received this message because you are subscribed to the
>         Google Groups "blink-dev" group.
>         To unsubscribe from this group and stop receiving emails from
>         it, send an email to blink-dev+...@chromium.org

>         <mailto:blink-dev+...@chromium.org>.

>         To view this discussion on the web visit

>         https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fe029f97-cd0b-4271-86ec-1b93b9504ab7%40chromium.org
>         <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fe029f97-cd0b-4271-86ec-1b93b9504ab7%40chromium.org?utm_medium=email&utm_source=footer>.

>
>     --
>     You received this message because you are subscribed to the Google
>     Groups "blink-dev" group.
>     To view this discussion on the web visit
>     https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw-PfUt%2BYa%3DyQbDRKpeZ8NvpnyozLadpLRD_9qK7cL8o8A%40mail.gmail.com

>     <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw-PfUt%2BYa%3DyQbDRKpeZ8NvpnyozLadpLRD_9qK7cL8o8A%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>

[Dale Curtis]

We've implemented our own box reader, https://chromium-review.googlesource.com/c/chromium/src/+/1178929/6/media/formats/mp4/box_definitions.cc -- from the spec side I think it's mostly fine. The biggest issue is that no tooling can generate the mp4 style edit lists necessary for end trimming to be per-spec. I've reached out to Xiph/Firefox folks on another thread about this. I.e., there are no mp4 versions of these files https://people.xiph.org/~greg/opus_testvectors/ -- correctness_trimming_nobeeps.opus being the main issue.

Whether this is a tooling concern or something the spec should address (i.e., by making the end trimming value part of the dOps box), I don't know. 

- dale 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYeZc7JMEBXmC8T6H12eRZ_wypcMVtyS5X7F66k98YviCg%40mail.gmail.com.

[Ralph Giles]

Great news!

The spec is incomplete in that there's been little feedback from
implementors, and afaik no one's gone to the trouble to officially
register the new boxes.

I don't anticipate any changes, but if you find issues, please let me know!

-r

+ Matthew Gregan who as been overseeing the Firefox implementation.

On 2018-08-21 11:56 AM, Philip Jägenstedt wrote:

> emailsdalec...@chromium.orgSpechttp://opus-codec.org/docs/opus_in_isobmff.html
> https://www.w3.org/TR/mse-byte-stream-format-isobmff/
> SummaryOpus is an audio codec, already

> supported in normal src=<url> Chrome HTML5
> playback within the mp4, ogg, and webm
> containers as well as within the webm
> container using MSE. This feature adds
> support for the Opus codec in the mp4

> container to MSE.MotivationSince the AV1

> video codec has adopted the MP4 container,
> it's important to have a modern,
> royalty-free audio codec available alongside

> it. Interoperability riskFirefox: Shipped

> Edge: No public signals (but supports opus
> in webm) Safari: No public signals (no opus
> support) Web developers:

> PositiveCompatibility riskThe spec is listed

> as incomplete, despite Chrome already having
> support in src= and Firefox having support
> in MSE and src=. As such there's always the
> risk that the spec changes in some future way.
>
>
> That's unfortunate. Are you active in the
> standardization process? (Is there a process? Not

> familiar with opus.org <http://opus.org>)

> Can we make sure that future changes will be backwards
> compatible? If they won't be, what would be our options?
>
>
> We've reached out to the spec author to see about providing
> assistance or at least understanding what's incomplete.
>
> There's really not much that's opus specific, so I don't
> expect complications. I.e., the implementation CL below
> doesn't even implement the Opus box reading yet (it will
> before submission), but still play all Opus content (just
> without preroll or delay working).
>
>
> Not sure if this is a blocker, but I would like to
> better understand the situation.
>

> Ongoing technical constraintsNone.Will this

> feature be supported on all six Blink
> platforms (Windows, Mac, Linux, Chrome OS,
> Android, and Android WebView)? Yes or

> no.Yes, with a caveat for encrypted opus

> content. Android versions prior to Lollipop
> do not support Opus and we must use the
> platform decoder for encrypted content. For
> clear content all versions of Android will
> have support via Chrome's built-in decoder.
>
>
> I'm guessing that wherever it's not supported by the OS,
> isTypeSupported will return false, right?
>
>
> Yes, but that signaling will be through the EME
> API, MediaKeysSystemAccess.getConfiguration().
>
> OWP launch tracking
> bughttps://bugs.chromium.org/p/chromium/issues/detail?id=649438Link
> to entry on the Chrome Platform

> Statushttps://www.chromestatus.com/features/5100845653819392Requesting
> approval to ship?Yes.

> https://chromium-review.googlesource.com/c/chromium/src/+/1178929
>
> --
> You received this message because you are
> subscribed to the Google Groups "blink-dev"
> group.
> To unsubscribe from this group and stop
> receiving emails from it, send an email to
> blink-dev+...@chromium.org.
> To view this discussion on the web visit
> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPUDrwf-A2cH-x6YSFqN8c_xdAivEakJoSXcrRFiWwD6G37UQw%40mail.gmail.com

> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPUDrwf-A2cH-x6YSFqN8c_xdAivEakJoSXcrRFiWwD6G37UQw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the
> Google Groups "blink-dev" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to blink-dev+...@chromium.org

> <mailto:blink-dev+...@chromium.org>.

> To view this discussion on the web visit

> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fe029f97-cd0b-4271-86ec-1b93b9504ab7%40chromium.org
> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fe029f97-cd0b-4271-86ec-1b93b9504ab7%40chromium.org?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "blink-dev" group.
> To view this discussion on the web visit
> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw-PfUt%2BYa%3DyQbDRKpeZ8NvpnyozLadpLRD_9qK7cL8o8A%40mail.gmail.com

> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw-PfUt%2BYa%3DyQbDRKpeZ8NvpnyozLadpLRD_9qK7cL8o8A%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>

[Dale Curtis]

Thanks for review. Given approval and non-blocking concerns with end trimming, we landed support for this in M70:

https://chromium-review.googlesource.com/1178929

- dale