Intent to Ship: MediaSession API

Zhiqiang Zhang

unread,

Jan 6, 2017, 12:13:24 PM1/6/17

to blin...@chromium.org

Contact emails

zqz...@chromium.org, mlam...@chromium.org

Spec

https://wicg.github.io/mediasession/

TAG review: https://github.com/w3ctag/spec-reviews/issues/149

Summary

The MediaSession API allows websites to customize media metadata enabling control of media in page from outside the page, such as platform UI, media keys etc. In addition, the API also allows some specification of the out-of-page UI like media title, subtitle, etc.

Explainer: https://github.com/WICG/mediasession/blob/master/explainer.md

Link to “Intent to Implement” blink-dev discussion

https://groups.google.com/a/chromium.org/d/msg/blink-dev/dLWDxYgxzQ8/vXt0ntWFNBwJ

The spec has changed significantly since the Intent to Implement. The scope has been reduced and no longer includes audio focus. It also moved to WICG and the contacts have changed.

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

This Intent to Ship is Android only (excluding WebView). While there are plans to bring the API to other platforms, it will not be exposed until it is being used, to allow for feature detection.

Demo links

https://xxyzzzq.github.io/sandbox/media-session/full-test.html

https://beaufortfrancois.github.io/sandbox/media-session/playground.html

https://beaufortfrancois.github.io/sandbox/media-session/slow-connection-with-service-worker.html

Interoperability and Compatibility Risk

* Backward compatibility risk doesn’t exist since this is a new API.

* Apple was implementing an earlier version of this API (https://bugs.webkit.org/show_bug.cgi?id=145411), and is actively involved in polishing the API on GitHub issue tracker.

* Mozilla expressed interest before but no public signal recently. They gave positive feedback regarding the scope of the API in face to face conversations.

* No feedback from Microsoft.
* Web developers: positive, there was early involvement from SoundCloud.

OWP launch tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=497735

Entry on the feature dashboard

https://www.chromestatus.c intent to shipom/features/5639924124483584

--

Zhiqiang Zhang

Chen Zhixiang

unread,

Jan 7, 2017, 1:05:25 AM1/7/17

to blink-dev

As a chromium custom-embedder dev, i know the "MediaSession" concept is due to an AudioFocus bug.

The explainer says MediaSession API has 2 purposes: for metadata display, for `allowed` play controls. So i guess there is some "concept shift"?

IMO, these APIs give pages too much control over use can locally overwrite/modify. e.g. There is a video page, which says, (in its JS code), "You are not seek forward unless having the AD video pieces viewed first...", so MediaSession API just gives much convenience for them to do this.

Zhiqiang

unread,

Jan 9, 2017, 5:40:24 PM1/9/17

to blink-dev

On Saturday, January 7, 2017 at 6:05:25 AM UTC, Chen Zhixiang wrote:

As a chromium custom-embedder dev, i know the "MediaSession" concept is due to an AudioFocus bug.
The explainer says MediaSession API has 2 purposes: for metadata display, for `allowed` play controls. So i guess there is some "concept shift"?

Yes, the AudioFocus part of is moved to a potential AudioFocus API. As we mentioned in the explainer, the AudioFocus part is less useful and less mature with a number of issues.

IMO, these APIs give pages too much control over use can locally overwrite/modify. e.g. There is a video page, which says, (in its JS code), "You are not seek forward unless having the AD video pieces viewed first...", so MediaSession API just gives much convenience for them to do this.

IIUC, I think the website can do similar things without MediaSession API -- display a seek button on the page and do the same actions when the button is clicked? If I understand wrong, can you elaborate why the API gives them too much convenience?

Philip Jägenstedt

unread,

Jan 12, 2017, 10:56:08 AM1/12/17

to Zhiqiang Zhang, blin...@chromium.org

Happy to see this making progress. Things have changed quite a bit since I was involved, so I've tried to take a look with fresh eyes.

Metadata

The members of MediaMetadata are now mutable. I wondered if that really works for FrozenArray<MediaImage>. It should work per Web IDL, but results in some funny behavior. Are there tests for this? This is the first such attribute in Blink and probably on the web platform, and Boris suspects that readonly attribute + setter might be more sensible.

Mutability also means that MediaMetadata internally knows which session owns it. The case where the same metadata object is assigned to multiple sessions is handled in the spec and matches what's implemented, but is a bit unusual:

metadata = new MediaMetadata();

navigator.mediaSession.metadata = metadata;

iframe.contentWindow.navigator.mediaSession.metadata = metadata;

// now navigator.mediaSession.metadata is null

Given that there's no obvious good reason to attempt this to begin with, did you consider simply throwing an exception instead? That's easier to explain and test.

Playback state

The API no longer deals with audio focus, but playbackState seems to be a hint affecting what will actually get focus. I've requested some examples and enough detail for a second implementer to achieve interoperability.

Controls

setActionHandler avoids the problems that event listeners would have had, and feature detection is possible as unsupported actions would throw TypeError.

The "previoustrack" and "nexttrack" actions make good sense, that's logic that has to be implemented by the page itself and there can be no default action given the lack of declarative playlists or similar.

"seekback" and "seekforward" seem problematic as they don't include the number of seconds to seek. On iOS that number is included in the now playing controls, so would it be possible to implement this on iOS? It seems to me that if seeking is under the page's control, which it has to be if using Web Audio, then in order to respect platform conventions the skip duration needs to be included. WDYT?

For "play" and "pause", the spec says "It is RECOMMENDED for user agents to implement a default handler for the play and pause media session actions if none was provided for the active media session." I've filed an issue to flesh this out. As actually implemented in Chromium, can pages register for for the "pause" action and ignore it, allowing playback to continue anyway? To ignore "play" I think should be possible, but I wonder if "pause" shouldn't have the semantics of "already-paused-now-you-may-react".

Interoperability risk

Has there been any recent activity from other vendors, after the move to WICG and the changes that followed? I think the interoperability risk of this API is particularly high since it integrates with OS-level features which work quite differently on Android, iOS and other systems. Have you reached out to the relevant folks to say that you're now planning to ship this? (When not actively implementing, I wouldn't expect anyone to pay close attention.)

Mounir Lamouri

unread,

Jan 12, 2017, 11:11:30 AM1/12/17

to Philip Jägenstedt, Zhiqiang Zhang, blin...@chromium.org

On Thu, 12 Jan 2017, at 15:55, Philip Jägenstedt wrote:
> Happy to see this making progress. Things have changed quite a bit since
> I
> was involved, so I've tried to take a look with fresh eyes.
>

> *Metadata*

>
> The members of MediaMetadata are now mutable. I wondered if that really
> works for FrozenArray<MediaImage>

> <https://github.com/heycam/webidl/issues/265>. It should work per Web

> IDL,
> but results in some funny behavior. Are there tests for this? This is the
> first such attribute in Blink and probably on the web platform, and Boris
> suspects that readonly attribute + setter might be more sensible

> <https://github.com/WICG/mediasession/issues/176>.

We do have tests. I actually have a page open to upload them to
web-platform-tests but didn't got around this.

> Mutability also means that MediaMetadata internally knows which session
> owns it. The case where the same metadata object is assigned to multiple
> sessions is handled in the spec and matches what's implemented, but is a
> bit unusual:
> metadata = new MediaMetadata();
> navigator.mediaSession.metadata = metadata;
> iframe.contentWindow.navigator.mediaSession.metadata = metadata;
> // now navigator.mediaSession.metadata is null
>
> Given that there's no obvious good reason to attempt this to begin with,
> did you consider simply throwing an exception instead? That's easier to
> explain and test.

That sounds reasonoable. It's probably uncommon enough to not matter
much but it could be surprising to get the metadata back to null
otherwise. Did you file an issue?

> *Playback state*

>
> The API no longer deals with audio focus, but playbackState seems to be a
> hint affecting what will actually get focus. I've requested some examples

> <https://github.com/WICG/mediasession/issues/175> and enough detail for a

> second implementer to achieve interoperability

> <https://github.com/WICG/mediasession/issues/177>.

We will add examples. Though, this is a feature request from Apple,
there are issues and PR with context if you want. The TL;DR is that some
websites might look like not playing but there UI is in playing mode
(for example, if they are seeking). The `playbackState allows the UA to
be aware of the state and overrides its own knowledge. There is a
similar API on Android and iOS.

> *Controls*

>
> setActionHandler avoids the problems that event listeners would have had,
> and feature detection is possible as unsupported actions would throw
> TypeError.
>
> The "previoustrack" and "nexttrack" actions make good sense, that's logic
> that has to be implemented by the page itself and there can be no default
> action given the lack of declarative playlists or similar.
>
> "seekback" and "seekforward" seem problematic as they don't include the
> number of seconds to seek. On iOS that number is included in the now
> playing controls

> <http://cdn.iphonehacks.com/wp-content/uploads/2016/06/ios-10-control-center-5.png>,

> so would it be possible to implement this on iOS? It seems to me that if
> seeking is under the page's control, which it has to be if using Web
> Audio,
> then in order to respect platform conventions the skip duration needs to
> be
> included. WDYT?

We never got this feedback from Apple so I would assume this is fine.
The API assumes there will be no UA default behaviour for these actions
so defining the time wouldn't make sense, the website should handle
this.

> For "play" and "pause", the spec says "It is RECOMMENDED for user agents
> to
> implement a default handler for the play and pause media session actions
> if
> none was provided for the active media session." I've filed an issue to

> flesh this out <https://github.com/WICG/mediasession/issues/178>. As

> actually implemented in Chromium, can pages register for for the "pause"
> action and ignore it, allowing playback to continue anyway? To ignore
> "play" I think should be possible, but I wonder if "pause" shouldn't have
> the semantics of "already-paused-now-you-may-react".

The specification allows a UA to force pause. At the moment, we've
decided to not do that in Chromium mostly because web pages can already
listen for the pause event and call `play()` again. We found a page that
had a bug that was doing just that. Force pausing in the pause action
would lead to timing issues. I do not think we should do
"already-paused-now-you-may-react" because we want these websites to
actually

> *Interoperability risk*

>
> Has there been any recent activity from other vendors, after the move to
> WICG and the changes that followed? I think the interoperability risk of
> this API is particularly high since it integrates with OS-level features
> which work quite differently on Android, iOS and other systems. Have you
> reached out to the relevant folks to say that you're now planning to ship
> this? (When not actively implementing, I wouldn't expect anyone to pay
> close attention.)

We've got good collaboration with Jer Noble from Apple. His last
contributions were on the `setActionHandler` just before the holidays.
As usual with Apple, we don't have launch timeline but if there was
anything that would prevent the API from being implemented by Apple on
iOS, I would expect them to express their concerns.

-- Mounir

> On Fri, Jan 6, 2017 at 6:13 PM 'Zhiqiang Zhang' via blink-dev <
> blin...@chromium.org> wrote:
>
> > Contact emails
> >
> > zqz...@chromium.org, mlam...@chromium.org
> >
> > Spec
> >
> > https://wicg.github.io/mediasession/
> >
> > TAG review: https://github.com/w3ctag/spec-reviews/issues/149
> >
> > Summary
> >
> > The MediaSession API allows websites to customize media metadata enabling
> > control of media in page from outside the page, such as platform UI, media
> > keys etc. In addition, the API also allows some specification of the
> > out-of-page UI like media title, subtitle, etc.
> >
> >
> > Explainer: https://github.com/WICG/mediasession/blob/master/explainer.md
> >
> > Link to “Intent to Implement” blink-dev discussion
> >
> >
> > https://groups.google.com/a/chromium.org/d/msg/blink-dev/dLWDxYgxzQ8/vXt0ntWFNBwJ
> >
> > Th

> > ed and no longer includes audio focus. It also moved to WICG and t

> > <https://groups.google.com/a/chromium.org/d/msg/blink-dev/dLWDxYgxzQ8/vXt0ntWFNBwJ>
> > he contacts have ch
> > <https://groups.google.com/a/chromium.org/d/msg/blink-dev/dLWDxYgxzQ8/vXt0ntWFNBwJ>
> > an
> > <https://groups.google.com/a/chromium.org/d/msg/blink-dev/dLWDxYgxzQ8/vXt0ntWFNBwJ>
> > ge
> > <https://groups.google.com/a/chromium.org/d/msg/blink-dev/dLWDxYgxzQ8/vXt0ntWFNBwJ>
> > d.
> > <https://groups.google.com/a/chromium.org/d/msg/blink-dev/dLWDxYgxzQ8/vXt0ntWFNBwJ>

> >
> > Is this feature supported on all six Blink platforms (Windows, Mac, Linux,
> > Chrome OS, Android, and Android WebView)?
> >
> > This Intent to Ship is Android only (excluding WebView). While there are
> > plans to bring the API to other platforms, it will not be exposed until it
> > is being used, to allow for feature detection.
> >
> > Demo links
> >
> > https://xxyzzzq.github.io/sandbox/media-session/full-test.html
> >
> > https://beaufortfrancois.github.io/sandbox/media-session/playground.html
> >
> >
> > https://beaufortfrancois.github.io/sandbox/media-session/slow-connection-with-service-worker.html
> >
> > Interoperability and Compatibility Risk
> >
> > * Backward compatibility risk doesn’t exist since this is a new API.
> >
> > * Apple was implementing an earlier version of this API (
> > https://bugs.webkit.org/show_bug.cgi?id=145411), and is actively involved
> > in polishing the API on GitHub issue tracker

> > <https://github.com/WICG/mediasession/issues>.

> >
> > * Mozilla expressed interest before but no public signal recently. They
> > gave positive feedback regarding the scope of the API in face to face
> > conversations.
> >
> > * No feedback from Microsoft.
> > * Web developers: positive, there was early involvement from SoundCloud.
> >
> > OWP launch tracking bug
> >
> > https://bugs.chromium.org/p/chromium/issues/detail?id=497735
> >
> > Entry on the feature dashboard
> >
> > https://www.chromestatus.c intent to shipom/features/5639924124483584

> > <https://www.chromestatus.com/features/5639924124483584>
> >
> > --
> > Zhiqiang Zhang
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "blink-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to blink-dev+...@chromium.org.

Boris Zbarsky

unread,

Jan 12, 2017, 11:29:02 AM1/12/17

to Mounir Lamouri, Philip Jägenstedt, Zhiqiang Zhang, blin...@chromium.org

On 1/12/17 11:11 AM, Mounir Lamouri wrote:
> We do have tests.

I'm curious as to what they test (and what Blink actually implements),
given the spec issue I raised about the "artwork" attribute setter....

-Boris

Mounir Lamouri

unread,

Jan 12, 2017, 11:36:01 AM1/12/17

to Boris Zbarsky, Philip Jägenstedt, Zhiqiang Zhang, blin...@chromium.org

You can find them here:
https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/media/mediasession/

-- Mounir

Boris Zbarsky

unread,

Jan 12, 2017, 11:59:49 AM1/12/17

to Mounir Lamouri, Philip Jägenstedt, Zhiqiang Zhang, blin...@chromium.org

On 1/12/17 11:35 AM, Mounir Lamouri wrote:
> You can find them here:
> https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/media/mediasession/

OK. So the answer is:

1) These tests don't quite test what the spec says to do.

and

2) These tests don't test the really interesting edge cases.

as far as I can tell...

-Boris

Boris Zbarsky

unread,

Jan 12, 2017, 12:01:58 PM1/12/17

to Mounir Lamouri, Philip Jägenstedt, Zhiqiang Zhang, blin...@chromium.org

On 1/12/17 11:59 AM, Boris Zbarsky wrote:
> 2) These tests don't test the really interesting edge cases.

Note that this means I still can't tell what Blink implements just given
the information that it passes these tests.

-Boris

Philip Jägenstedt

unread,

Jan 13, 2017, 11:25:44 AM1/13/17

to Mounir Lamouri, Zhiqiang Zhang, blin...@chromium.org

Mounir and I chatted about this on VC, since I had misunderstood playbackState in particular. Details inline.

On Thu, Jan 12, 2017 at 5:11 PM Mounir Lamouri <mou...@lamouri.fr> wrote:

On Thu, 12 Jan 2017, at 15:55, Philip Jägenstedt wrote:
> Happy to see this making progress. Things have changed quite a bit since
> I
> was involved, so I've tried to take a look with fresh eyes.
>
> *Metadata*
>
> The members of MediaMetadata are now mutable. I wondered if that really
> works for FrozenArray<MediaImage>
> <https://github.com/heycam/webidl/issues/265>. It should work per Web
> IDL,
> but results in some funny behavior. Are there tests for this? This is the
> first such attribute in Blink and probably on the web platform, and Boris
> suspects that readonly attribute + setter might be more sensible
> <https://github.com/WICG/mediasession/issues/176>.

We do have tests. I actually have a page open to upload them to
web-platform-tests but didn't got around this.

In addition to what's in LayoutTests/media/mediasession/mediasessionmetadata.html it would be good to test some of the things that aren't covered by something like idlharness.js tests and which is wrong by default in some sense. I've tested in dev with the feature enabled. It would be good to test the following, and fix where needed, with inspiration from GitHub:

// the same value is returned every time (currently fails)

assert_equals(metadata.artwork, metadata.artwork);

// the setter makes a copy of the value

artwork = [];

metadata.artwork = artwork;

assert_not_equals(metadata.artwork, artwork);

// even assigning the same value changes it

old_artwork = metadata.artwork

metadata.artwork = metadata.artwork;

assert_not_equals(metadata.artwork, old_artwork);

I think adding [CachedAttribute=isArtworkDirty] is what you'll need. It's ugly, but currently the only way to get it right :/

FWIW, I don't think that switching to readonly+setter would necessarily be better, so let's not go there unless someone feels strongly about it.

> Mutability also means that MediaMetadata internally knows which session
> owns it. The case where the same metadata object is assigned to multiple
> sessions is handled in the spec and matches what's implemented, but is a
> bit unusual:
> metadata = new MediaMetadata();
> navigator.mediaSession.metadata = metadata;
> iframe.contentWindow.navigator.mediaSession.metadata = metadata;
> // now navigator.mediaSession.metadata is null
>
> Given that there's no obvious good reason to attempt this to begin with,
> did you consider simply throwing an exception instead? That's easier to
> explain and test.

That sounds reasonoable. It's probably uncommon enough to not matter
much but it could be surprising to get the metadata back to null
otherwise. Did you file an issue?

https://github.com/WICG/mediasession/issues/180 which you filed will cover this.

> *Playback state*
>
> The API no longer deals with audio focus, but playbackState seems to be a
> hint affecting what will actually get focus. I've requested some examples
> <https://github.com/WICG/mediasession/issues/175> and enough detail for a
> second implementer to achieve interoperability
> <https://github.com/WICG/mediasession/issues/177>.

We will add examples. Though, this is a feature request from Apple,
there are issues and PR with context if you want. The TL;DR is that some
websites might look like not playing but there UI is in playing mode
(for example, if they are seeking). The `playbackState allows the UA to
be aware of the state and overrides its own knowledge. There is a
similar API on Android and iOS.

There are links to the original discussion in #177, thanks! This comment by Jer Noble in #141 is particularly important: 'Netflix will display a "playing" state while seeking, even though the underlying video element is paused. Without [this] the UA would send a play action in this state, not pause.'

Is there something about this that would be tricky to spec in exact detail because of platform differences?

> *Controls*
>
> setActionHandler avoids the problems that event listeners would have had,
> and feature detection is possible as unsupported actions would throw
> TypeError.
>
> The "previoustrack" and "nexttrack" actions make good sense, that's logic
> that has to be implemented by the page itself and there can be no default
> action given the lack of declarative playlists or similar.
>
> "seekback" and "seekforward" seem problematic as they don't include the
> number of seconds to seek. On iOS that number is included in the now
> playing controls
> <http://cdn.iphonehacks.com/wp-content/uploads/2016/06/ios-10-control-center-5.png>,
> so would it be possible to implement this on iOS? It seems to me that if
> seeking is under the page's control, which it has to be if using Web
> Audio,
> then in order to respect platform conventions the skip duration needs to
> be
> included. WDYT?

We never got this feedback from Apple so I would assume this is fine.
The API assumes there will be no UA default behaviour for these actions
so defining the time wouldn't make sense, the website should handle
this.

OK, given that Jer has commented on GitHub recently, let's assume then that there is a way to remove the number from the UI. (If that's not true, there are ways to extend the API to support it.)

> For "play" and "pause", the spec says "It is RECOMMENDED for user agents
> to
> implement a default handler for the play and pause media session actions
> if
> none was provided for the active media session." I've filed an issue to
> flesh this out <https://github.com/WICG/mediasession/issues/178>. As
> actually implemented in Chromium, can pages register for for the "pause"
> action and ignore it, allowing playback to continue anyway? To ignore
> "play" I think should be possible, but I wonder if "pause" shouldn't have
> the semantics of "already-paused-now-you-may-react".

The specification allows a UA to force pause. At the moment, we've
decided to not do that in Chromium mostly because web pages can already
listen for the pause event and call `play()` again. We found a page that
had a bug that was doing just that. Force pausing in the pause action
would lead to timing issues. I do not think we should do
"already-paused-now-you-may-react" because we want these websites to
actually

As discussed on VC, it's actually not so easy to define what the UA is supposed to do with a misbehaving page, at best it might have to mute it. Let's wait until this is a problem.

> *Interoperability risk*
>
> Has there been any recent activity from other vendors, after the move to
> WICG and the changes that followed? I think the interoperability risk of
> this API is particularly high since it integrates with OS-level features
> which work quite differently on Android, iOS and other systems. Have you
> reached out to the relevant folks to say that you're now planning to ship
> this? (When not actively implementing, I wouldn't expect anyone to pay
> close attention.)

We've got good collaboration with Jer Noble from Apple. His last
contributions were on the `setActionHandler` just before the holidays.
As usual with Apple, we don't have launch timeline but if there was
anything that would prevent the API from being implemented by Apple on
iOS, I would expect them to express their concerns.

I was looking at recent issues, and it seems like while Jer didn't file any, he has been commenting a lot. So yes, there has been recent activity.

All in all, there are a few spec issues and bugs that matter somewhat. The bugs are actually pretty trivial so it wouldn't be terrible if they weren't fixed until after the branch cut, and spec issues can be fixed in parallel. LGTM1.

Jochen Eisinger

unread,

Jan 16, 2017, 9:24:36 AM1/16/17

to Philip Jägenstedt, Mounir Lamouri, Zhiqiang Zhang, blin...@chromium.org

lgtm2

Dimitri Glazkov

unread,

Jan 17, 2017, 11:30:44 AM1/17/17

to Mounir Lamouri, Zhiqiang Zhang, blin...@chromium.org

LGTM3

Reply all

Reply to author

Forward