Intent to ship: block audible autoplay media intervention

[Chris Pearce]

Intent to ship: block audible autoplay media intervention

SUMMARY:

We intend to change the behaviour of HTMLMediaElement to block autoplay of audible audio and video in Firefox on desktop and mobile.

We are not going to block WebAudio at the same time. While we do plan to block audible autoplay of WebAudio content in the near future, we have not finalized our WebAudio blocking logic or intended ship date for blocking WebAudio.


TIMELINE:

We intend to run shield studies on the user impact of enabling HTMLMediaElement autoplay blocking. If those go well we hope to ship in Firefox 63 (2018-10-23) or Firefox 64 (2018-12-11). Upon conclusion of our experiments, I’ll follow up here with a confirmed ship date for this feature.

We hope to block autoplay in WebAudio in a release soon after, hopefully Firefox 64 or 65.


DETAILS:

We intend to block autoplay of HTMLMediaElement in tabs which haven't had user interaction. Web authors should assume that they require a user gesture (mouse click on a "play" button for example) in order to play audible media.

HTMLMediaElements with a "muted" attribute or "volume=0" are still allowed to play.

As with other browsers implementing this feature, we express playback being blocked by rejecting the promise returned by HTMLMediaElement.play(). Web authors should always check whether the promise returned by HTMLMediaElement.play() is rejected, and handle that case accordingly.

We also plan to allow users to create their own whitelist of sites which they trust to autoplay.

We are planning to experiment via shield studies with prompting users to approve/block playback on sites that try to autoplay before receiving user interaction.


ADVICE FOR WEB AUTHORS:

In general, the advice that applies to other browsers [1][2] with respect to autoplaying media will apply to Firefox as well; you cannot assume that you can just call HTMLMediaElement.play() for audible media and expect it to always play. You should always check whether the play promise is rejected, and handle that case accordingly.

For example:

var promise = document.querySelector('video').play();

if (promise !== undefined) {
promise.catch(error => {
// Auto-play was prevented
// Show a UI element to let the user manually start playback
}).then(() => {
// Auto-play started
});
}

(This example comes from WebKit’s announcement on blocking autoplay [2])

To test block autoplay in Firefox 63 (currently in Firefox Nightly channel), download the latest Nightly and open about:config in the URL bar and set the preferences:

media.autoplay.enabled=false
media.autoplay.enabled.user-gestures-needed=true
media.autoplay.ask-permission=true


Tracking bug: (block-autoplay) https://bugzilla.mozilla.org/show_bug.cgi?id=1376321


If you find bugs, please file them via this link and CC or need-info me (cpearce), and mark them blocking bug 1376321:
https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Audio%2FVideo%3A%20Playback


I will follow up with a confirmed ship date for block audible autoplay in Firefox once we have one.



[1] https://developers.google.com/web/updates/2017/09/autoplay-policy-changes
[2] https://webkit.org/blog/7734/auto-play-policy-changes-for-macos/

[Mounir Lamouri]

Hi Chris,

Very excited to see Firefox going forward with autoplay blocking. A couple of comments inline.

On Tue, 3 Jul 2018, at 19:38, Chris Pearce wrote:
> DETAILS:
>
> We intend to block autoplay of HTMLMediaElement in tabs which haven't
> had user interaction. Web authors should assume that they require a user
> gesture (mouse click on a "play" button for example) in order to play
> audible media.

I assume you will not allow autoplay when navigating into the same website? What about iframes? Will they be allowed to autoplay if the main frame is allowed to? Will the feature policy be used instead?

> HTMLMediaElements with a "muted" attribute or "volume=0" are still
> allowed to play.

Chrome and Safari allows autoplay only when the playback is muted. The spec allows side effect for setting `muted = false;`. What will happen for a website that changes the volume from 0 to something else?

Also, will autoplay be allowed when the video has no audio track?

-- Mounir

[Chris Pearce]

Hi Mounir,

Replies inline below...

On Thu, Jul 5, 2018 at 2:56 AM, Mounir Lamouri <mou...@lamouri.fr> wrote:

> Hi Chris,
>
> Very excited to see Firefox going forward with autoplay blocking. A couple
> of comments inline.
>
> On Tue, 3 Jul 2018, at 19:38, Chris Pearce wrote:
> > DETAILS:
> >
> > We intend to block autoplay of HTMLMediaElement in tabs which haven't
> > had user interaction. Web authors should assume that they require a user
> > gesture (mouse click on a "play" button for example) in order to play
> > audible media.
>
> I assume you will not allow autoplay when navigating into the same
> website? What about iframes? Will they be allowed to autoplay if the main
> frame is allowed to? Will the feature policy be used instead?
>
>

A gesture in any document in the document hierarchy gesture activates the
top level document, and a document uses the top level document's gesture
activation status to determine whether it can play (our implementation
actually differs a bit, but this is the effective behaviour). Gesture
activation status isn't preserved across navigation. So if the top level
document navigates, the incoming document hierarchy won't be able to
autoplay. If a non-top-level document navigates, the new document in the
iframe will still use the top-level document's gesture activation status,
and so the iframe's new content document will be able to autoplay.

> > HTMLMediaElements with a "muted" attribute or "volume=0" are still
> > allowed to play.
>
> Chrome and Safari allows autoplay only when the playback is muted. The
> spec allows side effect for setting `muted = false;`. What will happen for
> a website that changes the volume from 0 to something else?
>
>

If an HTMLMediaElement starts playing with muted/volume=0 and then script
sets it to unmuted/volume>0, then we'll pause the media.

> Also, will autoplay be allowed when the video has no audio track?
>
>
>

Our autoplay logic ignores whether the media resource has an audio track;
it makes our implementation simpler, and if an author knows a media should
be inaudible, they can set the muted attribute. So a video element playing
a media resource with no audio track and without the muted attribute would
still be blocked.


Cheers,
Chris.

[Mounir Lamouri]

On Wed, 4 Jul 2018, at 18:22, Chris Pearce wrote:
> Hi Mounir,
>
> Replies inline below...
>
>
> On Thu, Jul 5, 2018 at 2:56 AM, Mounir Lamouri <mou...@lamouri.fr> wrote:
>

> > Hi Chris,
> >
> > Very excited to see Firefox going forward with autoplay blocking. A couple
> > of comments inline.
> >
> > On Tue, 3 Jul 2018, at 19:38, Chris Pearce wrote:
> > > DETAILS:
> > >
> > > We intend to block autoplay of HTMLMediaElement in tabs which haven't
> > > had user interaction. Web authors should assume that they require a user
> > > gesture (mouse click on a "play" button for example) in order to play
> > > audible media.
> >
> > I assume you will not allow autoplay when navigating into the same
> > website? What about iframes? Will they be allowed to autoplay if the main
> > frame is allowed to? Will the feature policy be used instead?
> >
> >

> A gesture in any document in the document hierarchy gesture activates the
> top level document, and a document uses the top level document's gesture
> activation status to determine whether it can play (our implementation
> actually differs a bit, but this is the effective behaviour). Gesture
> activation status isn't preserved across navigation. So if the top level
> document navigates, the incoming document hierarchy won't be able to
> autoplay. If a non-top-level document navigates, the new document in the
> iframe will still use the top-level document's gesture activation status,
> and so the iframe's new content document will be able to autoplay.

With autoplay coming from third party content, do you consider using Feature Policy to block cross origin iframes by default?

> > > HTMLMediaElements with a "muted" attribute or "volume=0" are still
> > > allowed to play.
> >
> > Chrome and Safari allows autoplay only when the playback is muted. The
> > spec allows side effect for setting `muted = false;`. What will happen for
> > a website that changes the volume from 0 to something else?
> >
> >
>

> If an HTMLMediaElement starts playing with muted/volume=0 and then script
> sets it to unmuted/volume>0, then we'll pause the media.

The volume=0 behaviour isn't compatible with Blink and WebKit. Would it be worth only use muted for consistency between browsers?

> > Also, will autoplay be allowed when the video has no audio track?
> >
> >
> >

> Our autoplay logic ignores whether the media resource has an audio track;
> it makes our implementation simpler, and if an author knows a media should
> be inaudible, they can set the muted attribute. So a video element playing
> a media resource with no audio track and without the muted attribute would
> still be blocked.

FWIW, WebKit uses the audio track availability and Blink intends to do this at some point.

-- Mounir

[Jean-Yves Avenard]

Hi

On 05/07/2018 17:10, Mounir Lamouri wrote:
> FWIW, WebKit uses the audio track availability and Blink intends to do
> this at some point.
>

There are fundamental technical issues to resolve before this can be
achieved. And at this stage, I'm not convinced it could ever be
achieved, nor if that's worth it.

It would likely require spec changes
The media element play() has synchronous steps required, while the
entire playback demuxing/decoding steps are fully asynchronous. We can't
reliably determine if a particular track is available in a synchronous
fashion before the play() steps complete. If MSE is in use that becomes
even more tricky.


JY

[Chris Pearce]

I won't rule out doing something in the spirit providing a way for sites to control whether subframes are allowed to autoplay in future, but we won't be doing it as part of our MVP.

> > > > HTMLMediaElements with a "muted" attribute or "volume=0" are still
> > > > allowed to play.
> > >
> > > Chrome and Safari allows autoplay only when the playback is muted. The
> > > spec allows side effect for setting `muted = false;`. What will happen for
> > > a website that changes the volume from 0 to something else?
> > >
> > >
> >
> > If an HTMLMediaElement starts playing with muted/volume=0 and then script
> > sets it to unmuted/volume>0, then we'll pause the media.
>
> The volume=0 behaviour isn't compatible with Blink and WebKit. Would it be worth only use muted for consistency between browsers?

This code gives me the impression that WebKit does in fact block volume > 0 media:
https://github.com/WebKit/webkit/blob/2d78ab563d64545ead630115bcd55047d0400fac/Source/WebCore/html/MediaElementSession.cpp#L303

So maybe Blink should consider changing to be consistent with other browsers? ;)

> > > Also, will autoplay be allowed when the video has no audio track?
> > >
> > >
> > >
> > Our autoplay logic ignores whether the media resource has an audio track;
> > it makes our implementation simpler, and if an author knows a media should
> > be inaudible, they can set the muted attribute. So a video element playing
> > a media resource with no audio track and without the muted attribute would
> > still be blocked.
>
> FWIW, WebKit uses the audio track availability and Blink intends to do this at some point.

I actually implemented this, landed it, and then ended up reverting it.

The main problem here is if HTMLMediaElement.play() is called before the load algorithm has reached readyState>=HAVE_METADATA, you won't know whether the media you're loading has audio tracks.

If you hit this case, you have two options; decide whether to block immediately based on incomplete information, or return the promise and wait until you reach loadedmetadata before deciding whether to play.

If you decide based on incomplete information you become racy; the decision you make on incomplete information may contradict the decision you'd make based on more complete information when the load is further along, and network loads are inherently racy. That seems bad.

If you return the promise and yield control to JS you run into compat issues. As Jean-Yves mentioned, some sites expect that HTMLMediaElement.play() synchronously sets the paused attribute to false and enqueues tasks to fire "play", "playing" etc. Safari appears to have a quirks mode where they pretend to play and then dispatch "playing" and "pause" if they reach readyState HAVE_METADATA and they discover an autoplaying media has an audio track (even if they didn't play, they'll dispatch playing).

Experimenting with such hacks in Gecko led to compat issues. Even without such hacks I still ran into many subtle problems caused by timing changes due to the play/playing events being dispatched later on in the load, and by the paused attribute changing at a later time than script expects. That was enough to convince me to give up and just rely on the muted and volume attributes to express inaudibility.

[Mounir Lamouri]

To clarify, your MVP will allow subframes to autoplay if the main frame can, right?

> > > > > HTMLMediaElements with a "muted" attribute or "volume=0" are still
> > > > > allowed to play.
> > > >
> > > > Chrome and Safari allows autoplay only when the playback is muted. The
> > > > spec allows side effect for setting `muted = false;`. What will happen for
> > > > a website that changes the volume from 0 to something else?
> > > >
> > > >
> > >
> > > If an HTMLMediaElement starts playing with muted/volume=0 and then script
> > > sets it to unmuted/volume>0, then we'll pause the media.
> >
> > The volume=0 behaviour isn't compatible with Blink and WebKit. Would it be worth only use muted for consistency between browsers?
>
> This code gives me the impression that WebKit does in fact block volume
> > 0 media:
> https://github.com/WebKit/webkit/blob/2d78ab563d64545ead630115bcd55047d0400fac/Source/WebCore/html/MediaElementSession.cpp#L303
>
> So maybe Blink should consider changing to be consistent with other browsers? ;)

That's interesting. We spec'd something in HTML and Safari did not mention this. It's not in their documentation nor ours. What's the benefit of allowing volume=0 to autoplay?

Thanks for the feedback. The reason why we did not implement this yet is for similar reasons: getting the information may be async and our code make this decision fairly early today. This said, I think this would be beneficial for web developers. Is this something you think Gecko may implement at some point?

-- Mounir

[Chris Pearce]

That's correct. If any document in the hierarchy is gesture activated, the entire hierarchy is activated.

> > > > > > HTMLMediaElements with a "muted" attribute or "volume=0" are still
> > > > > > allowed to play.
> > > > >
> > > > > Chrome and Safari allows autoplay only when the playback is muted. The
> > > > > spec allows side effect for setting `muted = false;`. What will happen for
> > > > > a website that changes the volume from 0 to something else?
> > > > >
> > > > >
> > > >
> > > > If an HTMLMediaElement starts playing with muted/volume=0 and then script
> > > > sets it to unmuted/volume>0, then we'll pause the media.
> > >
> > > The volume=0 behaviour isn't compatible with Blink and WebKit. Would it be worth only use muted for consistency between browsers?
> >
> > This code gives me the impression that WebKit does in fact block volume
> > > 0 media:
> > https://github.com/WebKit/webkit/blob/2d78ab563d64545ead630115bcd55047d0400fac/Source/WebCore/html/MediaElementSession.cpp#L303
> >
> > So maybe Blink should consider changing to be consistent with other browsers? ;)
>
> That's interesting. We spec'd something in HTML and Safari did not mention this. It's not in their documentation nor ours. What's the benefit of allowing volume=0 to autoplay?

Setting volume to 0 is another easy way to make videos inaudible, and we want inaudible video to be able to autoplay. So I'd flip that question around and ask why you think volume=0 video should not autoplay but a video without an audio track should? Both are inaudible.

Do you have data on what proportion of media elements which autoplay don't have an audio track?

It seems to me to be much easier for web developers to be certain that a media resource is inaudible by using the muted attribute than by relying upon what tracks are contained in the resource.

[Mounir Lamouri]

On Fri, 6 Jul 2018, at 15:35, Chris Pearce wrote:
> On Saturday, July 7, 2018 at 5:04:28 AM UTC+12, Mounir Lamouri wrote:
> > On Fri, 6 Jul 2018, at 00:49, Chris Pearce wrote:
> > > On Friday, July 6, 2018 at 3:10:58 AM UTC+12, Mounir Lamouri wrote:
> > > > On Wed, 4 Jul 2018, at 18:22, Chris Pearce wrote:
> > > > > > > HTMLMediaElements with a "muted" attribute or "volume=0" are still
> > > > > > > allowed to play.
> > > > > >
> > > > > > Chrome and Safari allows autoplay only when the playback is muted. The
> > > > > > spec allows side effect for setting `muted = false;`. What will happen for
> > > > > > a website that changes the volume from 0 to something else?
> > > > > >
> > > > > >
> > > > >
> > > > > If an HTMLMediaElement starts playing with muted/volume=0 and then script
> > > > > sets it to unmuted/volume>0, then we'll pause the media.
> > > >
> > > > The volume=0 behaviour isn't compatible with Blink and WebKit. Would it be worth only use muted for consistency between browsers?
> > >
> > > This code gives me the impression that WebKit does in fact block volume
> > > > 0 media:
> > > https://github.com/WebKit/webkit/blob/2d78ab563d64545ead630115bcd55047d0400fac/Source/WebCore/html/MediaElementSession.cpp#L303
> > >
> > > So maybe Blink should consider changing to be consistent with other browsers? ;)
> >
> > That's interesting. We spec'd something in HTML and Safari did not mention this. It's not in their documentation nor ours. What's the benefit of allowing volume=0 to autoplay?
>
>
> Setting volume to 0 is another easy way to make videos inaudible, and we
> want inaudible video to be able to autoplay. So I'd flip that question
> around and ask why you think volume=0 video should not autoplay but a
> video without an audio track should? Both are inaudible.

`muted` and the presence of an audio track clear distinguishable signals as they are boolean and declarative. A video that has volume 0 could be for various reasons. Furthermore, requiring `muted` or `volume` seems slightly odd as they both apply to the HTML content while the audio track metadata is at a different level so can offer benefits if for example, modifying the content is possible but not the page.

FWIW, we could add this in Blink if it can help with x-browser compat but I've never heard of a website asking for this nor did I see articles suggesting using volume=0 even from Apple. Given that it would require the website to set `video.volume = 0;` in script, it is less powerful than `muted` that can be used as a content attribute too.

I agree that setting `muted` should be enough and that's why Blink does this but we had developers complain to us that muted autoplay didn't work and when digging into it, they didn't set muted and "it worked on Safari". I would rather have compatibility across browsers if we can even if that mean implementing things that I wouldn't otherwise. If Gecko has a strong position, we should consider asking Safari to drop this behaviour instead of having web developers pay the cost.

Chrome should have metrics about audio track and muted autoplay. I will look into them.

In general, what I'm worried about is that Firefox's implementation doesn't match Safari's nor Chrome's but something in between which isn't going to help developers. I understand that general autoplay policies are still young and we may want to wait a bit before talking about merging policies but muted autoplay is straightforward enough that we should aim for compatibility. For what it's worth, Chrome's implementation is made so that if it works on Chrome, it will work on Safari (with `playsinline` added on iOS).

-- Mounir

[Chris Pearce]

Compatibility with other implementations is an important consideration for us too. We don't want pages that work in either Safari or Chrome to not work in Firefox.

So if you can convince Safari to change to not allow volume=0 to autoplay, I commit that we will remove that provision too.

I'd prefer for authors to rely on the muted attribute rather than relying on whether a media doesn't contain an audio track, as it's quite opaque whether a media has an audio track. Just setting the muted attribute is simpler and more reliable.

That said, I'm not strongly opposed to allowing media without an audio track to autoplay, as clearly they're not audible. I would however like to get together and agree with other browser vendors how we'd resolve the race issue I mentioned above before shipping that provision.

Chris Pearce.

[Chris Pearce]

Block autoplay is now enabled by default in Nightly (only).

If you find bugs, please file them in "Core > Audio/Video > Playback" or via this link https://mzl.la/2JHYjlF and CC or need-info me (cpearce), and mark them blocking bug 1376321.

If you have any feedback on whether you find the feature good, bad, or ugly, please email/IRC me, or let the team know in the block_autoplay Mozilla slack channel.