Intent to Implement: Web Audio requires user gesture on Android

[Mounir Lamouri]

Contact emails

mlam...@chromium.org (eng), renga...@chromium.org (pm)

 

Spec

No spec.

Public discussion at https://github.com/WebAudio/web-audio-api/issues/836.

Summary of WebKit iOS behaviour at https://docs.google.com/document/d/16EzKmZJNNyvKC1UN-heRN2h32ZF6H1EUCbcrcxRH7H4/edit

Both spec editors are interested to either specify WebKit’s behaviour or at least something similar.

 

Summary

Chrome Android will require a user gesture in order to start playback using Web Audio, the same way a user gesture is required to play an audio or video element.

 

Motivation

A more consistent experience for playback without a user gesture in Chrome Android and mobile Safari. The proposal is to require a user gesture for Web Audio - just like mobile Safari and HTMLMediaElement and brings consistency to the mobile web platform.

 

Interoperability and Compatibility Risk

Firefox: Public support

Edge: No public signals

Safari: Shipped

Web developers: No signals

 

If we ship exactly what WebKit does or something compatible with it, interoperability will increase. However, websites doing UA sniffing might break on Chrome. We landed UMA to measure this.

If we ship something too different from Safari, interoperability might be affected unless Safari follows. However, given that the benefit of shipping something very different is low, chances are we will not unless the editors insist on it.

We will start by implementing something compatible with Safari’s implementation.

 

Ongoing technical constraints

None.

 

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

As far as Chrome is concerned, this will only ship on Android. However, the feature itself doesn’t have platform specific constraints.

 

OWP launch tracking bug

https://crbug.com/614115

 

Link to entry on the feature dashboard

https://www.chromestatus.com/features/6406908126691328

 

Requesting approval to ship?

No.

[Jeffrey Yasskin]

On Tue, Jun 7, 2016 at 3:35 PM, Mounir Lamouri <mou...@lamouri.fr> wrote:

Contact emails

mlam...@chromium.org (eng), renga...@chromium.org (pm)

 

Spec

No spec.

Public discussion at https://github.com/WebAudio/web-audio-api/issues/836.

Summary of WebKit iOS behaviour at https://docs.google.com/document/d/16EzKmZJNNyvKC1UN-heRN2h32ZF6H1EUCbcrcxRH7H4/edit

Both spec editors are interested to either specify WebKit’s behaviour or at least something similar.

By "specify WebKit's behavior", do you mean "require a user gesture to play audio", or just "return matching values from .state"? #836 only seems to cover the second.

I got opposition to having the spec require user gestures for requesting permission from a Firefox developer in https://github.com/w3c/permissions/issues/77. He was fine with allowing UAs to do this. Ben Wells was sympathetic to using 'may' instead of 'must' for this too. Folks may have different opinions about requesting permission vs playing audio, of course.

 

Summary

Chrome Android will require a user gesture in order to start playback using Web Audio, the same way a user gesture is required to play an audio or video element.

 

Motivation

A more consistent experience for playback without a user gesture in Chrome Android and mobile Safari. The proposal is to require a user gesture for Web Audio - just like mobile Safari and HTMLMediaElement and brings consistency to the mobile web platform.

 

Interoperability and Compatibility Risk

Firefox: Public support

Do you have a link to this support? 

[Chris Wilson]

I wanted to point out the "public discussion" link isn't a discussion of the advisability of this feature; it's a discussion of exposing how to tell whether playback on a node started or not (i.e. a way to expose whether this feature has affected normal playback or not).

I want to go on record that I think this is a horrible idea.  Requiring a user gesture to play <audio> was imo a hack to mitigate sites out there that played annoying background music.  I have yet to see usages of webaudio that users would actually want to consistently automatically disable like this, and this "click so my web audio case can work" is a very poor user experience - especially since web audio would more likely be used in scenarios where you would get some OTHER kind of "user interaction" (like MIDI messages from a MIDI controller - in my web audio synth, for example, I'd need to detect the failure to start, and put a honking big button that says "touch a sound-producing key on the screen first" if the user tried to just play from a MIDI controller).

Additionally, AudioContexts never switching to running state will cause unfortunate timing side effects (we used to have an issue with Chrome not starting the clock immediately - this would likely cause a side effect like this.)  The normal process is that AudioContexts automatically switch to running state when created; this would break that expectation.  The messiness with ScriptProcessor (captured in your doc) also adds another layer of complexity - as would AudioParams being constructible (you could use an AudioParam to create sound) (issue).

In short - I think Webkit made a poor decision when they chose to implement this, and I would hate to see that pattern get propagated to other browsers.

[Hongchan Choi]

Sadly I am agreeing with Chris here.

As a WebAudio developer, I feel this blocking mechanism on Safari is making the web music application crippled by default. From the implementation perspective, both returning a promise or a boolean value for every AudioBufferSourceNode are not clean. If this has to be implemented somehow, the registration should be under the context or a tab - but I prefer not to put this check in our web audio code.

[Mounir Lamouri]

On Wed, 8 Jun 2016, at 00:45, Chris Wilson wrote:
> I wanted to point out the "public discussion" link isn't a discussion of
> the advisability of this feature; it's a discussion of exposing how to
> tell
> whether playback on a node started or not (i.e. a way to expose whether
> this feature has affected normal playback or not).

Correct. Chrome Android intents to intervene and block Web Audio
autoplay for consistency with its autoplay behaviour. As a product it
doesn't make sense to block autoplay for <audio> but not Web Audio while
you can simply play any audio file via Web Audio. See:
https://mounirlamouri.github.io/sandbox/autoplay/webaudio.html

I want to make sure that we don't do crazy hacks and that means working
directly with the specification to make sure this is handled as best as
possible. The bare minimum would be to specify WebKit's behaviour. Note
that this is not an intent to ship and we don't know yet for sure what
will ship.

> I want to go on record that I think this is a horrible idea. Requiring a
> user gesture to play <audio> was imo a hack to mitigate sites out there
> that played annoying background music. I have yet to see usages of
> webaudio that users would actually want to consistently automatically
> disable like this, and this "click so my web audio case can work" is a
> very
> poor user experience - especially since web audio would more likely be
> used
> in scenarios where you would get some OTHER kind of "user interaction"
> (like MIDI messages from a MIDI controller - in my web audio synth, for
> example, I'd need to detect the failure to start, and put a honking big
> button that says "touch a sound-producing key on the screen first" if the
> user tried to just play from a MIDI controller).
>
> Additionally, AudioContexts never switching to running state will cause
> unfortunate timing side effects (we used to have an issue with Chrome not
> starting the clock immediately - this would likely cause a side effect
> like
> this.) The normal process is that AudioContexts automatically switch to
> running state when created; this would break that expectation. The
> messiness with ScriptProcessor (captured in your doc) also adds another
> layer of complexity - as would AudioParams being constructible (you could
> use an AudioParam to create sound) (issue

> <https://github.com/WebAudio/web-audio-api/issues/367>).

>
> In short - I think Webkit made a poor decision when they chose to
> implement
> this, and I would hate to see that pattern get propagated to other
> browsers.

I agree that WebKit's implementation is hacky in some ways but I assume
that they had to implement their behaviour without too many side effects
on the rest of the API behaviour. My hope, and this is why I started
this discussion on the spec is that we can come up with a good solution
that acknowledge this use case.

In other words, if the Web Audio specification doesn't accept that some
UA will not allow playback unless the user interacted with the page, UA
will find ways to enforce this restriction and developers will have to
write browser specific code. Embracing this is probably the best
solution for UA, web developers and users.

> On Tue, Jun 7, 2016 at 3:51 PM, Jeffrey Yasskin <jyas...@chromium.org>
> wrote:
>
> > On Tue, Jun 7, 2016 at 3:35 PM, Mounir Lamouri <mou...@lamouri.fr> wrote:
> >

> >> *Contact emails*
> >>
> >> *mlam...@chromium.org* <mlam...@chromium.org> (eng),
> >> *renga...@chromium.org* <renga...@chromium.org> (pm)
> >>
> >>
> >> *Spec*

> >>
> >> No spec.
> >>
> >> Public discussion at

> >> *https://github.com/WebAudio/web-audio-api/issues/836*
> >> <https://github.com/WebAudio/web-audio-api/issues/836>.

> >>
> >> Summary of WebKit iOS behaviour at

> >> *https://docs.google.com/document/d/16EzKmZJNNyvKC1UN-heRN2h32ZF6H1EUCbcrcxRH7H4/edit*

> >> <https://docs.google.com/document/d/16EzKmZJNNyvKC1UN-heRN2h32ZF6H1EUCbcrcxRH7H4/edit>
> >>
> >> Both spec editors are interested to either specify WebKit’s behaviour or
> >> at least something similar.
> >>
> >
> > By "specify WebKit's behavior", do you mean "require a user gesture to
> > play audio", or just "return matching values from .state"? #836 only seems
> > to cover the second.
> >
> > I got opposition to having the spec require user gestures for requesting
> > permission from a Firefox developer in
> > https://github.com/w3c/permissions/issues/77. He was fine with allowing
> > UAs to do this. Ben Wells was sympathetic to using 'may' instead of 'must'
> > for this too. Folks may have different opinions about requesting permission
> > vs playing audio, of course.

I think this is very different. The intent is not to require a user
gesture per spec but to allow such requirement. Similarly to the work
done with HTMLMediaElement. See step 1 of the play algorithm:
https://html.spec.whatwg.org/multipage/embedded-content.html#dom-media-play

> >> *Summary*

> >>
> >> Chrome Android will require a user gesture in order to start playback
> >> using Web Audio, the same way a user gesture is required to play an audio
> >> or video element.
> >>
> >>

> >> *Motivation*

> >>
> >> A more consistent experience for playback without a user gesture in
> >> Chrome Android and mobile Safari. The proposal is to require a user gesture
> >> for Web Audio - just like mobile Safari and HTMLMediaElement and brings
> >> consistency to the mobile web platform.
> >>
> >>

> >> *Interoperability and Compatibility Risk*

> >>
> >> Firefox: Public support
> >>
> >
> > Do you have a link to this support?

padenot (see github issue I linked) is the main Mozilla person working
on Web Audio. As you can see he is in favour of the spec change (CC'd
him).

-- Mounir

[win...@gmail.com]

On Tuesday, June 7, 2016 at 6:52:08 PM UTC-4, Jeffrey Yasskin wrote:

I got opposition to having the spec require user gestures for requesting permission from a Firefox developer in https://github.com/w3c/permissions/issues/77. He was fine with allowing UAs to do this. Ben Wells was sympathetic to using 'may' instead of 'must' for this too. Folks may have different opinions about requesting permission vs playing audio, of course.

That's me. Yes, #77 was about mitigating the browser's own chrome UX, which seems out of scope for a spec to meddler with (between agent and its user), so that's why I arrived at "may". I wouldn't necessarily apply the same reasoning here though (I'm neutral atm).

Except: Would this gesture requirement affect Web Audio usage with getUserMedia as a source (example)? I hope not, as in this case the user has already explicitly (through UX) or implicitly (through persistent permissions remembered from a previous visit) approved it, so it seems like it should be exempted.

.: Jan-Ivar :.

[Philip Jägenstedt]

I'm not at all surprised that this is a controversial change. I think what we should hope for, in the end, is a consistent behavior for both <audio> and Web Audio and that all browsers on all platforms have the same behavior, even if the spec allows for variation.

That's a very high bar, but at the very least it's weird for a single browser on a single platform to allow automatic audio playback in one context and not another if the user can't tell the difference. (There is another obvious way of unifying behavior, but let's stick to what is actually being proposed right now.)

Chris, Hongchan, even though you think this is a terrible idea as the default behavior, is there any context where you think disallowing automatic Web Audio playback would make sense (sandboxed iframes?) and what would be a good shape of the API for those situations?

 Jan-Ivar, interaction with gUM-originating MediaStreams is a good question. MediaStreams are special-cased in HTMLMediaElement in Blink as not requiring any user gesture to play, but that's not entirely without issues. It could be that some tab-level tracking of "user allows playback" is actually needed to make sense of all of this.

[Jan-Ivar Bruaroey]

On 6/8/16 11:38 AM, Philip Jägenstedt wrote:

Jan-Ivar, interaction with gUM-originating MediaStreams is a good question. MediaStreams are special-cased in HTMLMediaElement in Blink as not requiring any user gesture to play, but that's not entirely without issues.

Yeah, the one that comes to mind being Chrome deciding to implement https://github.com/w3c/mediacapture-fromelement
 

.: Jan-Ivar :.

[Philip Jägenstedt]

On Wed, Jun 8, 2016 at 5:45 PM Jan-Ivar Bruaroey <j...@mozilla.com> wrote:

On 6/8/16 11:38 AM, Philip Jägenstedt wrote:

Jan-Ivar, interaction with gUM-originating MediaStreams is a good question. MediaStreams are special-cased in HTMLMediaElement in Blink as not requiring any user gesture to play, but that's not entirely without issues.

Yeah, the one that comes to mind being Chrome deciding to implement https://github.com/w3c/mediacapture-fromelement

Yeah, that's https://bugs.chromium.org/p/chromium/issues/detail?id=590632

The comments are interesting there, but I don't know how that's going to be resolved really :/

[Chris Wilson]

On Wed, Jun 8, 2016 at 6:36 AM, Mounir Lamouri <mou...@lamouri.fr> wrote:

Correct. Chrome Android intents to intervene and block Web Audio
autoplay for consistency with its autoplay behaviour. As a product it
doesn't make sense to block autoplay for <audio> but not Web Audio while
you can simply play any audio file via Web Audio. See:
https://mounirlamouri.github.io/sandbox/autoplay/webaudio.html

This is the statement I would argue with.

When Webkit made the decision to block autoplay of media elements on mobile, it was for a couple of defined reasons - mostly, there were a fair amount of pages out there with autoplaying videos, and some with autoplaying audio as well.  I don't think the intent was necessarily that somehow playing any form of audio was something you needed the user's "permission" for - or that a touch/click was a rational permission - it was recognition of a user experience problem with the web as it existed.  To wit, there was a bunch of content out there that would blare video/music on mobile, because it might have made sense to do that on desktop, when that was counter to the expectation of the user in a mobile scenario.

By contrast, Web Audio doesn't have that set of content out there that is problematic for users - Web Audio is useful for games (where this UI block doesn't make sense for elements either, frankly), UI feedback in apps (probably not an issue), sonification of sorts (e.g. using sound to illustrate data - there was a demo in the I/O keynote about this a couple of years ago, playing clicks for queries served Geiger-counter-style), or music apps (which absolutely don't want to break).  Sure, it would be possible to rewrite a (currently autoplay-blocked) <audio> app to use web audio without blocking; I'd argue this isn't "we want this to never happen" as "user expectation was different on vast pre-mobile corpus", and we don't have that vast pre-mobile corpus with Web Audio.

At the very least, if we take it as an imperative that we need to block ANY audio output until there is some user interaction, I would suggest we need a permission to explicitly disable this behavior - so that applications like software synthesizers or guitar amp simulators, which users would absolutely expect to work without any user interaction (I plug my guitar in, it should just work; I plug in a MIDI controller, it should be able to just work too).

I want to make sure that we don't do crazy hacks and that means working
directly with the specification to make sure this is handled as best as
possible. The bare minimum would be to specify WebKit's behaviour. Note
that this is not an intent to ship and we don't know yet for sure what
will ship.

I would wildly object to the Web Audio specification defining this and stating that it is a good idea, and I expect you'd find a plurality of the members of the web audio WG would express the same.  This puts an arbitrary UI roadblock in the way of getting to audio, and it's caused repeated questions (on stackoverflow, e.g.) about why this is.  I don't disagree with the issue you filed on returning promises for buffers, et al, necessarily - but I think that's an interaction feedback issue.

In short, I (and others building on web audio) would argue the user interaction before functioning *IS* a crazy hack.  It might have been expedient for the <audio>/<video> case, but it's not necessary at a lower level.

> Additionally, AudioContexts never switching to running state will cause
> unfortunate timing side effects (we used to have an issue with Chrome not
> starting the clock immediately - this would likely cause a side effect
> like
> this.)  The normal process is that AudioContexts automatically switch to
> running state when created; this would break that expectation.  The
> messiness with ScriptProcessor (captured in your doc) also adds another
> layer of complexity - as would AudioParams being constructible (you could
> use an AudioParam to create sound) (issue
> <https://github.com/WebAudio/web-audio-api/issues/367>).
>
> In short - I think Webkit made a poor decision when they chose to
> implement
> this, and I would hate to see that pattern get propagated to other
> browsers.

I agree that WebKit's implementation is hacky in some ways but I assume
that they had to implement their behaviour without too many side effects
on the rest of the API behaviour. My hope, and this is why I started
this discussion on the spec is that we can come up with a good solution
that acknowledge this use case.

No, when Webkit added this (blocking web audio) there wasn't much Web Audio content around.   I think they felt it was expedient to dump it in the same category, but I don't see the use case as anywhere near as necessary as it was for <audio><video>.

In other words, if the Web Audio specification doesn't accept that some
UA will not allow playback unless the user interacted with the page, UA
will find ways to enforce this restriction and developers will have to
write browser specific code. Embracing this is probably the best
solution for UA, web developers and users.

I don't have a ton of confidence Apple will implement this in the near term anyway; they've fallen behind in a lot of the details that we, Mozilla and Microsoft have tracked together on in Web Audio.  I'll note in the issue you referenced Mozilla suggested coming up with a lighter weight way of doing this (a good comment, actually, given use cases).  I'd go even further and say if we must do this, we should do it by managing the initial state (and state transitions) of the AudioContext, not by blocking individual start()s.  And, as previously stated, we absolutely need a Permission to avoid this behavior, or we are a significantly hackier platform than native platforms for audio.

 > >> Firefox: Public support

> >>
> >
> > Do you have a link to this support?

padenot (see github issue I linked) is the main Mozilla person working
on Web Audio. As you can see he is in favour of the spec change (CC'd
him).

If you mean this comment, I don't see that as positive support - just trying to minimize the API changes to reflect playback state.  Happy to have Paul weigh in here.

[Chris Wilson]

On Wed, Jun 8, 2016 at 8:38 AM, Philip Jägenstedt <foo...@chromium.org> wrote:

I'm not at all surprised that this is a controversial change. I think what we should hope for, in the end, is a consistent behavior for both <audio> and Web Audio and that all browsers on all platforms have the same behavior, even if the spec allows for variation.

Browsers don't always have identical behavior in terms of UX and permissions, though.  I agree that it should be exposed when this is happening, if it is happening - but that ball is also heavily in Webkit's court.

 

That's a very high bar, but at the very least it's weird for a single browser on a single platform to allow automatic audio playback in one context and not another if the user can't tell the difference. (There is another obvious way of unifying behavior, but let's stick to what is actually being proposed right now.)

I'm not sure what you mean by "context" here; but I agree that the user/developer should be able to tell the difference. 

Chris, Hongchan, even though you think this is a terrible idea as the default behavior, is there any context where you think disallowing automatic Web Audio playback would make sense (sandboxed iframes?) and what would be a good shape of the API for those situations?

I think it's POSSIBLE that we would want to disallow automatic audio output in some situations - but to me, that is more a detail I'd like to associate with a permission than something I want to chain to a UI gesture.  (And I don't think the default situation calls for this, either.  Even in desktop, it seems most audio has started being muted by the developer in potentially-annoying contexts, because they know they'll lose users otherwise.)

 

 Jan-Ivar, interaction with gUM-originating MediaStreams is a good question. MediaStreams are special-cased in HTMLMediaElement in Blink as not requiring any user gesture to play, but that's not entirely without issues. It could be that some tab-level tracking of "user allows playback" is actually needed to make sense of all of this.

That's DEFINITELY not without issues.  It gets harder and harder to explain what can play and what can't, and what happens to the timeline.

[Dimitri Glazkov]

The discussion in this thread indicates that before we get to implementation, we might want to come up with an approach that all stakeholders agree on (the consensus thing). Mounir, Chris, would you be willing to facilitate that? I would be happy if y'all work it out in this thread -- whatever's the most optimal way to reach agreement.

:DG<

[Rick Byers]

If you guys will be there, this seems like a great discussion topic for BlinkOn!

We've repeatedly found ourselves having to retroactively restrict access to potentially annoying features on the web (starting with pop-up blockers).  There's a chicken and egg problem here.  When a feature is new and not widely supported, there's little incentive to try to abuse it and so legitimately little appetite for placing restrictions on it.  But history hast shown that in general anything that can be used to grab user's attention WILL be abused (eg. for ad monitization purposes).  And tightening restrictions after lots of content has been created under the existing model is hugely painful for users and developers due to the compat implications.  So I applaud Mounir's attempt to try to be more disciplined and unified about the approach.  This is probably the only way to work towards good long-term tradeoffs (rather than unfortunate hacks determined primary by web compat reality).

That said I agree there are some really difficult tradeoffs here and we absolutely don't want to unecessarily restrict what sort of UX can be built.  Blindly following Apple's lead is not a recipe for good stewardship of the web platform and healthy long-term tradeoffs.  IMHO Apple cares primarily about the user experience on the web today, and isn't particularly motivated (as a company anyway, not talking about specific engineers) to improve the capabilities of the web as a platform.

The good news is that if we're thoughtful and creative about this up-front, we have a huge array of tools at our disposal.  Eg:

• explicit permissions
• site engagement service (like add to homescreen)
• treating sandboxed and/or cross-origin iframes more strictly than the main document
• karma?

I don't have any answers, but I know we'll only strike a good long-term tradeoff for the platform by having folks on both side of the debate collaborate closely and creatively. 

[Chris Wilson]

1) Yes, I'm happy to work with Mounir to resolve.  Will follow up.

2) Rick, usually I'd be right with you on not needing to reclose Pandora's Box.  But audio abuse is actually not such a huge problem - the issues that led to not autoplaying were at least as related to not auto-LOADing (which isn't relevant to Web Audio), and even on the desktop web, this is why audio is usually auto-muted even when autoplay is supported.  The blare of audio is SUCH a strong deterrent that people leave sites and don't come back.  With explicit permission control, I still think it's solving a largely non-problem, but it would be at least able to be worked around.

[Kenji Baheux]

For what it's worth, I have heard reports that one of the largest group of advertisers is pushing for requiring unumuted state in order to record an ad has viewed (on top of the existing viewability requirements). As one can imagine, these unmuted ads tend to be more profitable (by necessity).

With these bad incentives, the risk that a non negligible amount of websites (say with unique content or appeal) will decide to downplay user retention concerns is palpable.

Afterall, while slightly less annoying, there is no shortage of examples where user retention was belittled: pop-over, interstitials, obnoxious app banners, heavy JS workaround for autoplay, etc...

I hope we can find a solution that doesn't end up like disabling autoplay.