Intent to Intervene: cross origin iframes require user gesture for Web Audio on Android

[Mounir Lamouri]

Contact emails

mlam...@chromium.org

Spec

WICG/interventions

Web Audio bug entry

Summary

On Android, using Web Audio inside a cross origin iframe will no longer be allowed if the request isn’t made from an event handler linked to a user gesture.

Motivation

Chrome currently has restrictions on autoplay which do not apply to Web Audio. That means that users can go to a website that can play anything on page load without any kind of user interaction. In order to reduce the surface for abuse without breaking genuine content, we can apply autoplay restrictions to cross origins iframes

Interoperability and Compatibility Risk

There is some compatibility risk but this is a compromise (thus the intent to intervene). We have metrics regarding Web Audio and user gesture and even though these metrics are overly conservative, we are afraid blocking Web Audio for top frames might have too much of web compat issues. However, usage on cross origin iframes is low. The assumption is that autoplaying Web Audio on cross origin iframes is unlikely a genuine use case so breaking these calls might be a risk worth taking to improve the user experience.

Regarding interoperability, Safari iOS doesn’t allow Web Audio to autoplay (not only in cross origin iframes). To improve interoperability, the applied restrictions will match WebKit’s restrictions. As pointed in the Web Audio bug, blocking playback should be incorporated into the Web Audio specification.

With future work and improvements regarding autoplay, we should consider a way for a top frame to allow an iframe to autoplay Web Audio.

Ongoing technical constraints

No.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Android only, including WebView. Technically, not limited to Android but it will only apply when the flag to disable autoplay is on which is a WebView setting and otherwise only on on Android.

OWP launch tracking bug

https://crbug.com/614115

Link to entry on the feature dashboard

https://www.chromestatus.com/feature/6406908126691328

Requesting approval to ship?

Yes

-- Mounir

[Chris Harrelson]

What about the use case of a cross-origin iframe that has a WebAudio player, but relies on the main frame for UI and start-playing logic via postMessage? Is it possible to implement such a site?

Chris

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Philip Jägenstedt]

I think that if audio playback were a permission, then this should be possible with permission delegation. But it's not a permission. Mounir, WDYT?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Harald Alvestrand]

There's a "speaker" permission defined as part of the media permissions set.

It would be logical to apply that to all cases where permission to play sound is questioned.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Mounir Lamouri]

As pointed in the original message, we will look into a solution for top
frames to opt-in an iframe to autoplay. Whether to use something like
Feature Policy, Permission Delegation or something else is orthogonal to
this.

As far as we know, cross origin Web Audio is rare enough on Chrome
Android that it doesn't require to solve this problem in order to get
the intervention going. Our numbers say that ~0.23% of page load have a
cross origin AudioContext created (based on Chrome Android Beta
population). Those that don't use a user gesture are probably a fraction
of them. The reason why we are confident it will have low breakage is
that these contents would be broken on Safari iOS.

-- Mounir

On Thu, 8 Sep 2016, at 09:01, 'Harald Alvestrand' via blink-dev wrote:
> There's a "speaker" permission defined as part of the media permissions
> set.
>
> It would be logical to apply that to all cases where permission to play
> sound is questioned.
>
>
> On Thu, Sep 8, 2016 at 9:48 AM, Philip Jägenstedt <foo...@chromium.org>
> wrote:
>
> > I think that if audio playback were a permission, then this should be
> > possible with permission delegation. But it's not a permission. Mounir,
> > WDYT?
> >
> > On Thu, Sep 8, 2016 at 2:24 AM Chris Harrelson <chri...@chromium.org>
> > wrote:
> >
> >> What about the use case of a cross-origin iframe that has a WebAudio
> >> player, but relies on the main frame for UI and start-playing logic via
> >> postMessage? Is it possible to implement such a site?
> >>
> >> Chris
> >>
> >> On Wed, Sep 7, 2016 at 6:01 AM, Mounir Lamouri <mou...@lamouri.fr> wrote:
> >>

> >> *Contact emails*
> >> mlam...@chromium.org
> >>
> >> *Spec*
> >> WICG/interventions <https://github.com/WICG/interventions/issues/28>
> >> Web Audio bug entry
> >> <https://github.com/WebAudio/web-audio-api/issues/836>
> >>
> >> *Summary*

> >> On Android, using Web Audio inside a cross origin iframe will no longer
> >> be allowed if the request isn’t made from an event handler linked to a user
> >> gesture.
> >>

> >> *Motivation*

> >> Chrome currently has restrictions on autoplay which do not apply to Web
> >> Audio. That means that users can go to a website that can play anything on
> >> page load without any kind of user interaction. In order to reduce the
> >> surface for abuse without breaking genuine content, we can apply autoplay
> >> restrictions to cross origins iframes
> >>

> >> *Interoperability and Compatibility Risk*

> >> There is some compatibility risk but this is a compromise (thus the
> >> intent to intervene). We have metrics regarding Web Audio and user gesture
> >> and even though these metrics are overly conservative, we are afraid
> >> blocking Web Audio for top frames might have too much of web compat issues.
> >> However, usage on cross origin iframes is low. The assumption is that
> >> autoplaying Web Audio on cross origin iframes is unlikely a genuine use
> >> case so breaking these calls might be a risk worth taking to improve the
> >> user experience.
> >>
> >> Regarding interoperability, Safari iOS doesn’t allow Web Audio to
> >> autoplay (not only in cross origin iframes). To improve interoperability,
> >> the applied restrictions will match WebKit’s restrictions. As pointed in
> >> the Web Audio bug, blocking playback should be incorporated into the Web
> >> Audio specification.
> >>
> >> With future work and improvements regarding autoplay, we should consider
> >> a way for a top frame to allow an iframe to autoplay Web Audio.
> >>

> >> *Ongoing technical constraints*
> >> No.
> >>
> >> *Will this feature be supported on all six Blink platforms (Windows, Mac,
> >> Linux, Chrome OS, Android, and Android WebView)?*

> >> Android only, including WebView. Technically, not limited to Android but
> >> it will only apply when the flag to disable autoplay is on which is a
> >> WebView setting and otherwise only on on Android.
> >>

> >> *OWP launch tracking bug*
> >> https://crbug.com/614115
> >>
> >> *Link to entry on the feature dashboard*
> >> https://www.chromestatus.com/feature/6406908126691328
> >>
> >> *Requesting approval to ship?*

> >> Yes
> >>
> >> -- Mounir
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "blink-dev" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to blink-dev+...@chromium.org.

> >>
> >> --
> > You received this message because you are subscribed to the Google Groups
> > "blink-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send an

> > email to blink-dev+...@chromium.org.

> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "blink-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to blink-dev+...@chromium.org.

[Philip Jägenstedt]

Yeah, a solution for delegating the autoplay capability to iframes seems orthogonal to this, and in the absence of things known to break I don't think that should block this intervention. If the implementation matches WebKit's, then people can already not depend on it to work.

I think we should do this, but what's the state of the spec discussion, might things still change in any significant way?

[Rick Byers]

This seems reasonable to me, especially given that Safari iOS already has a stricter restriction (if we can show abuse is exceedingly rare, maybe we could get them to relax theirs to match ours?).   Do any desktop browsers have a similar restriction?

1 in 400 page views seemed surprisingly high to me.  That's the 'AudioContextCrossOriginIframe' UseCounter, right?  I took at a look at the internal data and I see ~0.05% of pageVisits on Android and ~0.02% on Windows (in beta).  That's still not tiny, but given the consistency with Safari it's good enough for me.   LGTM1.

Please record metrics on how often this block is hit, and be sure to log a warning in the console saying the audio request was blocked with a link to the chromestatus entry (or better resource if one exists).  Also thanks for starting the discussion of how best to expose this failure mode to the API, please keep pushing that along!

> >> email to blink-dev+unsubscribe@chromium.org.

> >>
> >> --
> > You received this message because you are subscribed to the Google Groups
> > "blink-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send an

> > email to blink-dev+unsubscribe@chromium.org.

> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "blink-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to blink-dev+unsubscribe@chromium.org.

[Dimitri Glazkov]

LGTM2.

> >> email to blink-dev+...@chromium.org.

> >>
> >> --
> > You received this message because you are subscribed to the Google Groups
> > "blink-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send an

> > email to blink-dev+...@chromium.org.

> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "blink-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to blink-dev+...@chromium.org.

[Philip Jägenstedt]

LGTM3 to implement and ship when spec change has been merged. If that gets bogged down and shipping without the spec change starts looking like the best option, please poke the thread so that we know what happened.

[Mounir Lamouri]

To follow-up on this, I went to the Audio WG at TPAC and we discussed
the issue of user gesture restrictions in the Web Audio API. We agreed
on a behaviour that I've implemented in Chromium. I've also updated the
specification.

For those who care about the details, the specification allows the user
agent to block an AudioContext from activating at construction time and
when resume() is called. WebKit has a similar behaviour but they also
allow a node calling start() to activate the AudioContext. This is
probably from the time when resume() didn't exist so the group preferred
to try without this first. I have a pending CL adding metrics to check
how much content would be broken. Chrome 55 will ship this behaviour
only for cross origin iframes on Android so we are fine breaking content
but being aware of how much calls to node's start() methods would have
activated the AudioContext will be interesting.

-- Mounir

On Thu, 8 Sep 2016, at 18:49, Philip Jägenstedt wrote:
> LGTM3 to implement and ship when spec change has been merged. If that
> gets
> bogged down and shipping without the spec change starts looking like the
> best option, please poke the thread so that we know what happened.
>
> On Thu, Sep 8, 2016 at 5:44 PM Dimitri Glazkov <dgla...@chromium.org>
> wrote:
>
> > LGTM2.
> >
> > On Thu, Sep 8, 2016 at 7:53 AM Rick Byers <rby...@chromium.org> wrote:
> >
> > This seems reasonable to me, especially given that Safari iOS already has
> > a stricter restriction (if we can show abuse is exceedingly rare, maybe we
> > could get them to relax theirs to match ours?). Do any desktop browsers
> > have a similar restriction?
> >
> > 1 in 400 page views seemed surprisingly high to me. That's the
> > 'AudioContextCrossOriginIframe' UseCounter, right? I took at a look at the internal
> > data

> > <https://uma.googleplex.com/p/chrome/timeline_v2/?sid=dfb32c0bd137d741bd4d2357dd25bdbf> and

> > I see ~0.05% of pageVisits on Android and ~0.02% on Windows (in beta).
> > That's still not tiny, but given the consistency with Safari it's good
> > enough for me. LGTM1.
> >
> > Please record metrics on how often this block is hit, and be sure to log a
> > warning in the console saying the audio request was blocked with a link to
> > the chromestatus entry (or better resource if one exists). Also thanks for
> > starting the discussion of how best to expose this failure mode to the API

> > <https://github.com/WebAudio/web-audio-api/issues/836>, please keep

[Rick Byers]

Thank you for the follow-up.  Sounds great!

> > > >> email to blink-dev+unsubscribe@chromium.org.

> > > >>
> > > >> --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "blink-dev" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > an

> > > > email to blink-dev+unsubscribe@chromium.org.

> > > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "blink-dev" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an

> > > email to blink-dev+unsubscribe@chromium.org.

> >
> >
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "blink-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to blink-dev+unsubscribe@chromium.org.