Behavior of background tabs and webapps

[Brad Lassey]

Recently several bugs have been filed which boil down to weighing the functionality of background tabs in Firefox for Android against potential battery life issues that functionality brings. Since this sort of balancing act is also of interest people working on B2G and WebAPIs teams we had a call to talk through some of the implications. Now that we've fleshed out the discussion, we'd like to take it to the news groups. Hence this post.

First the question of what constitutes a background tab or web app was debated (I'm just going to call them background tabs going forward). At first defining background tabs as any tab without focus was assumed. Then Chris Jones pointed out that it will be possible in the future to interact with 2 or more windows simultaneously on a big enough multitouch screen. Given that, we settled on defining background tabs as any tab which is not currently visible. The question was then raised about hidden iframes in otherwise visible tabs. It is assumed that any iframe in a visible tab would be considered visible and therefore foreground.

Then to frame the discussion we brainstormed about some scenarios which we would specifically like to enable for background tabs:
* Listening to Pandora in the background
* run/jog tracking app
* gmail/twitter synching in background
* skype receiving phone calls
* skype remaining on call when switching to another app
* sms app receiving messages
* phone app receiving phone calls
* apps that need to be opened from startup
* webapps that use WebSockets or DOMRemoteEvents to get push messages and show notifications

During this brainstorming session we identified a scenario that we could not agree on and that is, if a user is playing a game and switches to check their email should the game continue to play (i.e. character continues to move through the world, music plays etc.). It is without doubt that the game should be able to detect this with existing APIs and pause itself for the sake of a good user experience. The question is whether the platform should enforce that behavior. This comes down to "the iOS model" vs. "the desktop model" and we determined that gecko as a platform should support both and the front end (Firefox for Android, B2G, Joe's awesome app engine) should be what makes that sort of policy decision.

Specifically, the things that we want to consider throttling or stopping when a tab is in the background are timer events, sensor events, video decoding and WebGL. There may be others, and we should identify them.

Timer events are on the list because they cause the tab to do work that isn't high priority for the user at the time. Sensor events are on the list for similar reasons, but in addition to that require the device to power sensors (gps, accelerometer, proximity sensor etc.) unnecessarily (perhaps?).

Video decoding and WebGL are on the list because the user can't see the product of these activities if the tab is not visible. Specifically, in the case of video decoding, audio should still be played, to support the case of listening to a video lecture or meeting which doing other things. Video should probably still be decoded if the page is reading the decoded data through js.

Finally, to frame the discussion we thought it would be most useful to define a set of states which a tab could be in. Up to this point we've only though about foreground and background, but given that we want to be able to throttle, pause or even kill background tabs to save batter and memory we came up with the following set of states and their definitions:
states:
foreground -> background -> paused -> killed -> closed

foreground: currently active tab or equivalent
background: current non-active tab behavior
paused: bfcache
killed: session data saved, but no longer loaded
closed: explicitly quit by user, no session data saved

Closed and killed may in the end be the same thing, with the only difference being a front end policy decision on whether to restore the session when reviving.

Assuming no one disagrees strongly with the basics we've laid out (big assumption, I know), the discussion we'd like to have on the news groups is what should be throttled and by how much in the background state. There is also a good discussion to be had about what should cause a tab to go into the paused or killed state, but I'd like to specifically exclude that from this thread as it should be a front-end policy decision which the gecko platform will need to support.

[Ian Melven]

----- Original Message -----
From: "Brad Lassey" <br...@lassey.us>
To: dev-pl...@lists.mozilla.org
Sent: Friday, March 30, 2012 9:03:47 AM
Subject: Behavior of background tabs and webapps

> Sensor events are on the list for similar reasons, but in addition to that require the device to power sensors (gps, accelerometer, proximity sensor etc.) unnecessarily (perhaps?).

just a small note to highlight that there are privacy and security reasons that it's desirable to block sensor events when a tab is backgrounded as well (using the accelerometer to
obtain keyboard events and hence keystrokes has been a hot area of research over the last year)

thanks,
ian

[Doug Turner]

On Monday, April 2, 2012 1:50:22 PM UTC-7, Ian Melven wrote:
> ----- Original Message -----
> From: "Brad Lassey"
>

> To: dev-pl...@lists.mozilla.org
> Sent: Friday, March 30, 2012 9:03:47 AM
> Subject: Behavior of background tabs and webapps
>
> > Sensor events are on the list for similar reasons, but in addition to that require the device to power sensors (gps, accelerometer, proximity sensor etc.) unnecessarily (perhaps?).
>
> just a small note to highlight that there are privacy and security reasons that it's desirable to block sensor events when a tab is backgrounded as well (using the accelerometer to
> obtain keyboard events and hence keystrokes has been a hot area of research over the last year)
>
> thanks,
> ian

I think some application will want sensor events (like orientation, acceleration, gps) when the UA is in the background. Consider a navigation application who uses those 3 sensors. If you switch to a phone call, do you really want to have the navigation application stop seeing data from the sensors... :)

[Justin Lebar]

For apps which want to continue processing in real time in the
background (e.g. Pandora, your voip client when you're on a call, your
turn-by-turn directions app, perhaps your jog tracking app), we're
going to have to add an API which allows the app to request permission
to "run unimpeded in the background".

We need this "run in the background" API so that, when we do throttle
and/or kill apps, we don't throttle or kill those apps which you'd
like to continue running.

Similarly, we need a separate API allowing your Gmail to periodically
check your e-mail. This isn't the same as "run unimpeded in the
background", but rather is "get woken up occasionally to do some
stuff." We need this API so we know that even after we've killed
Gmail due to memory pressure (or whatever), we need to re-load it
eventually so it can do its thing.

Since we need to give pages a way to ask to stay alive and be awoken,
it's my view that we should aggressively throttle, pause, and kill
other pages. My preference would be never to enter the "background"
state for apps which don't request it. We'd move straight from
"active" to "paused" (modulo giving the page a few seconds to finish
its network requests, or whatever). Audio would immediately stop when
the tab ceased to be active, timeouts would immediately stop firing,
etc.

Basically, I think that the requirement that we be able to kill any
background tab at any time (e.g. for memory reasons) is what makes the
mobile case so different from the desktop case. Since apps will have
to explicitly ask to be guaranteed to stay alive, we should
aggressively pause and kill those apps which don't.

-Justin

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[Boris Zbarsky]

On 3/30/12 12:03 PM, Brad Lassey wrote:
> Assuming no one disagrees strongly with the basics we've laid out (big assumption, I know), the discussion we'd like to have on the news groups is what should be throttled and by how much in the background state.

So the current state of things is that the following are throttled in
the background state:

1) Nested setTimeout/setInterval: the clamp is raised from 4ms in
foreground tabs to 1000ms in background ones.
2) Everything hanging off the refresh driver: reflow, restyling,
painting, requestAnimationFrame callbacks, SMIL animations, CSS
transitions, CSS animations. The refresh tick changes from 16ms to
(2^N)*1000ms, where N is the number of ticks that have happened so far
in the background state.

There is talk (and the spec currently calls for it) of turning
requestAnimationFrame callbacks off entirely in throttled tabs.

The things you list in your mail are definitely good candidates to be
throttled.

-Boris

[Taras Glek]

On 3/30/2012 9:03 AM, Brad Lassey wrote:
> Recently several bugs have been filed which boil down to weighing the functionality of background tabs in Firefox for Android against potential battery life issues that functionality brings. Since this sort of balancing act is also of interest people working on B2G and WebAPIs teams we had a call to talk through some of the implications. Now that we've fleshed out the discussion, we'd like to take it to the news groups. Hence this post.
>
> First the question of what constitutes a background tab or web app was debated (I'm just going to call them background tabs going forward). At first defining background tabs as any tab without focus was assumed. Then Chris Jones pointed out that it will be possible in the future to interact with 2 or more windows simultaneously on a big enough multitouch screen. Given that, we settled on defining background tabs as any tab which is not currently visible. The question was then raised about hidden iframes in otherwise visible tabs. It is assumed that any iframe in a visible tab would be considered visible and therefore foreground.

There is some related work that needs to happen for Snappy.
https://bugzilla.mozilla.org/show_bug.cgi?id=675539 is for unloading
idle tabs in the frontend.

Additionally, we will be adding infrastructure to punish background tab
cpu hogs https://bugzilla.mozilla.org/show_bug.cgi?id=712731


Taras

[Neil]

Brad Lassey wrote:

>states:
>foreground -> background -> paused -> killed -> closed
>
>foreground: currently active tab or equivalent
>background: current non-active tab behavior
>paused: bfcache
>killed: session data saved, but no longer loaded
>closed: explicitly quit by user, no session data saved
>

I would like (an API) to move a tab from background to paused.

--
Warning: May contain traces of nuts.

[Gervase Markham]

On 30/03/12 17:03, Brad Lassey wrote:
> During this brainstorming session we identified a scenario that we
> could not agree on and that is, if a user is playing a game and
> switches to check their email should the game continue to play (i.e.
> character continues to move through the world, music plays etc.). It
> is without doubt that the game should be able to detect this with
> existing APIs and pause itself for the sake of a good user
> experience. The question is whether the platform should enforce that
> behavior. This comes down to "the iOS model" vs. "the desktop model"
> and we determined that gecko as a platform should support both and
> the front end (Firefox for Android, B2G, Joe's awesome app engine)
> should be what makes that sort of policy decision.

Does this question boil down to: "should we actually throttle timer
events or not"? If not, how does it reflect into the list of things you
want to consider throttling?

> Specifically, the things that we want to consider throttling or
> stopping when a tab is in the background are timer events, sensor
> events, video decoding and WebGL. There may be others, and we should
> identify them.

This may be a dumb question, but why is WebGL a particular issue? If
it's not animated (which falls under "timer events" anyway), then surely
a rendered WebGL 3d image is just an image, like any other? Or does
WebGL have its own animation primitives?

> Timer events are on the list because they cause the tab to do work
> that isn't high priority for the user at the time. Sensor events are
> on the list for similar reasons, but in addition to that require the
> device to power sensors (gps, accelerometer, proximity sensor etc.)
> unnecessarily (perhaps?).

For sensor events, can we remember which sensors a page was using, and
immediately dispatch an event for each one when the page is
foregrounded, no matter what other timing or throttling we would
normally apply to how often events were posted?

> meeting which doing other things. Video should probably still be
> decoded if the page is reading the decoded data through js.

That sounds like a way a page could kill your performance without the
page author realising that it's what they are doing. If I'm reading
decoded video data in order to copy it to a canvas with an overlay (e.g.
a subtitle or something), it seems to me that this is just as useless in
a background tab as straight video decoding.

Can we ask the page to set policy here, and default to "not decoded"?

Gerv

[Gervase Markham]

On 02/04/12 22:01, Justin Lebar wrote:
> Since we need to give pages a way to ask to stay alive and be
> awoken, it's my view that we should aggressively throttle, pause, and
> kill other pages.

The downside to this policy is that pages then need to be explicitly
coded to use the new APIs to work as apps; if they aren't, they'll be
unbackground-able.

I don't know what the B2G/Apps team's view is on whether one of our
goals is for current web pages to work well as apps without special
enhancement, but if we'd like that, then our defaults for background
permissions stuff might need to support that use case.

Gerv

[Benoit Jacob]

2012/4/3 Gervase Markham <ge...@mozilla.org>:

>> Specifically, the things that we want to consider throttling or
>> stopping when a tab is in the background are timer events, sensor
>> events, video decoding and WebGL. There may be others, and we should
>> identify them.
>
> This may be a dumb question, but why is WebGL a particular issue? If
> it's not animated (which falls under "timer events" anyway), then surely
> a rendered WebGL 3d image is just an image, like any other? Or does
> WebGL have its own animation primitives?

That's not a dumb question at all. I don't see WebGL as being anything
special by itself in this regard. WebGL is not an animation method,
it's a drawing API. Drawing with WebGL is not generally more or less
expensive than with other APIs like SVG or Canvas 2D, at least that
depends considerably on the use case and on the system.

WebGL animations use the same animation mechanisms as others:
requestAnimationFrame or setTimeout.

FWIW, I am strongly in the camp that favors not sending any
requestAnimationFrame callback to background tabs. More generally,
tying requestAnimationFrame to actual compositing. People who want to
get callbacks regardless of compositing can use setTimeout. What I
don't know is how much setTimeout should be throttled for background
tabs.

Cheers,
Benoit

[Justin Lebar]

On Tue, Apr 3, 2012 at 5:04 AM, Gervase Markham <ge...@mozilla.org> wrote:
> On 02/04/12 22:01, Justin Lebar wrote:
>> Since we need to give pages a way to ask to stay alive and be
>> awoken, it's my view that we should aggressively throttle, pause, and
>> kill other pages.
>
> The downside to this policy is that pages then need to be explicitly
> coded to use the new APIs to work as apps; if they aren't, they'll be
> unbackground-able.

That's not the downside of specifically this policy. It's the
downside of the policy of "we can shoot any tab at any time for the
sake of memory pressure", which, as far as I know, is common to all
proposals here.

Indeed, Fennec (at one point, anyway) would nuke all but the active
tab on memory pressure. If you later opened a nuked tab, it would be
session-restored.

We're even thinking of doing something like this in desktop Firefox [1].

> I don't know what the B2G/Apps team's view is on whether one of our
> goals is for current web pages to work well as apps without special
> enhancement, but if we'd like that, then our defaults for background
> permissions stuff might need to support that use case.

None of the proposals thus far would allow background apps to work
without modification. I do not think unmodified background apps
working is compatible with allowing us to close apps on memory
pressure, and I think the latter is critical to the platform's
success.


I'd further argue that we'd need a "background app" API even if we
were to decide never to shoot apps when we're under memory pressure.
(Therefore we might as well aggressively pause and kill apps.) On a
mobile device, when the screen goes to sleep, we would like to put the
CPU into a low-power state.

But obviously, if you're playing music, or you're on a phone call, or
whatever, we want the CPU to stay alive.

Android has the concept of a "CPU wake lock" for this purpose. We're
implementing something similar for b2g. [2]. It will be a privileged
API, just like geolocation.

It seems to me that we can just say "if you hold a CPU wake lock,
we'll make every effort not to kill you while you're in the
background. Otherwise, we may aggressively pause you, and we may kill
you at any point."

If an app really wanted to stay alive only while your phone's screen
was on, then turn off as soon as the screen turns off, it could hold a
CPU wake lock and listen for an event indicating that the screen is
off. But I can't think of a use-case for this behavior, so I suspect
it would be rare.

-Justin

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=675539
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=739913

> Gerv

[Brad Lassey]

On Tuesday, April 3, 2012 5:01:25 AM UTC-4, Gervase Markham wrote:
> On 30/03/12 17:03, Brad Lassey wrote:
> > During this brainstorming session we identified a scenario that we
> > could not agree on and that is, if a user is playing a game and
> > switches to check their email should the game continue to play (i.e.
> > character continues to move through the world, music plays etc.). It
> > is without doubt that the game should be able to detect this with
> > existing APIs and pause itself for the sake of a good user
> > experience. The question is whether the platform should enforce that
> > behavior. This comes down to "the iOS model" vs. "the desktop model"
> > and we determined that gecko as a platform should support both and
> > the front end (Firefox for Android, B2G, Joe's awesome app engine)
> > should be what makes that sort of policy decision.
>
> Does this question boil down to: "should we actually throttle timer
> events or not"? If not, how does it reflect into the list of things you
> want to consider throttling?

My assertion is that the platform needs to support throttling, and it is the "front end's" job to define a policy of how things should be throttled and when.

>
> > Specifically, the things that we want to consider throttling or
> > stopping when a tab is in the background are timer events, sensor
> > events, video decoding and WebGL. There may be others, and we should
> > identify them.
>
> This may be a dumb question, but why is WebGL a particular issue? If
> it's not animated (which falls under "timer events" anyway), then surely
> a rendered WebGL 3d image is just an image, like any other? Or does
> WebGL have its own animation primitives?

Chris Jones called out WebGL as something to consider treating specially for the purpose of killing its context to free up resources.

>
> > Timer events are on the list because they cause the tab to do work
> > that isn't high priority for the user at the time. Sensor events are
> > on the list for similar reasons, but in addition to that require the
> > device to power sensors (gps, accelerometer, proximity sensor etc.)
> > unnecessarily (perhaps?).
>
> For sensor events, can we remember which sensors a page was using, and
> immediately dispatch an event for each one when the page is
> foregrounded, no matter what other timing or throttling we would
> normally apply to how often events were posted?
>
> > meeting which doing other things. Video should probably still be
> > decoded if the page is reading the decoded data through js.
>
> That sounds like a way a page could kill your performance without the
> page author realising that it's what they are doing. If I'm reading
> decoded video data in order to copy it to a canvas with an overlay (e.g.
> a subtitle or something), it seems to me that this is just as useless in
> a background tab as straight video decoding.

I think this is where the front end would have to save you by going from background to paused. The general idea is for the functionality of a page that a user can perceive to not be compromised when in the background.

> Can we ask the page to set policy here, and default to "not decoded"?

This is the sort of wake lock stuff that would be needed to prevent the page from getting paused or killed.

[Gervase Markham]

On 03/04/12 16:27, Justin Lebar wrote:
> That's not the downside of specifically this policy. It's the
> downside of the policy of "we can shoot any tab at any time for the
> sake of memory pressure", which, as far as I know, is common to all
> proposals here.

A fair point! :-)

> We're even thinking of doing something like this in desktop Firefox [1].

This might actually make the problem better - because more site owners
would code with this assumption.

Gerv

[Benoit Jacob]

2012/4/3 Brad Lassey <br...@lassey.us>:

Ah, this kind of throttling. Yes, that makes lots of sense. Already
considering doing this real soon now.

WebGL contexts can get lost at any time. That currently doesn't happen
often, so most WebGL apps don't currently handle webglcontextlost
events, but that will have to change. The only way to get there is to
gradually accustom WebGL app developers to that concept. So it's a
good idea that we start gradually generating WebGL context losses when
we need to.

Benoit

>>
>> > Timer events are on the list because they cause the tab to do work
>> > that isn't high priority for the user at the time. Sensor events are
>> > on the list for similar reasons, but in addition to that require the
>> > device to power sensors (gps, accelerometer, proximity sensor etc.)
>> > unnecessarily (perhaps?).
>>
>> For sensor events, can we remember which sensors a page was using, and
>> immediately dispatch an event for each one when the page is
>> foregrounded, no matter what other timing or throttling we would
>> normally apply to how often events were posted?
>>
>> > meeting which doing other things. Video should probably still be
>> > decoded if the page is reading the decoded data through js.
>>
>> That sounds like a way a page could kill your performance without the
>> page author realising that it's what they are doing. If I'm reading
>> decoded video data in order to copy it to a canvas with an overlay (e.g.
>> a subtitle or something), it seems to me that this is just as useless in
>> a background tab as straight video decoding.
> I think this is where the front end would have to save you by going from background to paused. The general idea is for the functionality of a page that a user can perceive to not be compromised when in the background.
>
>> Can we ask the page to set policy here, and default to "not decoded"?
> This is the sort of wake lock stuff that would be needed to prevent the page from getting paused or killed.

[Henri Sivonen]

On Tue, Apr 3, 2012 at 6:27 PM, Justin Lebar <justin...@gmail.com> wrote:
> That's not the downside of specifically this policy.  It's the
> downside of the policy of "we can shoot any tab at any time for the
> sake of memory pressure", which, as far as I know, is common to all
> proposals here.

...

> None of the proposals thus far would allow background apps to work
> without modification.

If background app tabs were evicted from memory last in the case a
memory pressure, apps could work without modification, since the user
action of making a tab an app tab would be the indication against
evicting that tab.

--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/

[Justin Lebar]

On Thu, Apr 5, 2012 at 3:25 AM, Henri Sivonen <hsiv...@iki.fi> wrote:
> If background app tabs were evicted from memory last in the case a
> memory pressure, apps could work without modification, since the user
> action of making a tab an app tab would be the indication against
> evicting that tab.

On b2g and (currently) on Fennec, there is no distinction between
"normal" tabs and "app" tabs.

Nor do I think we want to create this distinction, certainly not on
b2g. This basically amounts to telling the user that they need to
indicate when they want their app to stay alive. I don't want to have
to indicate that I want my music player to stay alive -- It knows it's
playing music, and it should take care of this for me.

Indeed, I think we rely too much on the distinction between "sites"
and "apps" in general. They're all "web things" -- that's the whole
point of b2g and our apps initiatives.

I also think app tabs are a power-user feature on desktop; I think
it's unreasonable to expect a "regular" user to know that Pandora is
being randomly killed because they didn't right-click and choose "pin
as app tab".

-Justin