Intent-to-implement-and-ship: Purge + suspend

[Kentaro Hara]

Hi

Contact emails

har...@chromium.org, ta...@chromium.org

Spec

None.

Summary

When the system is under high memory pressure, purge as much memory as possible from background tabs and suspend them (Design document)

Motivation

Imagine that the OS is under high memory pressure and Chrome is running 1 foreground tab and N background tabs. The most efficient way to reduce the memory would be to purge memory from the N background tabs.

With that motivation in mind, we propose purge + suspend. Specifically, we purge as much memory as possible and then immediately suspend the backgrounded tabs (in order not to let the purged memory come back). This may break web compatibility because the suspended tabs stop running (e.g., favicon update, tab title update, notifications). To mitigate the concern, we resume the suspended tabs periodically (e.g., 2 mins).

tasak@ implemented the prototype and collected performance/memory numbers. The high-level summary is as follows:

• In average, the purge + suspend reduced 28% of renderer's memory. In particular, it reduced 72% from Facebook and 61% from Tumblur. The number looks very promising.

• In average, the number stays at 28% even after (periodically) resuming the suspended tab. This means that periodic resuming doesn't resurrect much memory in common websites. There are a couple of websites (e.g. Google Drive, Yahoo) where periodic resuming resurrects a lot of memory, but the resurrected memory is gone when the renderer is purged again.

• To evaluate the tab-switching cost, we manually created a custom Chrome that forcibly suspends all background tabs every 1 second and crawled many websites. However, we didn't observe any UX regression.

See this document for more detailed performance/memory analysis.

Interoperability and Compatibility Risk

First of all, we blacklist tabs that are playing video, audio etc using the mechanism of the existing tab-discarding. Second, we give the suspended tabs a chance to wake up periodically so that the tabs can update favicons, tab titles, notifications etc.

Remember that the tab-suspension may break compatibility, but at the very least it will provide a much better UX than the existing tab-discarding in terms of both compatibility and tab-switching cost.

Ongoing technical constraints

Experiments are needed to optimize heuristics to determine the purging targets (work-in-progress CL) and the timings. Our plan is to start a Finch experiment, collect more performance/memory numbers from the wild, and optimize the heuristics.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes. On all platforms that have background tabs.

OWP launch tracking bug

http://crbug.com/635419

Requesting approval to ship?

Yes. As mentioned above, we want to start a Finch experiment to collect more performance/memory numbers and optimize the heuristics.

--

Kentaro Hara, Tokyo, Japan

[ben.m...@gmail.com]

How would this impact a site that uses long polling for things like chat? Even if the tabs are woken up frequently the experience of having chat messages be delayed by 2 minutes could be pretty bad. Is there any way that a site could be cooperative in this scheme -- eg to tell the site not to do operations that cause layout / stylesheet info to be allocated.

-b

[Kentaro Hara]

How would this impact a site that uses long polling for things like chat? Even if the tabs are woken up frequently the experience of having chat messages be delayed by 2 minutes could be pretty bad.

That's a valid concern, but we can mitigate the risk by blacklisting tabs that have received notifications so that the tabs won't get suspended again.

Of course, there is still a risk that the first notification may be delayed by 2 mins, but remember that purge + suspend will provide a much better UX than existing tab-discarding (where the tab completely stops running).

[Chris Harrelson]

Hi,

Do you intend to purge and suspend more aggressively than the current tab discarder? Assuming so; if so what are the conditions?

Also, a followup to Ben Maurer's comment: by "notification" do you mean push notification or similar? A long polling system would not have one of those.

Chris

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Douglas Stockwell]

On Tue, Aug 9, 2016 at 3:37 PM Kentaro Hara <har...@chromium.org> wrote:

Hi

Contact emails

har...@chromium.org, ta...@chromium.org

Spec

None.

Summary

When the system is under high memory pressure, purge as much memory as possible from background tabs and suspend them (Design document)

Motivation

Imagine that the OS is under high memory pressure and Chrome is running 1 foreground tab and N background tabs. The most efficient way to reduce the memory would be to purge memory from the N background tabs.

With that motivation in mind, we propose purge + suspend. Specifically, we purge as much memory as possible and then immediately suspend the backgrounded tabs (in order not to let the purged memory come back). This may break web compatibility because the suspended tabs stop running (e.g., favicon update, tab title update, notifications). To mitigate the concern, we resume the suspended tabs periodically (e.g., 2 mins).

 

tasak@ implemented the prototype and collected performance/memory numbers. The high-level summary is as follows:

The link to the performance numbers seems to have expired.

• In average, the purge + suspend reduced 28% of renderer's memory. In particular, it reduced 72% from Facebook and 61% from Tumblur. The number looks very promising.

• In average, the number stays at 28% even after (periodically) resuming the suspended tab. This means that periodic resuming doesn't resurrect much memory in common websites.

In that case why does the resuming need to be periodic?

 

• There are a couple of websites (e.g. Google Drive, Yahoo) where periodic resuming resurrects a lot of memory, but the resurrected memory is gone when the renderer is purged again.

How frequently do these sites actually resume -- do they have timers that execute every period? Sounds like we will end up burning more CPU+battery on such sites.

[Kentaro Hara]

On Wed, Aug 10, 2016 at 9:19 AM, Chris Harrelson <chri...@chromium.org> wrote:

Hi,

Do you intend to purge and suspend more aggressively than the current tab discarder? Assuming so; if so what are the conditions?

Yes. Since tab-suspension can reduce less memory than tab-discarding, we'll need to suspend tabs more aggressively than tab-discarding to get the same memory saving.

Regarding conditions, it's hard to tell at this point and that's why we want to start a Finch experiment to collect more performance/memory numbers from the wild to optimize the heuristics.

In the long term, we're planning to unify all the purging mechanisms into MemoryCoordinator so that we can control all tabs with the following prioritization (with some blacklisting):

1) Throttle memory of background tabs, from the fattest tab to the thinnest tab.

2) Purge and suspend memory of background tabs, from the fattest tab to the thinnest tab.

3) Discard background tabs, from the fattest tab to the thinnest tab.

2) is the purge + suspend we're proposing in this thread. 3) is the existing tab-discarding. We're now implementing 1).

Given that the purge + suspend is already a complicated and impactful optimization, we want to start a Finch experiment separately from the MemoryCoordinator.

 

Also, a followup to Ben Maurer's comment: by "notification" do you mean push notification or similar? A long polling system would not have one of those.

Right. I meant push notifications.

[Kentaro Hara]

On Wed, Aug 10, 2016 at 9:33 AM, Douglas Stockwell <dstoc...@chromium.org> wrote:

On Tue, Aug 9, 2016 at 3:37 PM Kentaro Hara <har...@chromium.org> wrote:

Hi

Contact emails

har...@chromium.org, ta...@chromium.org

Spec

None.

Summary

When the system is under high memory pressure, purge as much memory as possible from background tabs and suspend them (Design document)

Motivation

Imagine that the OS is under high memory pressure and Chrome is running 1 foreground tab and N background tabs. The most efficient way to reduce the memory would be to purge memory from the N background tabs.

With that motivation in mind, we propose purge + suspend. Specifically, we purge as much memory as possible and then immediately suspend the backgrounded tabs (in order not to let the purged memory come back). This may break web compatibility because the suspended tabs stop running (e.g., favicon update, tab title update, notifications). To mitigate the concern, we resume the suspended tabs periodically (e.g., 2 mins).

 

tasak@ implemented the prototype and collected performance/memory numbers. The high-level summary is as follows:

The link to the performance numbers seems to have expired.

Sorry, I think this link will work.

 

• In average, the purge + suspend reduced 28% of renderer's memory. In particular, it reduced 72% from Facebook and 61% from Tumblur. The number looks very promising.

• In average, the number stays at 28% even after (periodically) resuming the suspended tab. This means that periodic resuming doesn't resurrect much memory in common websites.

In that case why does the resuming need to be periodic?

Because as mentioned below, there're websites where the resuming resurrects a non-trivial amount of memory.

Also, in long term, we want to extend the tab-suspension mechanism for battery+CPU saving. In that world, it's important to suspend the tabs (with periodic resuming).

 

 

• There are a couple of websites (e.g. Google Drive, Yahoo) where periodic resuming resurrects a lot of memory, but the resurrected memory is gone when the renderer is purged again.

How frequently do these sites actually resume -- do they have timers that execute every period? Sounds like we will end up burning more CPU+battery on such sites.

My current plan is to resume suspended tabs every 2 mins, but we need some experiment to adjust the interval.

Even if the suspending + resuming wastes some amount of battery+CPU, I believe that it will be a clear win overall. For example, when I look at a tracing of Google Drive, it is doing a ton of work in background tabs. The benefit of suspending those wasteful work would be much larger than the cost of producing some amount of wasteful work at every 2 mins.

I'm planning to (approximately) evaluate the cost by measuring time taken to resume a suspended tab (using UMA).

 

• To evaluate the tab-switching cost, we manually created a custom Chrome that forcibly suspends all background tabs every 1 second and crawled many websites. However, we didn't observe any UX regression.

See this document for more detailed performance/memory analysis.

Interoperability and Compatibility Risk

First of all, we blacklist tabs that are playing video, audio etc using the mechanism of the existing tab-discarding. Second, we give the suspended tabs a chance to wake up periodically so that the tabs can update favicons, tab titles, notifications etc.

Remember that the tab-suspension may break compatibility, but at the very least it will provide a much better UX than the existing tab-discarding in terms of both compatibility and tab-switching cost.

Ongoing technical constraints

Experiments are needed to optimize heuristics to determine the purging targets (work-in-progress CL) and the timings. Our plan is to start a Finch experiment, collect more performance/memory numbers from the wild, and optimize the heuristics.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes. On all platforms that have background tabs.

OWP launch tracking bug

http://crbug.com/635419

Requesting approval to ship?

Yes. As mentioned above, we want to start a Finch experiment to collect more performance/memory numbers and optimize the heuristics.

--

Kentaro Hara, Tokyo, Japan

[Ben Maurer]

I think the issue here is that a tab like this would not actually get push notifications (since they'd be delivered via direct communication with the server). One possible approach would be to disable the discarding for tabs that have actively been making network requests. But that may fail on sites that do various types of background updates. 

[Kentaro Hara]

I think the issue here is that a tab like this would not actually get push notifications (since they'd be delivered via direct communication with the server). One possible approach would be to disable the discarding for tabs that have actively been making network requests. But that may fail on sites that do various types of background updates. 

I agree that's a valid concern, but considering the benefit of saving memory and battery, I think that the tab-suspension would be a clear win overall.

Note that we're already trading off compatibility concerns on background tabs for performance/memory benefits. The existing tab-discarding is one thing. Gmail also very aggressively throttles timers to update the tab title (# of incoming emails on the tab title is not updated for mins).

As mentioned above, there are a couple of things we can do to mitigate the risk; e.g., blacklist tabs that have received push notifications so that the tabs are not suspended again.

[Yoichi Osato]

How about sound streaming? I use YouTube as a jukebox and which is always in background.

Ping sound from a background hanguout page is also very useful and instancy is very important.

2016年8月10日(水) 11:14 Kentaro Hara <har...@chromium.org>:

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Kentaro Hara]

How about sound streaming? I use YouTube as a jukebox and which is always in background.

As mentioned in the first email, we already have a mechanism to blacklist tabs playing video, audio etc. So those tabs won't be suspended or discarded.

Ping sound from a background hanguout page is also very useful and instancy is very important.

 

I guess this would be the same issue as the push notification.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Kentaro Hara]

Just to be clear: The proposal is about enabling the purge + suspend *under high memory pressure*. In long term, we might want to more aggressively suspend background tabs to save more memory and battery, but that's out of the scope of this proposal (I'll make another announcement when we do this :-). Hence, the question is: which is better for users, causing OOM or suspending background tabs in a way that may slightly break compatibility in some limited scenarios?

[Harald Alvestrand]

When you say "existing tab-discarding", is this the sad face saying "something has crashed" that happens whenever I run out of swap space?

Or is it something different?

[Kentaro Hara]

We're already forcibly discarding background tabs (with some blacklisting) when the system is under high memory pressure. You can find a design doc here.

[Elliott Sprehn]

Why does the memory not go back up when we resume? What are we discarding that stays discarded?

[Harald Alvestrand]

hm. the design doc refers to chrome://discards as the place to get info on tab discarding, but I can't find that in any of my (Linux) browsers.

How do I tell that tab discarding is running in my browsers?

(FWIW, I regularly kill off multi-gigabyte tabs - they seem to make Chrome slow, even when there's plenty of RAM on the machine.)

[Kentaro Hara]

Why does the memory not go back up when we resume?

Note that we're just resuming the tab in background and processing pending tasks in the task queue. So CC's memory won't come back.

You might want to open the "Total" column of tasak's result. In average, memory doesn't come back when the tab is resumed, but a non-trivial amount of memory comes back in a couple of websites (e.g., Google Drive, Yahoo) (although the memory is gone when the tab is purged again).

 

What are we discarding that stays discarded?

Hmm, what do you mean?

[Kentaro Hara]

On Wed, Aug 10, 2016 at 4:20 PM, Harald Alvestrand <h...@google.com> wrote:

hm. the design doc refers to chrome://discards as the place to get info on tab discarding, but I can't find that in any of my (Linux) browsers.

How do I tell that tab discarding is running in my browsers?

(FWIW, I regularly kill off multi-gigabyte tabs - they seem to make Chrome slow, even when there's plenty of RAM on the machine.)

+chrisha +georgesak: Do you have any idea on this?

[Elliott Sprehn]

On Wed, Aug 10, 2016 at 4:57 AM, Kentaro Hara <har...@chromium.org> wrote:

Why does the memory not go back up when we resume?

Note that we're just resuming the tab in background and processing pending tasks in the task queue. So CC's memory won't come back.

But we should already discard cc memory in background tabs under memory pressure. What's different about this purge system?

 

You might want to open the "Total" column of tasak's result. In average, memory doesn't come back when the tab is resumed, but a non-trivial amount of memory comes back in a couple of websites (e.g., Google Drive, Yahoo) (although the memory is gone when the tab is purged again).

 

What are we discarding that stays discarded?

Hmm, what do you mean?

You're saying letting the website run script every 2m on average doesn't make it reclaim all of the memory. Some part of the picture seems to be missing though, since we should already discard cc memory in background tabs, is there a cc bug here?

[Chris Hamilton]

On Wed, 10 Aug 2016 at 07:59 Kentaro Hara <har...@chromium.org> wrote:

On Wed, Aug 10, 2016 at 4:20 PM, Harald Alvestrand <h...@google.com> wrote:

hm. the design doc refers to chrome://discards as the place to get info on tab discarding, but I can't find that in any of my (Linux) browsers.

How do I tell that tab discarding is running in my browsers?

(FWIW, I regularly kill off multi-gigabyte tabs - they seem to make Chrome slow, even when there's plenty of RAM on the machine.)

+chrisha +georgesak: Do you have any idea on this?

Tab discarding was never enabled on Linux due to lack of a MemoryPressureMonitor implementation on that platform. A couple of folks have been working on one recently, and when available it will also be used over in MemoryCoordinator land.

[Kentaro Hara]

On Thu, Aug 11, 2016 at 2:44 AM, Elliott Sprehn <esp...@chromium.org> wrote:

On Wed, Aug 10, 2016 at 4:57 AM, Kentaro Hara <har...@chromium.org> wrote:

Why does the memory not go back up when we resume?

Note that we're just resuming the tab in background and processing pending tasks in the task queue. So CC's memory won't come back.

But we should already discard cc memory in background tabs under memory pressure. What's different about this purge system?

Conceptually it won't be really different but the purge + suspend will work much more reliably.

Remember that the purge + suspend is a part of MemoryCoordinator, which is a replacement of the existing memory pressure system. The existing memory pressure system is seriously broken in many ways, so our plan is to fix a bunch of memory problems in MemoryCoordinator instead of investing more on the broken system :)

 

 

You might want to open the "Total" column of tasak's result. In average, memory doesn't come back when the tab is resumed, but a non-trivial amount of memory comes back in a couple of websites (e.g., Google Drive, Yahoo) (although the memory is gone when the tab is purged again).

 

What are we discarding that stays discarded?

Hmm, what do you mean?

You're saying letting the website run script every 2m on average doesn't make it reclaim all of the memory. Some part of the picture seems to be missing though, since we should already discard cc memory in background tabs, is there a cc bug here? 

ericrk@: Do you have any idea on this?

[Kentaro Hara]

Getting back to the original point, are there any concerns to be resolved before starting a Finch experiment of the purge + suspend?

[Chris Harrelson]

On Thu, Aug 11, 2016 at 6:20 AM, Kentaro Hara <har...@chromium.org> wrote:

Getting back to the original point, are there any concerns to be resolved before starting a Finch experiment of the purge + suspend?

Not from me.

[Takashi Sakamoto]

Hi

I would like to report our Purge+suspend finch experiment’s result. The finch experiment started on December, 2016 with the following configuration:

• platform: Linux, Windows, MacOS and ChromeOS

• milestone: 57

• channel: dev and canary

Summary

Since we found that Purge+Suspend can break many web sites, we have decided to go with a simpler approach: Purge+Throttle. The idea is just to periodically send purge notifications to background renderers that have been already throttled by the Blink Scheduler.The new design is https://docs.google.com/document/d/1L5qRmCwjidiZWKmQRiuqMjZivBWV7XSW9RTypg9XxlI/edit?usp=sharing.

We enabled Purge+Throttle in a Finch experiment and got the following results:

• The purge notifications change 50th percentile of total memory usages from 61.0MB(control) to 52.9MB(experiment) (-13.3%). 75th percentile is 125.0(control) and 112.5(experiment). We can expect that the notifications save about 8 - 13MB memory per renderer on the average. This is high impact.

• The purge notifications change 50th percentile of tab switching cost from 19.5m to 21.8ms. This shouldn't be a user-visible regression. Not that the regression happens only on a tab that has been backgrounded for >20 mins.

c.f.

Compare Purge+Throttle memory usage vs Control’s after purging: https://docs.google.com/spreadsheets/d/1VbEI_yQ_0InlWYW-fxxniM4nB3YeAffUHgq-GOaf-lA/pubchart?oid=85121757&format=interactive

Compare Purge+Throttle time-to-first-active-paint vs Control’s:

https://docs.google.com/spreadsheets/d/1VbEI_yQ_0InlWYW-fxxniM4nB3YeAffUHgq-GOaf-lA/pubchart?oid=42443163&format=interactive

c.f. memory growth after purging

https://docs.google.com/spreadsheets/d/1VbEI_yQ_0InlWYW-fxxniM4nB3YeAffUHgq-GOaf-lA/pubchart?oid=1142777682&format=interactive

Purge+Throttle will be eventually integrated into MemoryCoordinator. The current heuristics of Purge+Throttle is very simple, but in the future MemoryCoordinator will monitor backgrounded renderers and will purge the memory caches more smartly.

Best regards,

Takashi Sakamoto

--

[Kentaro Hara]

Any objection to enabling the Purge+Throttle?

This is high impact. 8 - 13 MB of memory reduction per background renderer!

[Chris Harrelson]

LGTM!

[Dimitri Glazkov]

LGTM2

LGTM!

--

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Philip Jägenstedt]

LGTM3, the memory saving suggested by the finch experiment are quite impressive :)

[Wez]

I don't actually see a Finch experiment; we have experiments for MemoryCoordinator and PurgeAndSuspend, but nothing for PurgeAndThrottle, and no recent modifications to the other two?

LGTM2

LGTM!

--

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Philip Jägenstedt]

Uh, sorry, upthread I read "Purge+suspend finch experiment’s result".

Should we expect the memory savings from purge+throttle to be similar? Should there be a reverse finch experiment to verify that in case guesses are wrong?

LGTM2

LGTM!

--

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Wez]

From Takashi's PurgeAndSuspend results email, there was this on PurgeAndThrottle:

"We enabled Purge+Throttle in a Finch experiment and got the following results:

• The purge notifications change 50th percentile of total memory usages from 61.0MB(control) to 52.9MB(experiment) (-13.3%). 75th percentile is 125.0(control) and 112.5(experiment). We can expect that the notifications save about 8 - 13MB memory per renderer on the average. This is high impact.

• The purge notifications change 50th percentile of tab switching cost from 19.5m to 21.8ms. This shouldn't be a user-visible regression. Not that the regression happens only on a tab that has been backgrounded for >20 mins."

So it sounds like an experiment was run, I'm just not sure when, nor on which user population.

LGTM2

LGTM!

--

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Takashi Sakamoto]

Firstly I started PurgeAndSuspend, but I modified the feature while running the experiment.
The finch experiment name is still PurgeAndSuspend.
So I didn't use the old data to report PurgeAndThrottle.

Should we expect the memory savings from purge+throttle to be similar?

Yes. I disabled suspending backgrounded renderers, but I didn't change any logic to invoke Purge.

However, I need to say that I disabled FontCache() invalidation while running the experiment. I found that it always causes full re-layout.

Should there be a reverse finch experiment to verify that in case guesses are wrong?

I think, I would like to do a reverse finch experiment. Because PurgeAndThrottle UMAs (the names are PurgeAndSuspend.*) are useful for A/B testing. So if we don't have any reverse finch experiment, it is very difficult to verify. 
 

[Philip Jägenstedt]

OK, sounds good, let's hope the results are confirmed by the reverse experiment :)

LGTM2

LGTM!

--

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Nat Duca]

Very exciting!

Out of an abundance of caution, let us also make sure non-blink stakeholders in memory are supportive of this launch. How we manage memory in Chrome is something that affects a lot of other product stakeholders that aren't currently following along to blink-dev, and when we change things, understandably, a lot of people can get pretty nervous.

If you would like, @tasak, I would be happy to work with you offline to make sure we build consensus with those stakeholders as well. :)

LGTM2

LGTM!

--

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Joe Medley]

Kentaro,

Can you please add an entry for this to Chrome Status.

Thanks

Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195

If an API's not documented it doesn't exist.

[PhistucK]

No need, not web exposed.

☆PhistucK