OOM handling

[Amruth Raj Padmanabhuni]

Hi,
In an OOM situation, renderer processes in Chromium are terminated based on oom_adj/oom_score_adj values.
Some preventive measures can be taken before reaching this state, for example, by using MemoryPurger::Purge() to free some memory, showing a dialog to the user to close a tab etc. But these have to be done at a state which is ahead of an out-of-memory situation.

To do this, I have the following approach in mind:
- Browser process forks a dummy Chrome process(which does nothing but stay idle) with --type=oom. This will have a high oom_adj score(say x) and lowmemorykiller's minfree parameter can be configured with a value little higher(say y) for a score of x. y bytes is the situation when the preventive measures can be taken.
- When system's free memory reaches y bytes, dummy process will get killed first and Browser's SIGCHLD handler can invoke the preventive steps.
- The dummy process can be re-spawned after it is killed.

I can put up a patch if this approach is agreeable to the community.

Regards,
Amruth Raj.

[Adam Langley]

On Wed, Sep 7, 2011 at 7:11 AM, Amruth Raj Padmanabhuni
<ckq...@motorola.com> wrote:
> Hi,
> In an OOM situation, renderer processes in Chromium are terminated based on
> oom_adj/oom_score_adj values.

Your CC list suggests that this is intended for ChromeOS. In that
case, you can patch the kernel to do this in a cleaner fashion:
http://lwn.net/Articles/267013/

Cheers

AGL

[Evan Martin]

To elaborate, for non-ChromeOS Chrome I believe we don't have the
privileges to adjust our oom scores anyway.

See e.g. http://codesearch.google.com/codesearch#OAMlx_jo-ck/src/chrome/browser/oom_priority_manager.cc&type=cs&l=22

[Luigi Semenzato]

On Wed, Sep 7, 2011 at 4:11 AM, Amruth Raj Padmanabhuni
<ckq...@motorola.com> wrote:
>
> Hi,
> In an OOM situation, renderer processes in Chromium are terminated based on oom_adj/oom_score_adj values.
> Some preventive measures can be taken before reaching this state, for example, by using MemoryPurger::Purge() to free some memory, showing a dialog to the user to close a tab etc. But these have to be done at a state which is ahead of an out-of-memory situation.

The current strategy is to keep users out of memory management, by
killing renderers and reloading tabs quasi-transparently.

>
> To do this, I have the following approach in mind:
> - Browser process forks a dummy Chrome process(which does nothing but stay idle) with --type=oom. This will have a high oom_adj score(say x) and lowmemorykiller's minfree parameter can be configured with a value little higher(say y) for a score of x. y bytes is the situation when the preventive measures can be taken.
> - When system's free memory reaches y bytes, dummy process will get killed first and Browser's SIGCHLD handler can invoke the preventive steps.
> - The dummy process can be re-spawned after it is killed.

It's a clever idea, but there are probably other ways to detect
almost-out-of-memory situations without getting the OOM killer
involved, for instance by poling /proc/meminfo. If there is no good
way, we should change the kernel so it provides one.

>
> I can put up a patch if this approach is agreeable to the community.
>
> Regards,
> Amruth Raj.
>

> --
> Chromium OS Developers mailing list: chromiu...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

[Amruth Raj Padmanabhuni]

I intend to do this for Chrome OS. If mem_notify is an acceptable approach, I can try doing the Chrome side of changes by polling /dev/mem_notify etc.

[Luigi Semenzato]

On Wed, Sep 7, 2011 at 8:48 AM, Amruth Raj Padmanabhuni
<ckq...@motorola.com> wrote:
> I intend to do this for Chrome OS. If mem_notify is an acceptable approach,
> I can try doing the Chrome side of changes by polling /dev/mem_notify etc.

Sorry, I thought you were referring to Chrome OS only. For the other
systems, this approach is confusing. You open a large image in
Photoshop, and Chrome wakes up and suggests you kill one of its tabs?

> On Wed, Sep 7, 2011 at 9:01 PM, Evan Martin <ev...@chromium.org> wrote:
>>
>> On Wed, Sep 7, 2011 at 5:07 AM, Adam Langley <a...@chromium.org> wrote:
>> > On Wed, Sep 7, 2011 at 7:11 AM, Amruth Raj Padmanabhuni
>> > <ckq...@motorola.com> wrote:
>> >> Hi,
>> >> In an OOM situation, renderer processes in Chromium are terminated
>> >> based on
>> >> oom_adj/oom_score_adj values.
>> >
>> > Your CC list suggests that this is intended for ChromeOS. In that
>> > case, you can patch the kernel to do this in a cleaner fashion:
>> > http://lwn.net/Articles/267013/
>>
>> To elaborate, for non-ChromeOS Chrome I believe we don't have the
>> privileges to adjust our oom scores anyway.
>>
>> See e.g.
>> http://codesearch.google.com/codesearch#OAMlx_jo-ck/src/chrome/browser/oom_priority_manager.cc&type=cs&l=22
>

[Amruth Raj Padmanabhuni]

On Wed, Sep 7, 2011 at 9:10 PM, Luigi Semenzato <seme...@chromium.org> wrote:

On Wed, Sep 7, 2011 at 4:11 AM, Amruth Raj Padmanabhuni
<ckq...@motorola.com> wrote:
>

> Hi,
> In an OOM situation, renderer processes in Chromium are terminated based on oom_adj/oom_score_adj values.
> Some preventive measures can be taken before reaching this state, for example, by using MemoryPurger::Purge() to free some memory, showing a dialog to the user to close a tab etc. But these have to be done at a state which is ahead of an out-of-memory situation.

The current strategy is to keep users out of memory management, by
killing renderers and reloading tabs quasi-transparently. 

>
> To do this, I have the following approach in mind:
> - Browser process forks a dummy Chrome process(which does nothing but stay idle) with --type=oom. This will have a high oom_adj score(say x) and lowmemorykiller's minfree parameter can be configured with a value little higher(say y) for a score of x. y bytes is the situation when the preventive measures can be taken.
> - When system's free memory reaches y bytes, dummy process will get killed first and Browser's SIGCHLD handler can invoke the preventive steps.
> - The dummy process can be re-spawned after it is killed.

It's a clever idea, but there are probably other ways to detect
almost-out-of-memory situations without getting the OOM killer
involved, for instance by poling /proc/meminfo.  If there is no good
way, we should change the kernel so it provides one.

The available memory is not very easily available from meminfo. It is something like memfree + buffered + cached - shared + ...
I feel parsing this file periodically to get this number is going to be complex than to get it indirectly through OOM killer. mem_notify is also a good approach.

[James Cook]

We actually discussed this problem at the Chrome OS eng review yesterday.  This document describes the approach we're taking for OOM on Chrome OS:

https://docs.google.com/a/google.com/document/pub?id=1VrG24w-NDoW8SgNIMvVZNOczjkE9Y__NvNHPcZ1NNEY

ups@ is on-board to get the kernel to send Chrome a notification when Chrome OS is low on memory, and gspencer@ and I are planning to do work on discarding the "most boring" tab.  We looked at memory purging too -- we'll probably end up triggering it, but it doesn't usually get us enough RAM back to avoid having to close a tab.

Please have a look at the doc -- if it seems like a good approach to you, we would love some help!

James

[James Cook]

I moved the design document to chromium.org:

https://sites.google.com/a/chromium.org/dev/chromium-os/chromiumos-design-docs/out-of-memory-handling

Sorry this wasn't up before.

James

[Lei Zhang]

I thought we did in ZygoteHost::AdjustRendererOOMScore. However, my
sandboxed renderers all have oom_adj = 0, so I wonder if that code is
broken.

> --
> Chromium Developers mailing list: chromi...@chromium.org

> View archives, change email options, or unsubscribe:

>    http://groups.google.com/a/chromium.org/group/chromium-dev
>

[James Cook]

I thought we did in ZygoteHost::AdjustRendererOOMScore. However, my
sandboxed renderers all have oom_adj = 0, so I wonder if that code is
broken.

The AdjustRendererOOMScore code works on Chrome OS (or at least, worked last week when I played with it).  If it's broken on Chrome OS please file a bug in crosbug.com for me and I'll either fix it or find someone who can.

I'm not sure if AdjustRendererOOMScore can/should work on other platforms.

James

[Lei Zhang]

Ah, the problem is on Ubuntu, I have a /selinux directory. Thus
Chromium thinks SELinux might be active and avoids adjusting the OOM
score.

On a related note, I see the OOM score range is [0,1000] instead of
[0,15]. Are we using the CrOS OOM score range on Linux as well, or did
Linux change its OOM score range?

[Greg Spencer]

On Wed, Sep 7, 2011 at 10:31 PM, Lei Zhang <the...@chromium.org> wrote:

On a related note, I see the OOM score range is [0,1000] instead of
[0,15]. Are we using the CrOS OOM score range on Linux as well, or did
Linux change its OOM score range?

Yes, we switched from using oom_adj (which is now deprecated, and will be removed soon) to using oom_score_adj, which has a range from -1000 to 1000 instead of -17 to 15.

On systems that don't yet support oom_score_adj, we scale the values into the old range and use oom_adj.

-Greg.

[Greg Spencer]

On Wed, Sep 7, 2011 at 10:31 PM, Lei Zhang <the...@chromium.org> wrote:

On a related note, I see the OOM score range is [0,1000] instead of
[0,15]. Are we using the CrOS OOM score range on Linux as well, or did
Linux change its OOM score range?

[Mandeep Singh Baines]

The code to allow Chrome to adjust its oom score went upstream (2.6.38):

https://github.com/torvalds/linux/commit/dabb16f639820267b3850d804571c70bd93d4e07

Should be possible to modify Chrome to test whether the feature works and then
to use it if it does.

> --
> Chromium OS Developers mailing list: chromiu...@chromium.org

> View archives, change email options, or unsubscribe:

> http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

[James Ketrenos]

On Sep 7 2011, 8:40 am, Luigi Semenzato <semenz...@chromium.org>
wrote:

> On Wed, Sep 7, 2011 at 4:11 AM, Amruth Raj Padmanabhuni
>
> <ckq...@motorola.com> wrote:
>
> > Hi,

> > In anOOMsituation, renderer processes in Chromium are terminated based on oom_adj/oom_score_adj values.

> > Some preventive measures can be taken before reaching this state, for example, by using MemoryPurger::Purge() to free some memory, showing a dialog to the user to close a tab etc. But these have to be done at a state which is ahead of an out-of-memory situation.
>
> The current strategy is to keep users out of memory management, by
> killing renderers and reloading tabs quasi-transparently.

Bringing up an old thread; I've been looking through the various
documents on how OOM is being handled and I haven't found any
documents or extension APIs which allow a web application to hook into
the OOM imposed death sequence, which could then allow the application
to be loaded back to the state the user left it.

Is the right approach for an application to watch for when it is no
longer in focus (via chrome.windows onFocusChanged?) and then persist
its internal state so it can be "safely" killed if needed, and
subsequently resume its state when re-launched?

Short of trying to manage its own state internally, is there a way an
application can know if it is being loaded as part of the quasi-
transparent reload?

Thanks,
James

[James Cook]

Bringing up an old thread; I've been looking through the various

documents on how OOM is being handled and I haven't found any
documents or extension APIs which allow a web application to hook into
the OOM imposed death sequence, which could then allow the application
to be loaded back to the state the user left it.

As far as I know, we don't have any notification that a page/tab is being killed as part of OOM.  We try to free memory as quickly as possible in the tab discarder, so we may never provide such a notification.  (These OOM events are supposed to be rare, and the tab discarder is an attempt to be a little bit better than the old way, letting the kernel OOM killer kill a renderer process.)

 

Is the right approach for an application to watch for when it is no
longer in focus (via chrome.windows onFocusChanged?) and then persist
its internal state so it can be "safely" killed if needed, and
subsequently resume its state when re-launched?

I'm not sure what to advice here. Periodically saving state is probably a good idea.  Doing it on focus loss seems reasonable to me, though I am not a DOM/JS expert.

Short of trying to manage its own state internally, is there a way an
application can know if it is being loaded as part of the quasi-
transparent reload?

 

Right now these reloads go through exactly the same code path as a normal browser reload, so I don't think the page can tell that this reload is special.

These are probably not the answers you were looking for, but hopefully it will save you a bit of time looking for things that aren't there. :-(

James