min_filelist_kbytes too large on 4GB 64-bit Intel systems?

[Bill Wood]

I saw here that min_filelist_kbytes was too small for ARC++ systems at 50000 KiB, and that 200000 KiB was considered enough.

Then I saw here that min_filelist_kbytes was doubled due to new kernel handling of this value:

The full amount was mistakenly reserved on a per-zone basis.
A kernel change is correcting this by spreading the amount
across all zones.  Since both x86 and ARM have two large zones,
we need to double the old figure to preserve the existing behavior.

On my Samsung Chromebook Pro, min_filelist_kbytes is now 400000 KiB, presumably it was doubled from 200000 KiB due to the above.

However, I see here that there is only one large zone on a 4GB 64-bit Intel machine.  So why was min_filelist_kbytes doubled?  Should it be 200000 KiB?

Hope this makes sense!  Thanks.

[Luigi Semenzato]

I haven't looked at that blog in detail, but there are two large
zones on those machines, not one. You can check by going to dev mode,
then cat /proc/zoneinfo. Shortly after boot:

Node 0, zone DMA32
pages free 295832
...
Node 0, zone Normal
pages free 296500

There's also a tiny DMA zone.

> --
> --
> Chromium OS Developers mailing list: chromiu...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en
>

[Bill Wood]

Thanks!  I can't find how the size of these zones are set, all the online docs seem to say that DMA32 is 4 GB.  Perhaps it is only 2 GB on these machines?

Perhaps you could answer another question I have... why was it decided to set extra_free_kbytes to 350?  That seems to have the effect of leaving about 500 to 550 MB of memory free, which seems like a waste on a 4 GB system and might reduce battery life if kswapd is often running?

[Luigi Semenzato]

On Tue, Aug 8, 2017 at 7:35 PM, Bill Wood <wpwo...@gmail.com> wrote:
> Thanks! I can't find how the size of these zones are set, all the online
> docs seem to say that DMA32 is 4 GB. Perhaps it is only 2 GB on these
> machines?

I'll take a look, but I've never worried about it. Even if we're
using the wrong size for DMA32, it will only impact fragmentation
insignificantly.

> Perhaps you could answer another question I have... why was it decided to
> set extra_free_kbytes to 350? That seems to have the effect of leaving
> about 500 to 550 MB of memory free, which seems like a waste on a 4 GB
> system and might reduce battery life if kswapd is often running?

Yes that's all correct, however in practice this has mostly the effect
of triggering tab discards earlier, so it slightly increases the
number of tab reloads (very little on average---we can monitor average
usage in the field via the stats reporting). In extreme cases it may
impact someone's workflow, if such workflow requires switching
cyclically through a set of tabs whose total footprint is too large to
fit, but we don't see that very often, not even inside Google, where
as you can imagine we have the most demanding users. (It's never
happened to me.)

Conversely, without that margin we run more often into situations in
which the memory allocation rate is higher than the freeing rate via
tab discards. When that happens the kernel starts OOM-killing
processes. This results in suboptimal loss of tabs (especially when
many tabs share the same "renderer" process), but, worse, the OOM-kill
code has a lot of lock contention and deadlocks, which cause temporary
or permanent freezes. This produces janky behavior, with freezes up
to seconds, and kernel crashes when the hang detector kicks in. This
is much worse than the extra discards, so it justifies the apparent
waste of RAM.

The OOM kill code is hard to fix. A good starting point for reading
about it is this LWN article, there are several more good ones on LWN.

https://lwn.net/Articles/668126/

[Bill Wood]

Thanks for linking me to a good read!

> Yes that's all correct, however in practice this has mostly the effect 
> of triggering tab discards earlier,

It is interesting that you say that, because my tests have a different result, perhaps due to how I am testing.  I start Chrome with 24 windows and 117 tabs.  After things settle down (2 minutes), and with extra_free_kbytes set to 350, I usually have no tab discards (chrome://discards/).  With extra_free_kbytes set to 0, usually every tab that is not visible is discarded.  What seems to be happening is that with extra_free_kbytes at 350, kswapd has more time while pages are loading to move stuff to swap and avoid tab discards.   It seems to be a trade-off between triggering fewer discards, and leaving memory unused except during periods of high allocation demands.

> the OOM-kill 
> code has a lot of lock contention and deadlocks, which cause temporary 
> or permanent freezes.  This produces janky behavior, with freezes up 
> to seconds, and kernel crashes when the hang detector kicks in.

This was happening to me when I first got the Pro with Chrome 58 (it was easy to trigger with lots of windows open, then hit the "switch windows" key).  After Chrome 59 came out, it stopped happening.  Now I can't get it to happen, even with extra_free_kbytes set to 0.  I have been trying to understand what else changed that made this behavior less likely?

[Luigi Semenzato]

On Thu, Aug 10, 2017 at 2:43 PM, Bill Wood <wpwo...@gmail.com> wrote:
> Thanks for linking me to a good read!
>
>> Yes that's all correct, however in practice this has mostly the effect
>> of triggering tab discards earlier,
>
> It is interesting that you say that, because my tests have a different
> result, perhaps due to how I am testing. I start Chrome with 24 windows and
> 117 tabs. After things settle down (2 minutes), and with extra_free_kbytes
> set to 350, I usually have no tab discards (chrome://discards/). With
> extra_free_kbytes set to 0, usually every tab that is not visible is
> discarded. What seems to be happening is that with extra_free_kbytes at
> 350, kswapd has more time while pages are loading to move stuff to swap and
> avoid tab discards. It seems to be a trade-off between triggering fewer
> discards, and leaving memory unused except during periods of high allocation
> demands.

Exactly :)

I was going to add more but my cube-mate is working on it as well :) See ya.

[Sonny Rao]

On Thu, Aug 10, 2017 at 2:43 PM, Bill Wood <wpwo...@gmail.com> wrote:
> Thanks for linking me to a good read!
>
>> Yes that's all correct, however in practice this has mostly the effect
>> of triggering tab discards earlier,
>
> It is interesting that you say that, because my tests have a different
> result, perhaps due to how I am testing. I start Chrome with 24 windows and
> 117 tabs. After things settle down (2 minutes), and with extra_free_kbytes
> set to 350, I usually have no tab discards (chrome://discards/). With
> extra_free_kbytes set to 0, usually every tab that is not visible is
> discarded. What seems to be happening is that with extra_free_kbytes at
> 350, kswapd has more time while pages are loading to move stuff to swap and
> avoid tab discards. It seems to be a trade-off between triggering fewer
> discards, and leaving memory unused except during periods of high allocation
> demands.

Wow, 24 windows and 117 tabs? Is that your normal workload or
something extreme to test memory?

You are correct that there is a trade-off between keeping some memory
reserved and hitting out of memory situations.
In order to give swap time to work and give the tab discarder time to
work, we raised point at which swapping begins by using
extra_free_kbytes.

>
>> the OOM-kill
>> code has a lot of lock contention and deadlocks, which cause temporary
>> or permanent freezes. This produces janky behavior, with freezes up
>> to seconds, and kernel crashes when the hang detector kicks in.
>
> This was happening to me when I first got the Pro with Chrome 58 (it was
> easy to trigger with lots of windows open, then hit the "switch windows"
> key). After Chrome 59 came out, it stopped happening. Now I can't get it
> to happen, even with extra_free_kbytes set to 0. I have been trying to
> understand what else changed that made this behavior less likely?

Chrome 59 (and later) have several changes to make memory management
and discarding work better and reduce the chances of OOM-kill.

> You received this message because you are subscribed to the Google Groups
> "Chromium OS dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to chromium-os-d...@chromium.org.

[Bill Wood]

> Wow, 24 windows and 117 tabs?  Is that your normal workload or 
> something extreme to test memory? 

Just for testing, although I do consider myself to be a "tab hoarder" ;)

15 windows and 50 tabs are normal for me.

> Chrome 59 (and later) have several changes to make memory management 
> and discarding work better and reduce the chances of OOM-kill. 

Where can I see the changes?  I'm not sure what to search for.  Thanks!

[Bill Wood]

Hi Sonny,

> Chrome 59 (and later) have several changes to make memory management 
> and discarding work better and reduce the chances of OOM-kill.

Is this the issue you were referring to?

Thanks,

- Bill

On Thursday, August 10, 2017 at 6:01:00 PM UTC-4, Sonny Rao wrote:

[Luigi Semenzato]

On Mon, Aug 14, 2017 at 1:43 PM, Bill Wood <wpwo...@gmail.com> wrote:
> Hi Sonny,
>
>> Chrome 59 (and later) have several changes to make memory management
>> and discarding work better and reduce the chances of OOM-kill.
> Is this the issue you were referring to?

Yes, that's one, but there are others. It shouldn't be too difficult
to search crbug.com and find more.

Thank you for your interest and help with testing!