Commit Charge and GC compilers

[Daniel Farina]

Hello,

The basic gripe is that I have uses for Go that are sensitive to the
amount of charged virtual memory in Linux, in which I have overcommit
turned off for the benefit of other programs that carefully deal with
it. This footprint seems to have fluctuated from release to release
depending on things like chosen span lengths and so on.

Previous archives on the list suggest the prevailing opinion is
"people who care about virtual memory footprint basically don't know
what they really want." That is is mostly true for people who turn on
overcommit and only run the Go daemon they wrote as the principal
program on a machine. In particular, overcommit being on is a default
and very few programs on that machine benefit from dealing with
ENOMEM.

My situation is not that: instead, I use Go to implement ancillary
agents where it is important not to use too much memory, and beyond
that it is important to not accrue a large amount of charged virtual
memory lest the primary processes doing "real" work receive a lot of
extra ENOMEMs that they *can* handle. I have been told recent Go
versions (in beta) might exacerbate the amount of charged memory and
that I should post here to indicate my concern.

I realize many aspects of the existing memory management strategy (in
particular, any approach meaningfully related to TCMalloc) leverage
demand paging to a large extent, but I was encouraged to write as to
represent Someone Who Cares About Virtual Memory Footprint.

Thanks for reading in advance.

[Dave Cheney]

Thanks for making your concerns known. FWIW we used to have the bitmap
for the heap as part of the bss segment which was demand paged into
ram as the heap grew, but backed by swap (I guess). This caused
problems with *BSDs that implement ulimit properly (linux doesn't), as
well as OpenVZ style hosts, so it was changed in issue,
https://code.google.com/p/go/issues/detail?id=4447.

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Dmitry Vyukov]

On Tue, Apr 9, 2013 at 12:05 AM, Daniel Farina <dan...@heroku.com> wrote:
> Hello,
>
> The basic gripe is that I have uses for Go that are sensitive to the
> amount of charged virtual memory in Linux, in which I have overcommit
> turned off for the benefit of other programs that carefully deal with
> it. This footprint seems to have fluctuated from release to release
> depending on things like chosen span lengths and so on.
>
> Previous archives on the list suggest the prevailing opinion is
> "people who care about virtual memory footprint basically don't know
> what they really want." That is is mostly true for people who turn on
> overcommit and only run the Go daemon they wrote as the principal
> program on a machine. In particular, overcommit being on is a default
> and very few programs on that machine benefit from dealing with
> ENOMEM.
>
> My situation is not that: instead, I use Go to implement ancillary
> agents where it is important not to use too much memory, and beyond
> that it is important to not accrue a large amount of charged virtual
> memory lest the primary processes doing "real" work receive a lot of
> extra ENOMEMs that they *can* handle.

Sorry, I do not understand why you care about virtual memory and how
it is related to ENOMEM in your case.

[Keith Rarick]

On Wed, Apr 10, 2013 at 1:59 AM, Dmitry Vyukov <dvy...@google.com> wrote:
> Sorry, I do not understand why you care about virtual memory and how
> it is related to ENOMEM in your case.

I'll try to restate as concretely as I can, but I may be misunderstanding
part of our situation. Dan, please correct me if I'm wrong.

Postgres consists of several processes that handle ENOMEM and
continue to operate normally (instead of, say, crashing), and they can
make good use of as much physical memory as is available. At the
same time, we run a few Go processes that require very little actual
memory but take a lot of virtual memory. Since overcommit is off, the
unused virtual memory of the Go processes consumes and wastes
physical memory. We'd prefer to make that physical memory available
to postgres.

[Jan Mercl]

Never touched virtual memory should normally have virtually zero real memory footprint. Or am I mistaken?

-j

[Keith Rarick]

On Wed, Apr 10, 2013 at 2:25 AM, Jan Mercl <0xj...@gmail.com> wrote:
> Never touched virtual memory should normally have virtually zero real memory
> footprint. Or am I mistaken?

In my (quite possibly wrong) understanding of Linux memory management,
this depends on whether overcommit is on. If off (setting "2" for "precise"),
virtual pages always correspond to either physical or swap pages.

[Kamil Kisiel]

Brief description over the overcommit settings can be found here: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/vm/overcommit-accounting

[Dmitry Vyukov]

I see, thanks.
Please copy-paste it into https://code.google.com/p/go/issues/detail?id=5236
And there is a reference to a patch that uses MAP_NORESERVE for heap
allocation, I think it should help in this case as well.

[minux]

Quote https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/vm/overcommit-accounting:
"In mode 2 the MAP_NORESERVE flag is ignored."

so i think MAP_NORESERVE won't help in this case.

[Daniel Farina]

On Tue, Apr 9, 2013 at 10:52 AM, Daniel Farina <dan...@heroku.com> wrote:

> On Tue, Apr 9, 2013 at 10:25 AM, Jan Mercl <0xj...@gmail.com> wrote:
>> Never touched virtual memory should normally have virtually zero real memory
>> footprint. Or am I mistaken?
>

> You are mistaken, but probably only in the context of parsing the very
> fine difference between "commit charge" and "use".
>
> The moment one touches anything in a virtual memory area, it is
> "charged" -- that's not the same as 'used': untouched memory can be
> used as file caching, for instance. However, with overcommit off, all
> that memory is fully reserved under the supposition that it can be
> written at any time, and it's not desirable to have the kernel invoke
> OOM killing on the system...hence, no more writable VMA space than
> backing memory (nominally swap and RAM) will be allowed.

Actually I re-read this and realized I stated a falsehood: it's more
conservative than that. Any time a VMA is marked as writable, the
memory is charged, because it may be written. The time to fail is
when changing the permissions on the memory, such as via mprotect.
mprotect can fail saying "well, that PROT_NONE memory you had actually
doesn't have enough space to be made writable".

But once the pages are writable, they are yours to write, and with
overcommit off they accounted for in such a way there ought to be no
way one can provoke the OOM killer.

[Daniel Farina]

I have gone ahead and done that, along with a back-reference to the
mail archives of this thread.

[Daniel Farina]

On Tue, Apr 9, 2013 at 10:25 AM, Jan Mercl <0xj...@gmail.com> wrote:

> Never touched virtual memory should normally have virtually zero real memory
> footprint. Or am I mistaken?

You are mistaken, but probably only in the context of parsing the very
fine difference between "commit charge" and "use".

The moment one touches anything in a virtual memory area, it is
"charged" -- that's not the same as 'used': untouched memory can be
used as file caching, for instance. However, with overcommit off, all
that memory is fully reserved under the supposition that it can be
written at any time, and it's not desirable to have the kernel invoke
OOM killing on the system...hence, no more writable VMA space than
backing memory (nominally swap and RAM) will be allowed.

The result is other programs ask for VMAs, and cannot get any. As a
small consolation prize, because the amount of memory "used" is small,
Linux will tend to fill up those regions with, say page caches. I
mention this to underscore the difference between what is "used" vs
"charged" for.

[Kyle Lemons]

Can you use cgroups to apply the limits you want to the applications for which you need such limits?  It seems to me that disabling overcommit would be akin to cutting a carrot with a bandsaw.  Sure, it works, but there are much more precise tools available.

https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/Documentation/cgroups/cgroups.txt?id=refs/tags/v3.8.6

[Daniel Farina]

On Wed, Apr 10, 2013 at 9:27 AM, Kyle Lemons <kev...@google.com> wrote:
> Can you use cgroups to apply the limits you want to the applications for
> which you need such limits? It seems to me that disabling overcommit would
> be akin to cutting a carrot with a bandsaw. Sure, it works, but there are
> much more precise tools available.
>
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/Documentation/cgroups/cgroups.txt?id=refs/tags/v3.8.6

Afraid not, I and my colleagues looked into (and experimented with)
many cgroups details very closely. This would be the ideal solution,
because the number of programs that want to deal with ENOMEM is but a
dwindling number.

The reason being at least as of 3.2 (and apparently as of 3.8) as well
memcg does not track virtual memory use in each container, so
overcommit off does something almost like nothing: the moment a
process in the container causes a page fault that would increase the
container's memory usage (not charge) beyond a certain amount, the OOM
killer is invoked. The only time overcommit off has the desired
effect is if the whole system was going to run out of available commit
charge anyway (i.e. memcg works on entirely orthogonal grounds).

I haven't quite poked the linux-mm list just yet, but talking to a few
people familiar with the matter suggest that adding virtual memory
tracking to memcg is not going to be easy to finesse on LKML. I do
have a patch with at the very least one or two questionable choices in
it that seems very roughly to perform the desired function (but it's
not really production vetted, and memcg churns so fast I'm not keen on
maintaining it out of tree), so I think that this is mechanically
doable. Nevertheless, this is definitely not the here and now to be
sure.

[Kyle Lemons]

On Wed, Apr 10, 2013 at 12:19 PM, Daniel Farina <dan...@heroku.com> wrote:

On Wed, Apr 10, 2013 at 9:27 AM, Kyle Lemons <kev...@google.com> wrote:
> Can you use cgroups to apply the limits you want to the applications for
> which you need such limits?  It seems to me that disabling overcommit would
> be akin to cutting a carrot with a bandsaw.  Sure, it works, but there are
> much more precise tools available.
>
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/Documentation/cgroups/cgroups.txt?id=refs/tags/v3.8.6

Afraid not, I and my colleagues looked into (and experimented with)
many cgroups details very closely.  This would be the ideal solution,
because the number of programs that want to deal with ENOMEM is but a
dwindling number.

The reason being at least as of 3.2 (and apparently as of 3.8) as well
memcg does not track virtual memory use in each container, so
overcommit off does something almost like nothing: the moment a
process in the container causes a page fault that would increase the
container's memory usage (not charge) beyond a certain amount, the OOM
killer is invoked.  The only time overcommit off has the desired
effect is if the whole system was going to run out of available commit
charge anyway (i.e. memcg works on entirely orthogonal grounds).

I haven't quite poked the linux-mm list just yet, but talking to a few
people familiar with the matter suggest that adding virtual memory
tracking to memcg is not going to be easy to finesse on LKML.

 

Having not hacked on this stuff in earnest, this may not be the ideal solution, but the most logical thing to me would be to provide an additional memory.oom_control setting that, instead of hanging when more resources are requested or engaging the OOM-killer, would simply return ENOMEM in the affected process.  This still has little to do with virtual memory, and thus you could still have a 512MB-limited app running with a multi-terabyte virtual memory arena.

[Daniel Farina]

The problem with this approach is that the memory has already been
ostensibly made available to the process: there is no call from which
to return ENOMEM from. A mundane assignment that results in a page
fault that requires new, real memory be made available to satisfy,
say, a 'mov' instruction. Such an instruction alone can trip all the
limiting machinery.

The process runs afoul the memcg memory limit by writing to memory.
Once too much memory use is incurred it is lights out (or, a stall
in newer versions of memcg if configured just right). The stalling
option is helpful for writing your own OOM-killer if you are
unsatisfied with Linux's, but not very helpful to programs that can
deal with ENOMEM: presumably, they want to do some clean-up that does
not require additional allocations so that other actors can continue
to function normally (e.g. Postgres: releasing locks).

So, regrettably, memcg is not a very viable looking workaround as it
exists today, although I think with a VM-constrained mode it'd be
pretty close to ideal: systems designed to cope with full reservation
of VMA can live in that world, and the rest of the world that can't
justify caring about that can exist under overcommit regimes, all on
the same running kernel.

[Kyle Lemons]

Oh, indeed.

 

The process runs afoul the memcg memory limit by writing to memory.
Once too much memory use is incurred it is lights out (or, a stall
in newer versions of memcg if configured just right).  The stalling
option is helpful for writing your own OOM-killer if you are
unsatisfied with Linux's, but not very helpful to programs that can
deal with ENOMEM: presumably, they want to do some clean-up that does
not require additional allocations so that other actors can continue
to function normally (e.g. Postgres: releasing locks).

So, regrettably, memcg is not a very viable looking workaround as it
exists today, although I think with a VM-constrained mode it'd be
pretty close to ideal: systems designed to cope with full reservation
of VMA can live in that world, and the rest of the world that can't
justify caring about that can exist under overcommit regimes, all on
the same running kernel.

Yeah, that does sound like the ideal solution.  Shame it isn't implemented yet :(.