Go memory usage inside containers
4,624 views
Skip to first unread message
Leigh McCulloch
Sep 12, 2018, 2:48:04 AM9/12/18
to golang-nuts
Hi,
Does anyone here know how Go interacts with memory limits inside containers? (e.g. Kubernetes, Docker)
Does the Go runtime have similar problems as the Java runtime, in that it doesn't know what the container limit is, and only knows what the physical limit is for the entire instance?
Or is Go aware of limits placed on the container?
(I've witnessed the Go runtime package say the NumCPU is the total physical count when the app has been limited to many less CPUs, so it seems like it doesn't have visibility into the CPU limitations.)
Thanks,
Leigh
Henrik Johansson
Sep 12, 2018, 3:01:20 AM9/12/18
to Leigh McCulloch, golang-nuts
Afaik it works fine for Go programs as long as these limits translates to things like taskset etc.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
robert engels
Sep 12, 2018, 8:19:30 AM9/12/18
to Leigh McCulloch, golang-nuts
I don’t think Java needs visibility into the max memory, as you need to set it via args (-Xmx) if you want a max - otherwise it will just keep allocation as it needs it - if an allocation from the OS fails, it will attempt a final GC to see if it can get more room, and if not, OOME.
The Runtime.maxMemory() returns the configured max, not the OS max limit.
The Runtime.maxMemory() returns the configured max, not the OS max limit.
Ian Lance Taylor
Sep 12, 2018, 9:46:34 AM9/12/18
to Leigh McCulloch, golang-nuts
attention to memory limits at all, whether running inside a container
or not. You may be interested in following
https://golang.org/issue/23044 which could be a first step toward
fixing this. In some sense the core issue is that using memory limits
in a garbage collected language is close to useless if the program
doesn't have a way to react when heap size is nearing the memory
limit.
On the other hand, on GNU/Linux systems, the CPU count is determined
by calling the sched_getaffinity system call, which as far as I know
is aware of container limits. What leads you to believe otherwise?
Ian
Leigh McCulloch
Sep 12, 2018, 9:58:22 AM9/12/18
to golang-nuts
Hi Ian,
Thanks for the link and answering. I'll look more into understanding the issue. I'm mostly trying to reconcile that there are so many articles about not using Java inside containers because it does nothing to stay in the memory limits, but then I see no other articles about Go, giving the appearance it plays well with container memory limits. Maybe Go's garbage collector is more aggressive at keeping memory usage down?
In regards to CPU, I was running an app in a Kubernetes cluster on a node that had 16 CPUs. Even if I limited the apps access to 1 CPU, the runtime.NumCPU() call still returned 16.
Leigh
robert engels
Sep 12, 2018, 10:08:01 AM9/12/18
to Ian Lance Taylor, Leigh McCulloch, golang-nuts
With the Azul VM (and I believe the G1 collector in Java), the VM is definitely aware of memory pressure as it approaches the maximum limit - then it will increase the concurrent GC activity trying to avoid a potential huge pause if the limit was reached - so throughput is lowered.
I would think the Go GC needs to do this as well in order to limit pause times.
I would think the Go GC needs to do this as well in order to limit pause times.
robert engels
Sep 12, 2018, 10:09:23 AM9/12/18
to Ian Lance Taylor, Leigh McCulloch, golang-nuts
You would need to refer me to those articles. We used Java in containers all of the time. You need to set a proper -Xmx though.
Ian Lance Taylor
Sep 12, 2018, 10:10:40 AM9/12/18
to Leigh McCulloch, golang-nuts
On Wed, Sep 12, 2018 at 6:58 AM, Leigh McCulloch <leig...@gmail.com> wrote:
>
> Thanks for the link and answering. I'll look more into understanding the
> issue. I'm mostly trying to reconcile that there are so many articles about
> not using Java inside containers because it does nothing to stay in the
> memory limits, but then I see no other articles about Go, giving the
> appearance it plays well with container memory limits. Maybe Go's garbage
> collector is more aggressive at keeping memory usage down?
I wouldn't know.
>
> Thanks for the link and answering. I'll look more into understanding the
> issue. I'm mostly trying to reconcile that there are so many articles about
> not using Java inside containers because it does nothing to stay in the
> memory limits, but then I see no other articles about Go, giving the
> appearance it plays well with container memory limits. Maybe Go's garbage
> collector is more aggressive at keeping memory usage down?
> In regards to CPU, I was running an app in a Kubernetes cluster on a node
> that had 16 CPUs. Even if I limited the apps access to 1 CPU, the
> runtime.NumCPU() call still returned 16.
there is something to be fixed in the Go runtime here.
Ian
Ian Lance Taylor
Sep 12, 2018, 10:14:37 AM9/12/18
to robert engels, Leigh McCulloch, golang-nuts
On Wed, Sep 12, 2018 at 7:07 AM, robert engels <ren...@ix.netcom.com> wrote:
> With the Azul VM (and I believe the G1 collector in Java), the VM is definitely aware of memory pressure as it approaches the maximum limit - then it will increase the concurrent GC activity trying to avoid a potential huge pause if the limit was reached - so throughput is lowered.
>
> I would think the Go GC needs to do this as well in order to limit pause times.
That does not sound precisely correct to me. The Go GC pause time is
> With the Azul VM (and I believe the G1 collector in Java), the VM is definitely aware of memory pressure as it approaches the maximum limit - then it will increase the concurrent GC activity trying to avoid a potential huge pause if the limit was reached - so throughput is lowered.
>
> I would think the Go GC needs to do this as well in order to limit pause times.
independent of the heap size, and I would expect that Java's pause
time is as well. It's certainly true that Go could run the GC harder
when coming closer to a memory limit, which will tend to starve the
program, but it won't actually pause the program. As I mentioned
earlier, though, when the program is coming close to its memory
limits, it needs to react. If it's a network server, it needs to stop
accepting new connections. If it has cached data, it needs to discard
it. And so forth. The GC can try harder but if the program keeps
allocating more memory there is nothing the GC can do to save it.
Ian
Henrik Johansson
Sep 12, 2018, 6:06:45 PM9/12/18
to Ian Lance Taylor, robert engels, Leigh McCulloch, golang-nuts
taskset works for sure and I managed to crash a program with ulimit.
I thought these stuff was what the container runtimes used under the hood.
Am I missing something?
Wojciech S. Czarnecki
Sep 12, 2018, 7:04:10 PM9/12/18
to golan...@googlegroups.com
On Tue, 11 Sep 2018 23:48:03 -0700 (PDT)
Leigh McCulloch <leig...@gmail.com> wrote:
> Hi,
>
> Does anyone here know how Go interacts with memory limits inside
> containers? (e.g. Kubernetes, Docker)
Real box info - parse /proc/meminfo
Leigh McCulloch <leig...@gmail.com> wrote:
> Hi,
>
> Does anyone here know how Go interacts with memory limits inside
> containers? (e.g. Kubernetes, Docker)
Inside Docker: use stats via socket. Mount /var/run/docker.sock into your
container
and push docker stats over it then from Go read live data from it:
docker run -it -v /var/run/docker.sock:/var/run/docker.sock docker stats
See: https://docs.docker.com/config/containers/runmetrics/
IDNK current vmware, but eg Paralels containers can be run with /proc/meminfo
virtualzed. i.e. giving info just about instance.
> Thanks,
> Leigh
>
--
Wojciech S. Czarnecki
<< ^oo^ >> OHIR-RIPE
robert engels
Sep 12, 2018, 7:39:01 PM9/12/18
to Wojciech S. Czarnecki, golan...@googlegroups.com
This may be of interest.
so it doesn’t look like it will happen in base linux, but looks like lxc already does this.
robert engels
Sep 14, 2018, 12:25:40 AM9/14/18
to Ian Lance Taylor, Leigh McCulloch, golang-nuts
I now have the time to give a longer explanation regarding GC, that anyone using Go might find interesting. There are some simplifications in the following, but they’re not significant.
At any given moment, the memory use of a program = live objects + garbage + unused heap.
When a mutator (application thread) wants to allocate a object, it will use the ‘unused heap’. If there is no room in the unused heap, it has two choices: 1) grow the heap (ask the OS for more memory) 2) perform GC to collect garbage, adding the memory to the unused heap, then try the allocation again.
If the runtime cannot allocate more memory from the OS (ulimit, CGroup/container limit, etc.) the only choice is to perform/wait (i.e. pause) until the GC frees enough garbage tp continue - this is usually called a “full GC/pause".If after the garbage collection(s) is performed, there is still not enough room for the object in the unused heap - then you will get an out of memory panic/exception. It is trivial to see that if the live object size reaches the process maximum memory there is no choice but to terminate/panic/OOME.
If there is garbage to be collected, the mutator might pause indefinitely waiting for enough free memory (although in most cases, there is a limit here, after which an OOME will be thrown/panic).
But there is another piece to the puzzle - the one Java was always hammered on in the beginning - incremental GC pauses.
To understand these, the important aspects are the “allocation rate” - the rate at which the mutators threads generate garbage (e.g. MB/sec), and the “collection rate”, the rate at which the GC threads can find and turn the garbage into ‘free memory’ - move the memory on the unused heap.
In a typical GC environment, the GC happens while the mutator threads run. There are two mains types of GC, 1) stop the world, and 2) concurrent.
The “stop the world” collectors pause all mutator threads, use all available processors, and run for a period X to try and free as much garbage as possible. These can be very efficient, but the longer the mutators are paused, the greater latency. So often the goal is to minimize X - but if the allocation rate is exceeding the collection rate at X, then X needs to be increased, OR, if it isn’t then eventually a "full GC/pause” will happen.
For concurrent GC, there is still a pause time Y, but it is not related to how much garbage can be collected, and is usually very short. It is typically only affected by the number of active mutator threads and their stack sizes. The pause is usually very small, only long enough to sample the threads stacks and some other housekeeping, and then the collector threads continue concurrently with the mutator threads. This is where things get interesting. If the too many GC threads run, or run too long, it starves the mutators threads from running (since they share the available processors) - this is not necessarily a pause, it is a scheduling/latency issue imposed by the OS - so typically the mutators threads have priority over the GC threads in low pressure scenarios. After the GC threads complete their cycle, the system has determined the current “collection rate”.
If the system determines the allocation rate is exceeding the collection rate, that doesn’t mean the system is going to run out of memory - it can always pause the mutators longer to collect more garbage (and stop them from generating garbage), or pause them more often (run more GC cycles), or increase the priority of the GC threads. This is why I stated that even though the pause time is not affected by the heap size, the mutators might be paused more frequently, causing latency/throughput issues. The efficiency of the GC algorithm, plus available CPU, determines the collection rate.
This is why having ample head-room (free heap) is very important for performance in GC systems, because often the mutator ‘allocation rate’ is not constant (e.g. heavy Internet traffic during the day, spikes, etc. lower at night), so if it has enough room to avoid more/longer GC cycles by just allocating memory from the heap/OS it will (to avoid pauses, more CPU for mutators), then wait for periods of lower activity to use more CPU for GC to collect the garbage.
*** Btw, technically with the concurrent collector, a pause phase is not required. The Azul C4 collector is pauseless in the design, but currently uses a small pause phase for simplicity of implementation. There is the real possibility in the future, given hardware advances, that all collectors may become pauseless - but that still doesn’t mean there won’t be a pause due to full GC - it’s a throughput & latency issue.
At any given moment, the memory use of a program = live objects + garbage + unused heap.
When a mutator (application thread) wants to allocate a object, it will use the ‘unused heap’. If there is no room in the unused heap, it has two choices: 1) grow the heap (ask the OS for more memory) 2) perform GC to collect garbage, adding the memory to the unused heap, then try the allocation again.
If the runtime cannot allocate more memory from the OS (ulimit, CGroup/container limit, etc.) the only choice is to perform/wait (i.e. pause) until the GC frees enough garbage tp continue - this is usually called a “full GC/pause".If after the garbage collection(s) is performed, there is still not enough room for the object in the unused heap - then you will get an out of memory panic/exception. It is trivial to see that if the live object size reaches the process maximum memory there is no choice but to terminate/panic/OOME.
If there is garbage to be collected, the mutator might pause indefinitely waiting for enough free memory (although in most cases, there is a limit here, after which an OOME will be thrown/panic).
But there is another piece to the puzzle - the one Java was always hammered on in the beginning - incremental GC pauses.
To understand these, the important aspects are the “allocation rate” - the rate at which the mutators threads generate garbage (e.g. MB/sec), and the “collection rate”, the rate at which the GC threads can find and turn the garbage into ‘free memory’ - move the memory on the unused heap.
In a typical GC environment, the GC happens while the mutator threads run. There are two mains types of GC, 1) stop the world, and 2) concurrent.
The “stop the world” collectors pause all mutator threads, use all available processors, and run for a period X to try and free as much garbage as possible. These can be very efficient, but the longer the mutators are paused, the greater latency. So often the goal is to minimize X - but if the allocation rate is exceeding the collection rate at X, then X needs to be increased, OR, if it isn’t then eventually a "full GC/pause” will happen.
For concurrent GC, there is still a pause time Y, but it is not related to how much garbage can be collected, and is usually very short. It is typically only affected by the number of active mutator threads and their stack sizes. The pause is usually very small, only long enough to sample the threads stacks and some other housekeeping, and then the collector threads continue concurrently with the mutator threads. This is where things get interesting. If the too many GC threads run, or run too long, it starves the mutators threads from running (since they share the available processors) - this is not necessarily a pause, it is a scheduling/latency issue imposed by the OS - so typically the mutators threads have priority over the GC threads in low pressure scenarios. After the GC threads complete their cycle, the system has determined the current “collection rate”.
If the system determines the allocation rate is exceeding the collection rate, that doesn’t mean the system is going to run out of memory - it can always pause the mutators longer to collect more garbage (and stop them from generating garbage), or pause them more often (run more GC cycles), or increase the priority of the GC threads. This is why I stated that even though the pause time is not affected by the heap size, the mutators might be paused more frequently, causing latency/throughput issues. The efficiency of the GC algorithm, plus available CPU, determines the collection rate.
This is why having ample head-room (free heap) is very important for performance in GC systems, because often the mutator ‘allocation rate’ is not constant (e.g. heavy Internet traffic during the day, spikes, etc. lower at night), so if it has enough room to avoid more/longer GC cycles by just allocating memory from the heap/OS it will (to avoid pauses, more CPU for mutators), then wait for periods of lower activity to use more CPU for GC to collect the garbage.
*** Btw, technically with the concurrent collector, a pause phase is not required. The Azul C4 collector is pauseless in the design, but currently uses a small pause phase for simplicity of implementation. There is the real possibility in the future, given hardware advances, that all collectors may become pauseless - but that still doesn’t mean there won’t be a pause due to full GC - it’s a throughput & latency issue.
Reply all
Reply to author
Forward
0 new messages