Memory leak when calling highly parallel goroutines

[robmusic....@gmail.com]

I've got a nasty memory leak. It appears that Go won't reclaim memory when goroutines are called in a highly parallel fashion. 

To illustrate the issue I've written a simple example below. When I run this on my Windows system, I get 800MB to 1 GB of memory consumption. 

What's fascinating is that if I change the time.Sleep to 100 milliseconds instead of 1, the process never goes above 6 MB of memory. This is what I would expect since there are no objects here that should be retained in memory. And, it doesn't matter how long I wait, the garbage collector never cleans up the mess either.

I've tried to profile it using pprof and it didn't help me. It helped me find and fix other issues, but not this. It's entirely possible I did it wrong though since I am new to using that tool.

Help! Thanks in advance!

package main

import (

  "bytes"

  "fmt"

  "io/ioutil"

  "math/rand"

  "runtime"

  "time"

)

func main() {

  for i := 0; i < 1000; i++ {

    time.Sleep(time.Millisecond * 1)

    go fakeGetAndSaveData()

  }

  runtime.GC()

  time.Sleep(10 * time.Minute)

}

func fakeGetAndSaveData() {

  var buf bytes.Buffer

  for i := 0; i < 40000; i++ {

    buf.WriteString(fmt.Sprintf("the number is %d\n", i))

  }

  ioutil.WriteFile(fmt.Sprintf("%d.txt", rand.Int()), buf.Bytes(), 0644)

}

Thanks,

Rob Archibald

[Ian Lance Taylor]

On Wed, Oct 25, 2017 at 12:06 AM, <robmusic....@gmail.com> wrote:
>
> I've got a nasty memory leak. It appears that Go won't reclaim memory when
> goroutines are called in a highly parallel fashion.
>
> To illustrate the issue I've written a simple example below. When I run this
> on my Windows system, I get 800MB to 1 GB of memory consumption.

How are you measuring memory consumption?

A memory leak implies that memory continues growing without bound.
That is not what you seem to be reporting. What you are reporting is
high memory usage in a steady state. It's certainly true that if you
create a large number of goroutines in parallel your program will need
a bunch of memory to support those goroutines. And once a program
requires a large amount of memory, it will hold on to that memory for
a while, on the assumption that it is likely to happen again. Over
time unneeded memory will be released back to the system, where "over
time" means several minutes.

That said, there is a current issue: the G structure that represents a
goroutine is never released. So if your program starts a large number
of goroutines at one time, and then never does that again, you will be
left with some memory that is never released. This is
https://golang.org/issue/9869.

Ian

[Rob Archibald]

Thanks Ian,

It is a memory leak because the memory continues to grow for as long as the loop continues to run. If you change the code example I gave to an infinite loop you'll see that it grows infinitely. I had it stop at 1000 for demonstration purposes because each scan request that my production app gets typically kicks off about 1000 goroutines. 

I posted to the issue you linked to as well. Thanks for letting me know about that. 

I'm measuring memory from the system perspective. Windows and Linux both report gigs of memory used on my production app before it crashes due to out of memory. This is a show-stopper issue for me. If Go can't release memory from a simple goroutine when it exits, I'll have to rewrite using something else. 

Blessings,

Rob Archibald

CTO, EndFirst LLC

r...@robarchibald.com

[Ian Lance Taylor]

On Wed, Oct 25, 2017 at 10:21 AM, Rob Archibald <r...@robarchibald.com> wrote:
>
> It is a memory leak because the memory continues to grow for as long as the
> loop continues to run. If you change the code example I gave to an infinite
> loop you'll see that it grows infinitely. I had it stop at 1000 for
> demonstration purposes because each scan request that my production app gets
> typically kicks off about 1000 goroutines.
>
> I posted to the issue you linked to as well. Thanks for letting me know
> about that.
>
> I'm measuring memory from the system perspective. Windows and Linux both
> report gigs of memory used on my production app before it crashes due to out
> of memory. This is a show-stopper issue for me. If Go can't release memory
> from a simple goroutine when it exits, I'll have to rewrite using something
> else.

Sorry, I didn't mean to suggest that you should add your program to
that issue. I only meant to point to that issue as describing a known
problem with programs that start a very large number of goroutines and
then go back to just using a small number of goroutines. If your
program continues to use an increasing amount of memory over time,
then it is a different problem.

When you say "measuring memory from the system perspective," what
precisely do you mean? It really matters, as different system
measurements report different things, and none of them really
correspond to how Go manages its heap. Or, to put it another way,
what is the real problem?

I tried running your program. For me the virtual memory size as
measured by the ps program goes up to 42176 (42M) and then stops.
This is on GNU/Linux, though, not Windows.

You say that you are describing a memory leak because memory continues
to grow for as long as the loop continues to run. Since the loop is
starting new goroutines more quickly than they can finish, that is not
surprising. If you want to control memory usage with a loop like
that, you need to limit the number of goroutines you start in
parallel.

Ian

[Rob Archibald]

Thanks for your help on this! I really appreciate it. I've been pulling my hair out for weeks on this issue, so I truly appreciate any help you can offer.

The way this runs in my production app is similar to how it works in the example app I provided. A request is sent to a web server and then up to 1000+ http requests are made. Data is pulled, parsed, and ultimately added to a database. Each of those requests are done in goroutines and each request finishes. But, the memory is never cleaned up. That's why I created this the way that I did. I'm aware that if the goroutines never finish or if I create them forever, it'll grow infinitely, but that's not what I'm doing. Requests come in and are processed and completed, but they're never cleaned up.

As I just mentioned in the Github issue you referenced, it looks like this example isn't actually breaking in Linux after all so I'll put together one that does break on Linux too.

Blessings,
Rob Archibald
CTO, EndFirst LLC
r...@robarchibald.com

[Tamás Gulácsi]

Just blind shots:
resp.Body.Close(),
sync.Pool the buffers,
Limit concurrency - the simplest is a make(chan struct{}, 512) ans push/pull empry tokens to it.

[Michael Jones]

Rob, perhaps it is not clear to you that what you describe is not a memory leak. Imagine this:

Rob to car wash robot: "Robot, Wash each car in the parking lot, one after another"

Robot #1: OK

Rob to parking lot robot: "Robot, Park a million cars in the parking lot"

Robot #2: OK.

Rob: "OMG! We have a 1000 acre parking lot! We have a parking lot leak!

Robot #2: We needed a bigger parking lot.

:-)

When you make stuff faster than it gets consumed, you're forcing the allocation of more parking lot.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael T. Jones
michae...@gmail.com

[Nathan Kerr]

Read Ways to limit concurrent resource use for different methods that can be used to limit concurrency along with example code.