compress/zlib large memory usage

[Jamie Hall]

Hi all,

Although I've taken a break from it recently, I've made a SPDY package for Go, and it generally works fine, but I've been having some memory use issues.

The package uses compress/zlib for both a reader and a writer, created as follows:

  c, err := zlib.NewReaderDict(buf, spdyDict)

  c, err := zlib.NewWriterLevelDict(buf, zlib.BestCompression, spdyDict)

The compression works fine, but the program's memory usage grows quite substantially and doesn't seem to drop. In a server using SPDY, the RAM usage

seems to grow to about 50 MB during the first five or so request cycles, and then plateaus. Since it doesn't grow indefinitely, it doesn't appear to be a leak,

but the memory isn't released during long periods of activity, which I would've expected. Nothing of the compression state is stored between connections,

so once a connection is ended, its memory (including the compression state) should be released.

Some profiling suggests that the vast bulk of the memory use comes from (IIRC) compress/flate.compressor.initDeflate().

Is there a known issue that's causing this, or am I doing something wrong? Or is this just a case of unavoidable overhead from zlib.BestCompression?

Many thanks,

Jamie

[David DENG]

I'm also suffering from the memory problem, too. Manually calling runtim.GC() helps a little. Log out the memory state with runtime.ReadMemStats.

David

[clba...@gmail.com]

Might want to use debug.SetGCPercent() to increase frequency and tighten up GC. 

[Carlos Castillo]

Are you closing the zlib objects when you are done?

Also please note the following:

1. From an external point of view Garbage Collectors (in general) rarely "Free" memory, they only allow the program to re-use it.
2. The GC in Go is still somewhat conservative (although less so w/ every version), meaning there are some situations where a non-pointer points to memory, erroneously making it still "Live" and thus not collected. As the collector becomes more precise, the chances of this happening get smaller and smaller. It is also far less likely (4 billion or so times less) for this to happen if you are compiling/running 64-bit apps.
3. In Go, there is a scavenger  run every few minutes that sees if Chunks of memory allocated by the program (memory is allocated in large chunks, say 32MB, or some other power of 2 size), are completely empty and marks them as Unused.
4. "Unused", means that the memory still belongs to go, but if the OS needs memory for another task, and no free memory is available, then the unused memory is claimed first, at virtually no extra penalty (similar to using free memory). This has the side effect of making the OS stats (where most OS's only consider free/not free) look bad, even though the system can operate identically to if the memory was "freed".
5. Unused memory isn't implemented in Windows (https://code.google.com/p/go/issues/detail?id=5584), which results in the OS being unable to tell which Go memory can be ignored, so it must page all memory it takes from the Go program to the disk.

To see a more detailed and accurate view of how go uses memory, you can use the runtime.ReadMemStats function to see memory stats at a point in time. You can also see the GC and scavenger in action by setting the environment variable GOGCTRACE=1. You can manually run the garbage collector with http://golang.org/pkg/runtime/#GC, and can run the scavenger with http://golang.org/pkg/runtime/debug/#FreeOSMemory. You can set the the GC percent low (try 0) using http://golang.org/pkg/runtime/debug/#SetGCPercent, or by setting the GOGC environment variable. By setting it to 0, you essentially run the GC on every allocation, minimizing the memory your app uses (often at a significant performance cost). The default value of GCPercent (100) means that the garbage collector is run if twice as much memory is in use since the last time the GC finished, this means that it is often expected that the amount of memory your program will use (according to the OS) is double it's peak memory use.

The advantage of using the environment variables (GOGC and GOGCTRACE), is that you don't need to modify your code or rebuild your app to try them out. 

[Jamie Hall]

Thanks for all the info, Carlos, I think that solves my issue.

To answer some of the stuff you raised specifically, I am closing the zlib objects, and I did try runtime/debug.FreeOSMemory to no effect. I suspect that as you say, some stuff isn't being completely freed for performance reasons, which may explain why the memory usage plateaus.

My main issue was that I wasn't sure why the memory was growing and I didn't trust it, but it does stop growing and I suspect the memory's being marked unused as you suggested. As a result, I'm happy.

Thanks for your help

[Kamil Kisiel]

I've had exactly the same experience, but fortunately the memory usage does stabilize. The problem seems to be that initDeflate has to allocate four slices for every single writer that's created, in the case of HTTP that's for every request.

With a high volume of requests that's a lot of generated garbage.

On Monday, July 8, 2013 6:16:54 AM UTC-7, Jamie Hall wrote:

[Bryan Whitehead]

Jamie, nice package. I've been looking at using this for SPDY support
but can you commit to a license for this before I make the commitment?

Thanks!

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Jamie Hall]

Sorry, I hadn't noticed I'd forgotten the license. I've added a 2-clause BSD license.

Thanks

[Kamil Kisiel]

In tip you can now reset the zlib writers to save on allocation:

https://codereview.appspot.com/13171046

On Monday, July 8, 2013 6:16:54 AM UTC-7, Jamie Hall wrote: