Baffling Apparent Native Memory Leak

[Gareth Collins]

Hello,

I am sure I must be doing something wrong here...but it isn't obvious.

I have my own websockets implementation using netty...initially designed using Netty 3

but recently moved to Netty 4 (currently 4.0.15). I am testing with jdk 7 51 (x64) on Cent OS 6.3. My own Netty

implementation uses the Netty native http functionality, just like the netty native implementation. I am not sure

if this is relevant but this is running under OSGi in Karaf 2.3.3.

When executing long running session load tests against my server (several thousand clients hammering the system),

everything is working fine. Memory appears to stabilize quite nicely. I can run these tests for hours/overnight with tens

of millions of requests.

However, If I test very short sessions (open a websocket connection, send/receive a couple of

messages and close the websocket session) I appear to be leaking a massive amount of native memory

(guesstimate of 50-100K bytes per connection). I believe it to be native memory as I can't see anything 

wrong in Java profilers (or when looking at the heap/native memory JMX beans),yet the memory keeps increasing

eventually causing the OS to kill the process.

To try to completely rule out Netty native memory allocation in this I have tried to stop Netty from using any

native memory via Unsafe. I setup:

- childOption(channelOption.ALLOCATOR, new UnpooledByteBufAllocator(false));

- Set the options "-Dio.netty.noUnsafe=true" and "-Dio.netty.noPreferDirect=true"

and verified these options are on from the netty logs.

Even with these options on I still appear to be leaking the same amount of native memory.

I have tried the "paranoid" leak detector option as well...but I don't see any errors.

I am a little bit baffled.

So my questions are:

(1) Any foolish things I could be doing wrong which could cause such a big native memory leak like this

with Netty?

(1) Are there any other options I need to turn on/off to make sure that Netty does not use Native memory?

(2) This is slightly off-topic, but does anyone know of any good tooling for looking into java native memory leaks?

I tried using valgrind, but that was painfully, painfully slow (x100000 times slower)...making it difficult to run

enough tests to make any memory leak stick out from all the other valgrind noise.

Any pointers/suggestions would be greatly appreciated.

thanks in advance,

Gareth

[Norman Maurer]

There is no way to turn off native memory completely in Netty out of the box. 

What I would do is run with leak detector enabled and see if you have any leaks (I guess so):

http://netty.io/wiki/reference-counted-objects.html#wiki-h2-9

-- 
Norman Maurer

nma...@redhat.com

JBoss, by Red Hat

--
 
---
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Gareth Collins]

Hello Norman,

Thank you very much for the response!

Is there anything special I need to do to force it to do the leak check now (e.g. force a garbage collection via JMX)?

I ask because I don't see anything, even on paranoid.

thanks in advance,

Gareth

[Serguei Mourachov]

Gareth, are you using SSL for your websocket connections?

Serguei Mourachov

[Gareth Collins]

I found the source of my problem. It wasn't related to netty.

I was using compression (to implement per-frame websockets compression) and I wasn't explicitly calling end() on the Inflater/Deflater (I was just calling close on the associated Deflater/Inflater Input/OutputStreams).

I would have caught this earlier but the Java 7 javadoc says:

public void end()

Closes the compressor and discards any unprocessed input. This method should be called when the compressor is no longer being used, but will also be called automatically by the finalize() method. Once this method is called, the behavior of the Deflater object is undefined.

public void end()

Closes the decompressor and discards any unprocessed input. This method should be called when the decompressor is no longer being used, but will also be called automatically by the finalize() method. Once this method is called, the behavior of the Inflater object is undefined.

Thus I assumed that any memory would be garbage collected anyway. I also had previously created Deflater/Inflater instances per frame (rather than reusing them via reset()) and I don't remember having a memory leak.

It was interesting that I couldn't see this memory leak in any java memory profilers...or even in valgrind.

Anyway I am happy that I found my problem. I now call end() in handlerRemoved just like what is done in JdkZlibDecoder.

thanks again,

Gareth

["이희승 (Trustin Lee)"]

Thanks for coming back to us with the post-mortem.

T

On Sun 09 Feb 2014 06:38:08 PM PST, Gareth Collins wrote:
> I found the source of my problem. It wasn't related to netty.
>
> I was using compression (to implement per-frame websockets
> compression) and I wasn't explicitly calling end() on the
> Inflater/Deflater (I was just calling close on the associated
> Deflater/Inflater Input/OutputStreams).
>
> I would have caught this earlier but the Java 7 javadoc says:
>
> public void end()
> Closes the compressor and discards any unprocessed input. This method

> should be called when the compressor is no longer being used,*but will
> also be called automatically by the finalize() method.* Once this

> method is called, the behavior of the Deflater object is undefined.
>
> public void end()
> Closes the decompressor and discards any unprocessed input. This
> method should be called when the decompressor is no longer being used,

> *but will also be called automatically by the finalize() method.* Once

> this method is called, the behavior of the Inflater object is undefined.
>
> Thus I assumed that any memory would be garbage collected anyway. I
> also had previously created Deflater/Inflater instances per frame
> (rather than reusing them via reset()) and I don't remember having a
> memory leak.
>
> It was interesting that I couldn't see this memory leak in any java
> memory profilers...or even in valgrind.
>
> Anyway I am happy that I found my problem. I now call end() in
> handlerRemoved just like what is done in JdkZlibDecoder.
>
> thanks again,
> Gareth
>
> On Sunday, February 9, 2014 12:42:25 AM UTC-5, Serguei Mourachov wrote:
>
> Gareth, are you using SSL for your websocket connections?
>
> Serguei Mourachov
>

> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Netty discussions" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to netty+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

--
https://twitter.com/trustin
https://twitter.com/trustin_ko
https://twitter.com/netty_project

[Norman Maurer]

Thanks for the info .... Ever thought about contribute websockets deflate to the netty project? We love contributions

Bye,

Norman

--

[Gareth Collins]

I would like to but I don't think I would get approval (at least at the moment). My implementation is also out of date - it is for the draft standard (per frame compression).

The current standard is per message (now supported by Chrome 32+).

thanks,

Gareth