SslHandler not respecting unpooled allocator

[Chris Conroy]

I have encountered a strange issue in some server code on top of Netty 4.1 Final using the Java SSL server engine.

When I try to send a very large file through the pipeline, I get an OutOfDirectMemoryError from an Sslhandler#flush.

Of course, this could happen if I failed to call release somewhere in the pipeline, but we’re actually setting .option(ChannelOption.ALLOCATOR, new UnpooledByteBufAllocator(false)) in the bootstrap for this server in order to get Netty 3-like memory semantics. (We plan to shift to the pooled allocator later after shaking out any other Netty 4 issues).

The fact that the SslHandler is still using a direct buffer pool despite the option above seems like a bug. Or, am I misunderstanding this option?

If things were generally leaking, I would expect to see this eventually even without large requests, but it seems to only impact channels that are servicing a large transfer. I’m still gathering more data on the issue, but FWIW the leak detector in advanced mode has not reported any leaks, and interestingly I cannot reproduce the issue when running under paranoid mode.

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 4110417927, max: 4127195136)
        at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:592)
        at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:546)
        at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:699)
        at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:688)
        at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237)
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:221)
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:141)
        at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:262)
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170)
        at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:107)
        at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1461)
        at io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1471)
        at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:523)
        at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:501)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:754)
        at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:735)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:523)
        at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
        at io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:754)
        at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:735)
        at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
....

​

[Chris Conroy]

Another point to note: the allocator thinks it has used up 4G of memory yet this happens on an otherwise mostly idle server after transferring <10MB of the response.

It does appear that the arena for the thread handling the response has filled while the others are empty, which makes sense. I've attached the output of dumpStats at the time the exception was thrown. Perhaps this is a case of needing to first check for channel writability? It looks like with 16 arenas and 4G total available, a single thread would exhaust its output buffers with 256MB of pending data.

Still, I'm quite surprised to see the pooled allocator in play at all here.

[Norman Maurer]

If its a ServerBootstrap you need to use childOption(...) to specify the allocator for the accepted Channels. Using option(...) will set it for the ServerChannel which is not what you want here.

--
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAr-JCgHSW%3DFmNUEX6sXatXVQQcPanX44a%2BxQhaTMNUQw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Conroy]

Ah. Yep that appears to do the trick!

It does appear this is a case of the server writing too fast for the client. I should be able to address this by utilizing the channel high/low watermarks.

--
You received this message because you are subscribed to a topic in the Google Groups "Netty discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netty/qL7MeOicyCE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/5B4F5055-BCB0-46B5-8EF4-73825152CA41%40googlemail.com.

[Norman Maurer]

Nice!

Let me know if you have any other questions

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKC%3D%2BnWMQqU8stEKuOKkf8fBRq2d_ho%2BJ2digp4bhDBSfA%40mail.gmail.com.

[su...@mediaiqdigital.com]

Hi,

I encountered a similar issue where Buffer was not being cleared. Modified code to use PooledByteBufAllocator. But this one ended with OutOfDirectMemoryError when tried direct memory and if switched to heap, it encountered GC overhead limit. How shall I proceed with it?

[Norman Maurer]

I think you have kind of a similar problem here. You need to put some back pressure in place Seems like you continue writing even if Channel.isWritable() starts to return true.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/123f9431-a827-4461-811a-b28d1d7c3e15%40googlegroups.com.