java.lang.OutOfMemoryError: Direct buffer memory - regresion in CR5?

[Dan Tran]

This problem has been reported and fixed before CR4, not sure why it comes back again in CR4+ ( due to recent big change? )

Basically, I am processing a 20Kmsg/sec with 1kMsg payload. All messages are released at the message handler.

The processing of the payload surely take some processing time as well, and my log are flushed with the below stack trace.  ( note my memory fluctuate normally )

if I just discard the msg, then the problem disappears.  

May be this is a regression in Netty CR4+?? 

Thanks

-D

2013-06-18 17:26:18,390 | WARN  | ioEventLoopGroup-3-1 UdpSyslogHandler                 59 |                      | Unexpected exception from downstream.
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:658)[:1.7.0_03]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)[:1.7.0_03]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)[:1.7.0_03]
        at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:371)[122:io.netty.buffer:4.0.0.CR5]
        at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:143)[122:io.netty.buffer:4.0.0.CR5]
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:132)[122:io.netty.buffer:4.0.0.CR5]
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:94)[122:io.netty.buffer:4.0.0.CR5]
        at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:233)[122:io.netty.buffer:4.0.0.CR5]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:130)[122:io.netty.buffer:4.0.0.CR5]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:121)[122:io.netty.buffer:4.0.0.CR5]
        at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:82)[122:io.netty.buffer:4.0.0.CR5]
        at io.netty.channel.FixedRecvByteBufAllocator$HandleImpl.allocate(FixedRecvByteBufAllocator.java:38)[121:io.netty.transport:4.0.0.CR5]
        at io.netty.channel.socket.nio.NioDatagramChannel.doReadMessages(NioDatagramChannel.java:210)[121:io.netty.transport:4.0.0.CR5]
        at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:64)[121:io.netty.transport:4.0.0.CR5]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:489)[121:io.netty.transport:4.0.0.CR5]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464)[121:io.netty.transport:4.0.0.CR5]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:358)[121:io.netty.transport:4.0.0.CR5]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)[123:io.netty.common:4.0.0.CR5]
        at java.lang.Thread.run(Thread.java:722)[:1.7.0_03]

[Dan Tran]

My issue relates to https://github.com/netty/netty/issues/1393#issuecomment-19434569 ??

Thanks

-D

--
 
---
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

["이희승 (Trustin Lee)"]

Did you try to increase the maximum direct memory of the JVM? Could
you tell me the current maximum direct memory size using
PlatformDependent.maxDirectMemory()?

Did you disable resource leak detection?
(-Dio.netty.noResourceLeakDetection) If so, please enable it to see if
there's a leak.

> <mailto:netty%2Bunsu...@googlegroups.com>.

> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Netty discussions" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to netty+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
https://twitter.com/trustin
https://twitter.com/trustin_ko
https://twitter.com/netty_project

[Dan Tran]

Thanks Trustin

Here are a few more details

  0. I dont shutdown server udp socket when issur occur ( continue to serve )

  1. My PlatformDependent.maxDirectMemory(): 1871642624

  2. I dont disable resource leak detection

  3. Since syslog max msg side is 60K,  this is my added .option( ChannelOption.RCVBUF_ALLOCATOR, new FixedRecvByteBufAllocator( 1024 * 64 ) );

  4. I also use .option( ChannelOption.SO_RCVBUF, if set more than 8M, OOM easily happen

  

  5. Have not try to increase maximum direct memory of the JVM yet

Thank you for helping out

-Dan

    send an email to netty+unsubscribe@googlegroups.com

    <mailto:netty%2Bunsubscribe@googlegroups.com>.

    For more options, visit https://groups.google.com/groups/opt_out.



--

---
You received this message because you are subscribed to the Google
Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send

an email to netty+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
https://twitter.com/trustin
https://twitter.com/trustin_ko
https://twitter.com/netty_project

--

--- You received this message because you are subscribed to the Google Groups "Netty discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to netty+unsubscribe@googlegroups.com.

[Dan Tran]

One more fact.

I dont see problem at my Centos 6.4 with OpenJdk 7

-D

["이희승 (Trustin Lee)"]

I guess, when a burst of datagram packets are received, Netty allocates
many 64KiB buffers for a very short peroid of time and that could lead
to OOME.

Could you try to limit the maximum number of reads in
AbstractNioMessageChannel.java?


https://github.com/netty/netty/blob/master/transport/src/main/java/io/netty/channel/nio/AbstractNioMessageChannel.java#L63

It's currently an infinite loop, so I believe there's a chance it will
allocate too many 64K buffers before triggering a messageReceived
event. Before making a related change, I'd like to get your
confirmation.

Thanks,
T

On Fri 21 Jun 2013 10:01:41 AM KST, Dan Tran wrote:
> One more fact.
>
> I dont see problem at my Centos 6.4 with OpenJdk 7
>
> -D
>
>
> On Thu, Jun 20, 2013 at 9:42 AM, Dan Tran <dan...@gmail.com
> <mailto:dan...@gmail.com>> wrote:
>
> Thanks Trustin
>
> Here are a few more details
>
> 0. I dont shutdown server udp socket when issur occur ( continue
> to serve )
> 1. My PlatformDependent.maxDirectMemory(): 1871642624
> 2. I dont disable resource leak detection
> 3. Since syslog max msg side is 60K, this is my added .option(
> ChannelOption.RCVBUF_ALLOCATOR, new FixedRecvByteBufAllocator(
> 1024 * 64 ) );
> 4. I also use .option( ChannelOption.SO_RCVBUF, if set more than
> 8M, OOM easily happen
> 5. Have not try to increase maximum direct memory of the JVM yet
>
> Thank you for helping out
>
> -Dan
>
>
> On Thu, Jun 20, 2013 at 12:41 AM, "이희승 (Trustin Lee)"
> <t...@motd.kr <mailto:t...@motd.kr>> wrote:
>
> Did you try to increase the maximum direct memory of the JVM?
> Could you tell me the current maximum direct memory size

> using PlatformDependent.__maxDirectMemory()?

>
> Did you disable resource leak detection?

> (-Dio.netty.__noResourceLeakDetection) If so, please enable it

> to see if there's a leak.
>
>
> On Thu 20 Jun 2013 04:17:29 PM KST, Dan Tran wrote:
>
> My issue relates to

> https://github.com/netty/__netty/issues/1393#__issuecomment-19434569

> <https://github.com/netty/netty/issues/1393#issuecomment-19434569>
> ??
>
> Thanks
>
> -D
>
>
> On Thu, Jun 20, 2013 at 12:07 AM, Dan Tran
> <dan...@gmail.com <mailto:dan...@gmail.com>

> <mailto:dan...@gmail.com <mailto:dan...@gmail.com>>> wrote:
>
> This problem has been reported and fixed before CR4,
> not sure why
> it comes back again in CR4+ ( due to recent big change? )
>
> Basically, I am processing a 20Kmsg/sec with 1kMsg
> payload. All
> messages are released at the message handler.
> The processing of the payload surely take some
> processing time as
> well, and my log are flushed with the below stack
> trace. ( note
> my memory fluctuate normally )
>
> if I just discard the msg, then the problem disappears.
>
> May be this is a regression in Netty CR4+??
>
> Thanks
>
> -D
>
>
>
> 2013-06-18 17:26:18,390 | WARN | ioEventLoopGroup-3-1
> UdpSyslogHandler 59 |
> | Unexpected exception from downstream.
> java.lang.OutOfMemoryError: Direct buffer memory
> at

> java.nio.Bits.reserveMemory(__Bits.java:658)[:1.7.0_03]
> at
> java.nio.DirectByteBuffer.<__init>(DirectByteBuffer.java:__123)[:1.7.0_03]
> at
> java.nio.ByteBuffer.__allocateDirect(ByteBuffer.__java:306)[:1.7.0_03]
> at
> io.netty.buffer.PoolArena$__DirectArena.newChunk(__PoolArena.java:371)[122:io.__netty.buffer:4.0.0.CR5]
> at
> io.netty.buffer.PoolArena.__allocateNormal(PoolArena.java:__143)[122:io.netty.buffer:4.0.__0.CR5]
> at
> io.netty.buffer.PoolArena.__allocate(PoolArena.java:132)[__122:io.netty.buffer:4.0.0.CR5]
> at
> io.netty.buffer.PoolArena.__allocate(PoolArena.java:94)[__122:io.netty.buffer:4.0.0.CR5]
> at
> io.netty.buffer.__PooledByteBufAllocator.__newDirectBuffer(__PooledByteBufAllocator.java:__233)[122:io.netty.buffer:4.0.__0.CR5]
> at
> io.netty.buffer.__AbstractByteBufAllocator.__directBuffer(__AbstractByteBufAllocator.java:__130)[122:io.netty.buffer:4.0.__0.CR5]
> at
> io.netty.buffer.__AbstractByteBufAllocator.__directBuffer(__AbstractByteBufAllocator.java:__121)[122:io.netty.buffer:4.0.__0.CR5]
> at
> io.netty.buffer.__AbstractByteBufAllocator.__ioBuffer(__AbstractByteBufAllocator.java:__82)[122:io.netty.buffer:4.0.0.__CR5]
> at
> io.netty.channel.__FixedRecvByteBufAllocator$__HandleImpl.allocate(__FixedRecvByteBufAllocator.__java:38)[121:io.netty.__transport:4.0.0.CR5]
> at
> io.netty.channel.socket.nio.__NioDatagramChannel.__doReadMessages(__NioDatagramChannel.java:210)[__121:io.netty.transport:4.0.0.__CR5]
> at
> io.netty.channel.nio.__AbstractNioMessageChannel$__NioMessageUnsafe.read(__AbstractNioMessageChannel.__java:64)[121:io.netty.__transport:4.0.0.CR5]
> at
> io.netty.channel.nio.__NioEventLoop.__processSelectedKey(__NioEventLoop.java:489)[121:io.__netty.transport:4.0.0.CR5]
> at
> io.netty.channel.nio.__NioEventLoop.__processSelectedKeysOptimized(__NioEventLoop.java:464)[121:io.__netty.transport:4.0.0.CR5]
> at
> io.netty.channel.nio.__NioEventLoop.run(NioEventLoop.__java:358)[121:io.netty.__transport:4.0.0.CR5]
> at
> io.netty.util.concurrent.__SingleThreadEventExecutor$2.__run(SingleThreadEventExecutor.__java:101)[123:io.netty.common:__4.0.0.CR5]
> at
> java.lang.Thread.run(Thread.__java:722)[:1.7.0_03]

>
> --
>
> ---
> You received this message because you are subscribed
> to the Google
> Groups "Netty discussions" group.
> To unsubscribe from this group and stop receiving
> emails from it,

> send an email to netty+unsubscribe@__googlegroups.com
> <mailto:netty%2Bunsu...@googlegroups.com>
> <mailto:netty%2Bunsubscribe@__googlegroups.com
> <mailto:netty%252Buns...@googlegroups.com>>.
>
> For more options, visit
> https://groups.google.com/__groups/opt_out
> <https://groups.google.com/groups/opt_out>.

>
>
>
> --
>
> ---
> You received this message because you are subscribed to
> the Google
> Groups "Netty discussions" group.
> To unsubscribe from this group and stop receiving emails
> from it, send

> an email to netty+unsubscribe@__googlegroups.com
> <mailto:netty%2Bunsu...@googlegroups.com>.
> For more options, visit
> https://groups.google.com/__groups/opt_out
> <https://groups.google.com/groups/opt_out>.

>
>
>
> --
> https://twitter.com/trustin
> https://twitter.com/trustin_ko

> https://twitter.com/netty___project

> <https://twitter.com/netty_project>
>
>
> --
>
> --- You received this message because you are subscribed to
> the Google Groups "Netty discussions" group.
> To unsubscribe from this group and stop receiving emails from

> it, send an email to netty+unsubscribe@__googlegroups.com
> <mailto:netty%2Bunsu...@googlegroups.com>.
> For more options, visit
> https://groups.google.com/__groups/opt_out
> <https://groups.google.com/groups/opt_out>.

>
>
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Netty discussions" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to netty+un...@googlegroups.com.

["이희승 (Trustin Lee)"]

Even better if you could log the number of datagram packets per
messageReceived event and let me know how large it is.

[shaox...@gmail.com]

switch the FixedRecvByteBufAllocator to AdaptiveRecvByteBufAllocator like this
bootstrap.option( ChannelOption.RCVBUF_ALLOCATOR, AdaptiveRecvByteBufAllocator.DEFAULT );
it may fix the OOM of direct memory, but less datagram packets are received.

在 2013年6月20日星期四UTC+8下午3时07分36秒，Dan Tran写道：

[Norman Maurer]

Yeah it is this problem. I fixed it before our big refactor but it seems like the change got reverted.

Bye,
Norman

[Dan Tran]

Thanks Norman, it is good to hear a confirmation for netty dev. So i guess a fix is coming soon, or should I follow thru with Trustin's advice?

-D

[Norman Maurer]

Just for the record this was the reported issue and the commits to fix it.

https://github.com/netty/netty/issues/1388

[Norman Maurer]

Maybe an even better way to fix would be to allow to configure the max bytes to read before fire an event. This should also be used for byte based channels

[Dan Tran]

Hi,

What is verdict here? should we reopen the original issue?

Thanks

-D

["이희승 (Trustin Lee)"]

No. Considering that you wasted enough of your time because of this
regression, we will take care of it today for you - we can file an
issue and close it.

[Dan Tran]

Big thanks to you Trustin.

-D

On Mon, Jun 24, 2013 at 5:35 PM, "이희승 (Trustin Lee)" <t...@motd.kr> wrote:

No. Considering that you wasted enough of your time because of this regression, we will take care of it today for you - we can file an issue and close it.

On Tue 25 Jun 2013 08:04:24 AM KST, Dan Tran wrote:

Hi,

What is verdict here? should we reopen the original issue?

Thanks

-D

--

---
You received this message because you are subscribed to the Google
Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send

an email to netty+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
https://twitter.com/trustin
https://twitter.com/trustin_ko
https://twitter.com/netty_project

--

--- You received this message because you are subscribed to the Google Groups "Netty discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to netty+unsubscribe@googlegroups.com.

["이희승 (Trustin Lee)"]

https://github.com/netty/netty/issues/1486

Fix pushed. Could you confirm the fix?

Cheers,
T

On Tue 25 Jun 2013 09:51:04 AM KST, Dan Tran wrote:
> Big thanks to you Trustin.
>
> -D
>
>
> On Mon, Jun 24, 2013 at 5:35 PM, "이희승 (Trustin Lee)" <t...@motd.kr
> <mailto:t...@motd.kr>> wrote:
>
> No. Considering that you wasted enough of your time because of
> this regression, we will take care of it today for you - we can
> file an issue and close it.
>
>
> On Tue 25 Jun 2013 08:04:24 AM KST, Dan Tran wrote:
>
> Hi,
>
> What is verdict here? should we reopen the original issue?
>
> Thanks
>
> -D
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Netty discussions" group.
> To unsubscribe from this group and stop receiving emails from
> it, send

> an email to netty+unsubscribe@__googlegroups.com
> <mailto:netty%2Bunsu...@googlegroups.com>.
> For more options, visit
> https://groups.google.com/__groups/opt_out
> <https://groups.google.com/groups/opt_out>.

>
>
>
> --
> https://twitter.com/trustin
> https://twitter.com/trustin_ko

> https://twitter.com/netty___project

> <https://twitter.com/netty_project>
>
> --
>
> --- You received this message because you are subscribed to the
> Google Groups "Netty discussions" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to netty+unsubscribe@__googlegroups.com
> <mailto:netty%2Bunsu...@googlegroups.com>.
> For more options, visit https://groups.google.com/__groups/opt_out
> <https://groups.google.com/groups/opt_out>.

>
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Netty discussions" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to netty+un...@googlegroups.com.

[Dan Tran]

It works.  

Thank you!!!

-D

[Dan Tran]

btw, looking forward to pickup official fix at CR7

Thanks

-D