UnpooledDirectByteBuf only released on GC?

519 views
Skip to first unread message

Kevin Burton

unread,
Jan 1, 2016, 3:43:09 PM1/1/16
to Netty discussions
I think that unpooled/direct byte buffers might actually be harmful.

The way anonymous direct buffers are releases (at least on Java 7... I didn't test this on Java 8 yet but I'm seeing similar behavior) is that they're only released on GC.

The problem comes in when you create too many buffers , still have plenty of JVM memory, but not enough OS memory.

At this point you run out of memory and you get OOM killed.

You could use pooled of course, and set a cap on the amount of direct memory you will use. 

Which leaves me with 2 questions:

- how do you set pooled as the default in 4.0..?

- should unpooled really be the default?

We tested this in production and found that our proxy setup requires a LOT of memory before running into memory problems.  4-8GB ... this is primarily because (at least I hypothesize) Java GC isn't coming in and freeing memory in time and the OS has no memory left to spare.

Norman Maurer

unread,
Jan 2, 2016, 5:41:48 AM1/2/16
to ne...@googlegroups.com
Hey Kevin,

comments inline…

On 01 Jan 2016, at 21:43, Kevin Burton <burto...@gmail.com> wrote:

I think that unpooled/direct byte buffers might actually be harmful.

The way anonymous direct buffers are releases (at least on Java 7... I didn't test this on Java 8 yet but I'm seeing similar behavior) is that they're only released on GC.

Yes that is still true in Java 8. If you ask me I think the whole idea of manager off-heap memory with a Garbage Collector is not going to fly, but that is just how it is :(

Anyway see below how we partially fixed it.


The problem comes in when you create too many buffers , still have plenty of JVM memory, but not enough OS memory.

Because of this we actually use a hack to release the memory once ByteBuf.release() is called. So we not depend on the GC here. This is done by getting a reference to the internal Cleaner interface and call it.

See:

That said this only fix one of the problems as the problem still is that the method is static synchronized which means it is a big bottleneck.



At this point you run out of memory and you get OOM killed.

You could use pooled of course, and set a cap on the amount of direct memory you will use. 

Which leaves me with 2 questions:

- how do you set pooled as the default in 4.0..?

You can do via system property:

-Dio.netty.allocator.type=pooled


Or you can also set on the Bootstrap / ServerBootstrap.


- should unpooled really be the default?

it is not in 4.1. 4.1 uses the pooled allocator by default. When we released 4.0.0.Final we did feel we did not have enough testing of the pooled allocator yet and so not made it the default back then. To not break exceptions of our users we not want to switch this now in a 4.0.x release.


We tested this in production and found that our proxy setup requires a LOT of memory before running into memory problems.  4-8GB ... this is primarily because (at least I hypothesize) Java GC isn't coming in and freeing memory in time and the OS has no memory left to spare.

At least for Netty buffers we should release the memory as soon as ByteBuf.release() is called (see above). That is even true with unpooled allocator. So I think your best bet would be a heap-dump and see if you can find out what the problem is.

Hope it helps.



--
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/d9826745-a5ea-46f8-8886-dbd1bd35f65a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kevin Burton

unread,
Jan 2, 2016, 2:19:03 PM1/2/16
to Netty discussions

Yes that is still true in Java 8. If you ask me I think the whole idea of manager off-heap memory with a Garbage Collector is not going to fly, but that is just how it is :(


Yeah. Super fun!   /s
 

That said this only fix one of the problems as the problem still is that the method is static synchronized which means it is a big bottleneck.
 
Ah.  Right. I remember we were talking about this about a year ago.  

-Dio.netty.allocator.type=pooled


Or you can also set on the Bootstrap / ServerBootstrap.


Ah.. right. I'm going to make sure noxy/littleproxy can do this properly now.

Are there any issues with pooled now in 4.0 or were some issues fixed in 4.1 which is why you feel its stable.

Kevin
 

Norman Maurer

unread,
Jan 4, 2016, 2:50:41 AM1/4/16
to ne...@googlegroups.com
There are no issues with the PooledByteBufAllocator in 4.0. The code in 4.1 and 4.0 is identical (at least almost), so I highly recommend it for production usage.

For some more background you may find the slides of my “Netty @ Apple” talk interesting:


If you have any more questions let me know, happy to help :)



--
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages