ChannelOutboundBuffer flushed buffer leak?

[Chris Conroy]

I’ve been trying to track down a NIO memory leak that occurs in a Netty application I am porting from Netty 3 to Netty 4. This leak does not occur in the Netty 3 version of the application.

For now, I’m using only unpooled heap buffers in Netty 4, but NIO buffers do come into play for socket communication.

I’ve captured a few heap dumps from affected instances, and in each it appears that the leaked DirectByteBuf java objects are rooted in an io.netty.util.Recycler.

These buffers remain indefinitely: I can disable the application to drain traffic and force GCs, but the # of NIO buffers and NIO allocated space stays flat.

The issue is likely related to slow readers. However, the leak persists long after all channels have been closed.

I implemented a writability listener and the leak does appear to go away if I stop writing to a channel after it goes unwritable. This is good, but I’m still worried that this just makes the problem less likely since it’s still possible to write/flush and have pending data: writability just limits how much data will be buffered.

Digging into ChannelOutBoundBuffer I see the following stanza in close:

// Release all unflushed messages.
try {
    Entry e = unflushedEntry;
    while (e != null) {
        // Just decrease; do not trigger any events via decrementPendingOutboundBytes()
        int size = e.pendingSize;
        TOTAL_PENDING_SIZE_UPDATER.addAndGet(this, -size);

        if (!e.cancelled) {
            ReferenceCountUtil.safeRelease(e.msg);
            safeFail(e.promise, cause);
        }
        e = e.recycleAndGetNext();
    }
} finally {
    inFail = false;
}
clearNioBuffers();

This seems a bit curious to me: why are flushed buffers not released here? Since the leak seems to be rooted in the Recycler, this could be the culprit…What do you think?

​

[Norman Maurer]

failFlushed(...) should be called to fail and release all flushed messages.

Are you saying this not happens?

--
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Conroy]

Just another data point: I confirmed the leak is still there with a writability listener: it just occurs very slowly. This isn't a big surprise if the above is indeed a correct analysis of the issue.

[Chris Conroy]

Ah okay: I didn't see the calls to failFlushed since they occur above the stanza I found suspicious. 

So, the above explanation is probably not correct. Still, I am seeing a leak where DirectByteBufs are rooted to the recycler, and the speed at which these buffers leak appears to be correlated with slow/partial readers.

[Norman Maurer]

Can you provide a reproducer? Also did you try to run with paranoid leak detection

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com.

[Chris Conroy]

I have not been able to reproduce locally yet, but I do see it in a cluster that takes a lot of varied traffic. The leak detector has not fired for this under advanced. I will give paranoid a shot to be safe, but it's my understanding that the leak detection framework is more for dealing with pooled byte buf misuse, but in this case I am exclusively using unpooled heap byte bufs: these are just the socket direct byte bufs that appear to be leaking.

I meant to add this earlier: The path to GC root goes:

io.netty.buffer.ByteBufUtil$ThreadLocalUnsafeDirectByteBuf

  io.netty.util.Recycler$DefaultHandle

    io.netty.util.Recycler$DefaultHandle[]

      io.netty.util.Recycler#Stack

        java.lang.Object[]

          io.netty.util.internal.InternalThreadLocalMap

            ... (more thread local map refs up to java.lang.Thread)

[Norman Maurer]

Are you using 4.0 or 4.1 ?

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com.

[Chris Conroy]

4.1.0.Final

--
You received this message because you are subscribed to a topic in the Google Groups "Netty discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com.

[Norman Maurer]

Can you do me a favour and run with: 

-Dio.netty.recycler.maxCapacity=0

And let me know if you still see some leaks ?

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com.

[Chris Conroy]

I ran with paranoid while observing the leak and got no messages from the ResourceLeakDetector.

Initial results from recycler maxCapacity of 0 are looking positive with respect to the leak. However, I did see a large spike of NIO memory allocated (1G) where the Netty 3 version of this app receiving similar traffic peaks at about 1/10th that. I'll let this run for a while and report back later with the full results.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com.

[Norman Maurer]

Btw all the “leaking” buffers are always:

ThreadLocalUnsafeDirectByteBuf

?

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com.

[Chris Conroy]

Yes all the leaked buffers look to be ThreadLocalUnsafeDirectByteBuf

Disabling the recycler w/o backpressure on slow readers resulted in several massive allocation events. I was able to free them by forcing GC so no leaks there. Memory usage with backpressure on slow readers is back down to Netty 3 w/o backpressure levels. I don't see any leaks so far.

Assuming that disabling the recycler does indeed prevent leaks, where do we go from there? Any ideas of things to look for in my application that might cause the recycler to get into a bad state?

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/A77C4A03-0A98-4104-9791-7F161A3AAD91%40googlemail.com.

[Norman Maurer]

Are you actually sure these are really leaked ?

The point of the “ThreadLocalUnsafeDirectByteBuf” is that it can be reused so it is expected that it not gets released after its written but put back in the recycler. Or you saying there are too many of these ?

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKCA8g-4E0AJCf%2BopC2NMufmQBGcWnMXHkwbDj-veW7x8w%40mail.gmail.com.

[Chris Conroy]

This leak resulted in exhaustion of 4G of NIO memory. The same application under Netty 3 only ever uses <200MB of NIO memory. I have run several experiments where I take some traffic, disable the node from serving traffic, and then force a full GC. The allocated NIO memory does not return to normal levels.

Spot checking the byte buf handles from my heap dump, I see lots (all?) `recycle` and `lastRecycleIds` of `int -2147483648 = 0x80000000`

The experiment yesterday with the recycler disabled did not result in any such leaks over several hours of traffic.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/D30BEE2E-9119-4FD3-A144-F6B765B189E8%40googlemail.com.

[Norman Maurer]

Thanks Chris,

this sounds really “fishy”. Let me try to debug this a bit more (not sure yet how tho).

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADHSVtqFoqMRak34S23OsDrZTEfADSTMpKGa3bguU-ig%40mail.gmail.com.

[Norman Maurer]

One last thing…

Can you tell me what the capacity() of the “leaked” buffer is ?

[Chris Conroy]

Here's an example leaked buf

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/297B68FF-2B8F-4EE6-92B3-81856A901A7A%40googlemail.com.

[Norman Maurer]

And how many of these you have ?

Sorry for all the questions :(

On 20 Jul 2016, at 19:19, Chris Conroy <cco...@squareup.com> wrote:

Here's an example leaked buf

<Screen Shot 2016-07-20 at 1.18.24 PM.png>

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKABgsVZaToNgE-UqK04v_fsy508Gwd35BTyq3_rZtfZ5Q%40mail.gmail.com.

[Norman Maurer]

And also the what is under the “Stack.thread” field stored ? Is it the same for all ?

Maybe you could even share a dump with me ?

[Chris Conroy]

It's no problem! I'm sorry for all the back and forth. I'd just send you the heap dump if I could, but alas it will be difficult to impossible to sanitize it from sensitive data. (As an aside, I really wish there were tools that let you interact with java heap dumps more programmatically...)

In the particular heap dump I'm looking at, I have 12,762 such buffers. Interestingly, I see 123k Recycler$DefaultHandles in the heap...

Here's the only path to GC roots from the leaked byte bufs:

The threads appear to all be server worker threads. The Biggest Objects - Dominators view for strong references in YourKit shows me that the server worker threads are the dominant roots in the heap:

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/F4977516-4D85-4995-AA5D-AFDC2EC18911%40googlemail.com.

[Norman Maurer]

Thanks I will have another look.

One thing you could try is to use “-Dio.netty.threadLocalDirectBufferSize=16” and see if the leaks are gone then. 

This is not a solution but would help me a bit :)

On 20 Jul 2016, at 19:39, Chris Conroy <cco...@squareup.com> wrote:

It's no problem! I'm sorry for all the back and forth. I'd just send you the heap dump if I could, but alas it will be difficult to impossible to sanitize it from sensitive data. (As an aside, I really wish there were tools that let you interact with java heap dumps more programmatically...)

In the particular heap dump I'm looking at, I have 12,762 such buffers. Interestingly, I see 123k Recycler$DefaultHandles in the heap...

<Screen Shot 2016-07-20 at 1.31.40 PM.png>

Here's the only path to GC roots from the leaked byte bufs:

<Screen Shot 2016-07-20 at 1.34.50 PM.png>

The threads appear to all be server worker threads. The Biggest Objects - Dominators view for strong references in YourKit shows me that the server worker threads are the dominant roots in the heap:

<Screen Shot 2016-07-20 at 1.38.19 PM.png>

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAMJyi3yLcSVunKLeRdtjJ7%2BQY0ExDq%2Bemb%2BoK9z8kg9A%40mail.gmail.com.

[Chris Conroy]

Sure thing. I'll give that a shot.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/F3DB7987-51F1-49A4-B77D-49AD2A270FE4%40googlemail.com.

[Chris Conroy]

I was a bit delayed as I had to roll back for an unrelated reason.

Using “-Dio.netty.threadLocalDirectBufferSize=16” appears to fix the leak.

Can you help me understand how much memory we should expect to sit resident in the recycler under the default settings? The documentation is very scarce... At first glance, the default of 266k objects in the recycler and 64k thread local buffer size limit would consume 17GB of NIO memory? But, that can't be right...

https://github.com/netty/netty/issues/3166 and https://github.com/netty/netty/issues/4146 seem relevant here.

[Norman Maurer]

Hey Chris,

the thing is that I would not expect to have so much stuff in there as it should pick things from the Stack per thread. Or do you create and destroy a lot of threads ?

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAWh8EQDe-D0hiY%2B4BXbLvDdtjUMDNoyY1B4_bBpYYNJg%40mail.gmail.com.

[Chris Conroy]

Nope: we use fixed size thread pools and are not killing threads.

I'm not saying that we've gotten that much allocated (we'd OOME well before then), but the defaults seem to imply that the recycler will hold on to 17G (per thread?) of memory if you were to allocate that much. This seems wrong.

I'm running a few experiments changing the recycler max capacity and the thread local direct buffer size which I think should unblock us.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/0328BF3C-794D-4821-8665-2C6C88DAF8E0%40googlemail.com.

[Norman Maurer]

Yeah I agree the default is not super good… What you think would be a “sane” default ?

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKBzJ84qpabpppj_SRQD6N5OVLN6kQnCFEV8nMYZ4Pcdjg%40mail.gmail.com.

[Chris Conroy]

If the recycler is used by each EventLoopGroup, then it probably should have a per EventLoopGroup configuration since you’ll need lower thresholds for more threads. In practice I’m only seeing much usage of the recycler on one of my EventLoopGroup s but I would be worried about running out of memory unnecessarily in some other situation where another group ends up buffering a large amount of data due to some slowdown.

This would be a bit easier to configure safe automatic defaults if it were a global (instead of per thread) recycler. How crazy would that be? Without that, the recyclers need to be small enough to multiply per thread in the app or there needs to be some kind of coordination mechanism to disable recycler growth in some threads if other threads are currently using a lot of capacity. There also might be some value in expiring older buffers so that after high pressure periods they are able to be reclaimed (I have had success doing this in MRU object pools elsewhere)

For HTTP applications, the default chunk size is 4k, so I imagine most buffers would be under that size? I’m currently seeing good memory usage results without too much extra GC using -Dio.netty.recycler.maxCapacity=4096 -Dio.netty.threadLocalDirectBufferSize=8192 but I haven’t explored too many other options yet. I did not seem to get much utilization with a thread local buffer size threshold of 4k for whatever reason though.

​

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/320953AF-FBE7-4C5E-A84C-F3A7CBEDFB80%40googlemail.com.

[Norman Maurer]

On 25 Jul 2016, at 20:00, Chris Conroy <cco...@squareup.com> wrote:

If the recycler is used by each EventLoopGroup, then it probably should have a per EventLoopGroup configuration since you’ll need lower thresholds for more threads. In practice I’m only seeing much usage of the recycler on one of my EventLoopGroup s but I would be worried about running out of memory unnecessarily in some other situation where another group ends up buffering a large amount of data due to some slowdown.

At the moment these Recyclers are not point to an EventLoopGroup as these are no even aware of anything like EventLoops. For example you may use buffers without EventLoops at all.

This would be a bit easier to configure safe automatic defaults if it were a global (instead of per thread) recycler. How crazy would that be? Without that, the recyclers need to be small enough to multiply per thread in the app or there needs to be some kind of coordination mechanism to disable recycler growth in some threads if other threads are currently using a lot of capacity. There also might be some value in expiring older buffers so that after high pressure periods they are able to be reclaimed (I have had success doing this in MRU object pools elsewhere)

So you are talking about automatically drop stuff if its not used for X timeframe ?

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKBJSvf3o9XAKLHYGkbgFvVhsRmx1CEuAJg1YQWOwAaDHA%40mail.gmail.com.

[Chris Conroy]

On Mon, Jul 25, 2016 at 2:35 PM, 'Norman Maurer' via Netty discussions <ne...@googlegroups.com> wrote:

At the moment these Recyclers are not point to an EventLoopGroup as these are no even aware of anything like EventLoops. For example you may use buffers without EventLoops at all.

This would be a bit easier to configure safe automatic defaults if it were a global (instead of per thread) recycler. How crazy would that be? Without that, the recyclers need to be small enough to multiply per thread in the app or there needs to be some kind of coordination mechanism to disable recycler growth in some threads if other threads are currently using a lot of capacity. There also might be some value in expiring older buffers so that after high pressure periods they are able to be reclaimed (I have had success doing this in MRU object pools elsewhere)

So you are talking about automatically drop stuff if its not used for X timeframe ?

Exactly! This works best if the pool is MRU retrieval instead of LRU or random access retrieval.

[Norman Maurer]

Something unrelated but why you use the UnpooledByteBufAllocator and not the PooledByteBufAllocator ?

--
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKC_f80VxeZOryQcOPHSZGEBLf0SuG6YS0eWx7T86%2Bb6zA%40mail.gmail.com.

[Chris Conroy]

In order to de-risk the rollout of the Netty 4 upgrade, we're starting out with the unpooled allocator. This was actually very helpful for digging into this issue since a full GC should have been able to reclaim most of the NIO memory.

We plan to transition to the pooled allocator once we shake out any other issues in the upgrade. Hopefully this is the last of those!

--
You received this message because you are subscribed to a topic in the Google Groups "Netty discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/A7B9770D-8385-4CFB-9D35-93DED749F1F2%40googlemail.com.

[Chris Conroy]

Ping: What do you think about a global recycler instead of many thread-local recyclers?

Also, can you provide some more context on the rationale behind the recycler? Especially with the PooledByteBufAllocator, NIO allocations should be very cheap, so why bother to reuse the buffers?

​

[Norman Maurer]

Comments inside..

On 29 Jul 2016, at 01:10, 'Chris Conroy' via Netty discussions <ne...@googlegroups.com> wrote:

Ping: What do you think about a global recycler instead of many thread-local recyclers?

Im not sure this can be done without too much overhead. But if you want to cook up a PR and show it with benchmarks I would be interested for sure :)

Also, can you provide some more context on the rationale behind the recycler? Especially with the PooledByteBufAllocator, NIO allocations should be very cheap, so why bother to reuse the buffers?

Its because of object allocation. It basically reuses the “ByteBuf” container object (non the actual memory here). 

Im working on another fix for the problem you see. And you also may be interested in these:

Allow to limit the maximum number of WeakOrderQueue instances per The…

Introduce allocation / pooling ratio in Recycler

Set Recycler DEFAULT_INITIAL_MAX_CAPACITY to a more sane value

Ensure shared capacity is updated correctly when WeakOrderQueue is co…

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDk8%3DhNfW1C6nDzj7wPEMZm7m9bkq%3D95fYX7hvnnR%3D0JQ%40mail.gmail.com.

[Chris Conroy]

On Fri, Jul 29, 2016 at 12:26 AM, ‘Norman Maurer’ via Netty discussions <ne...@googlegroups.com> wrote:

Comments inside..

On 29 Jul 2016, at 01:10, 'Chris Conroy' via Netty discussions <ne...@googlegroups.com> wrote:

Ping: What do you think about a global recycler instead of many thread-local recyclers?

Im not sure this can be done without too much overhead. But if you want to cook up a PR and show it with benchmarks I would be interested for sure :) 

Also, can you provide some more context on the rationale behind the recycler? Especially with the PooledByteBufAllocator, NIO allocations should be very cheap, so why bother to reuse the buffers?

Its because of object allocation. It basically reuses the “ByteBuf” container object (non the actual memory here).

The ByteBuf objects do pin the NIO memory with an unpooled allocator. Are you saying that this is not the case in the pooled allocator?

Object allocation is always very cheap. Garbage collection in the eden space is incredibly cheap, and most buffers are short-lived. I suspect that this may be a premature micro-optimization. I see no difference in JVM pause time or GC run rates when I disable the recycler completely.

[Norman Maurer]

On 29 Jul 2016, at 18:51, 'Chris Conroy' via Netty discussions <ne...@googlegroups.com> wrote:

On Fri, Jul 29, 2016 at 12:26 AM, ‘Norman Maurer’ via Netty discussions <ne...@googlegroups.com> wrote:

Comments inside..

On 29 Jul 2016, at 01:10, 'Chris Conroy' via Netty discussions <ne...@googlegroups.com> wrote:

Ping: What do you think about a global recycler instead of many thread-local recyclers?

Im not sure this can be done without too much overhead. But if you want to cook up a PR and show it with benchmarks I would be interested for sure :) 

Also, can you provide some more context on the rationale behind the recycler? Especially with the PooledByteBufAllocator, NIO allocations should be very cheap, so why bother to reuse the buffers?

Its because of object allocation. It basically reuses the “ByteBuf” container object (non the actual memory here).

The ByteBuf objects do pin the NIO memory with an unpooled allocator. Are you saying that this is not the case in the pooled allocator?

What you mean here ? In the PooledByteBufAllocator the memory is “pooled” separately from the ByteBuf instance.

Object allocation is always very cheap. Garbage collection in the eden space is incredibly cheap, and most buffers are short-lived. I suspect that this may be a premature micro-optimization. I see no difference in JVM pause time or GC run rates when I disable the recycler completely.

In the past I saw issues because of heavy object allocation (even for this short-lived objects). If you not have this issue you could just disable the recycler. 

Im working on another fix for the problem you see. And you also may be interested in these:

Allow to limit the maximum number of WeakOrderQueue instances per The…

Introduce allocation / pooling ratio in Recycler

Set Recycler DEFAULT_INITIAL_MAX_CAPACITY to a more sane value

Ensure shared capacity is updated correctly when WeakOrderQueue is co… 

--
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAKQjw%3DPcqWoYFtKznk6RMtnCDu2G4FP9VAiAug76%2BmTw%40mail.gmail.com.

[Norman Maurer]

Also another thing that we considered for a long time was if we should have the ability to configure recyclers for different objects differently. Like allow to disable them for buffers but still keep for others objects etc.

WDYT ?

[Chris Conroy]

On Fri, Jul 29, 2016 at 12:57 PM, 'Norman Maurer' via Netty discussions <ne...@googlegroups.com> wrote:

On 29 Jul 2016, at 18:51, 'Chris Conroy' via Netty discussions <ne...@googlegroups.com> wrote:

The ByteBuf objects do pin the NIO memory with an unpooled allocator. Are you saying that this is not the case in the pooled allocator?

What you mean here ? In the PooledByteBufAllocator the memory is “pooled” separately from the ByteBuf instance.

With the unpooled allocator, holding on to `ByteBuf` references causes the corresponding NIO memory to be held for the lifetime of the `ByteBuf`. This is why we exhausted our NIO space with the default settings as the recycler held `ByteBuf`s were taking up all available NIO memory. I haven't tested, but it looks like perhaps this is not the case when using pooled allocation since I don't see any `retain` or `release` calls inside the recycler.

 

Object allocation is always very cheap. Garbage collection in the eden space is incredibly cheap, and most buffers are short-lived. I suspect that this may be a premature micro-optimization. I see no difference in JVM pause time or GC run rates when I disable the recycler completely.

In the past I saw issues because of heavy object allocation (even for this short-lived objects). If you not have this issue you could just disable the recycler. 

Yep we can definitely do that.

[Norman Maurer]

On 29 Jul 2016, at 19:14, 'Chris Conroy' via Netty discussions <ne...@googlegroups.com> wrote:

On Fri, Jul 29, 2016 at 12:57 PM, 'Norman Maurer' via Netty discussions <ne...@googlegroups.com> wrote:

On 29 Jul 2016, at 18:51, 'Chris Conroy' via Netty discussions <ne...@googlegroups.com> wrote:

The ByteBuf objects do pin the NIO memory with an unpooled allocator. Are you saying that this is not the case in the pooled allocator?

What you mean here ? In the PooledByteBufAllocator the memory is “pooled” separately from the ByteBuf instance.

With the unpooled allocator, holding on to `ByteBuf` references causes the corresponding NIO memory to be held for the lifetime of the `ByteBuf`. This is why we exhausted our NIO space with the default settings as the recycler held `ByteBuf`s were taking up all available NIO memory. I haven't tested, but it looks like perhaps this is not the case when using pooled allocation since I don't see any `retain` or `release` calls inside the recycler.

Yeah this is different with the PooledByteBufAllocator. I hope I have a fix for the UnpooledByteBufAllocator in the next week. Just was too busy to finish it yet :(

 

Object allocation is always very cheap. Garbage collection in the eden space is incredibly cheap, and most buffers are short-lived. I suspect that this may be a premature micro-optimization. I see no difference in JVM pause time or GC run rates when I disable the recycler completely.

In the past I saw issues because of heavy object allocation (even for this short-lived objects). If you not have this issue you could just disable the recycler. 

Yep we can definitely do that.

--
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDOHrH_TWAGbLHHBCcLrF%3DWLVY8U-L1HCK8nMrXzqbHGA%40mail.gmail.com.

[Chris Conroy]

Configuration by type could be useful, but so far I’m unable to detect any performance degradation when leaving the recycler out altogether. Before adding any more complexity here, I think it would be illuminating to poll the community of Netty 4 users to see what sorts of workloads, if any, impact JVM pause time or GC rates.

Object pooling makes a lot of sense when object setup is expensive. For example, in my Netty based proxy, I pool Channels since connection setup (especially with TLS) is an expensive operation. There are definitely individual circumstances where it may make sense to pool objects without this characteristic, but it’s usually better to just let the JVM handle this. After all, an object pool ends up duplicating the same work that the GC would be doing.

​

--
You received this message because you are subscribed to a topic in the Google Groups "Netty discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netty+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/415234B2-5AED-46CB-AA6E-56C0D34BAFF2%40googlemail.com.

[Norman Maurer]

I fully agree that if you can operate without object pooling its a lot “nicer”. 

Like I said I saw problems without it.. That said not sure if the best is to pool or not pool by default. The problem you are descripted with the UnpooledBytebufAllocator should be fixed without the need to disable recycling all together. Like I said I will have a fix soon.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDa_Kw5TxyrPAU-g8UZwRyrFkKK-nE9weNPV8-hpM4GUg%40mail.gmail.com.

[Leonardo Gomes]

I think this article from Trustin explains a bit why buffer pooling was added in Netty 4:

https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/F98F76E5-19B1-4E48-8135-A541562AC60F%40googlemail.com.

[Chris Conroy]

That blog post touches on the PooledByteBufAllocator which operates at a level below the Recycler object pool.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CAEAdJ9rLwwFWnTBhZebq_%3DpQ%3DrAUSDxEYBDcVeAvmU%3DLyngYMQ%40mail.gmail.com.

​

[Norman Maurer]

Actually it touches on both.. PooledByteBufAllocator uses the Recycler as well for “pooling” the ByteBuf container (not the memory it refer to itself tho).

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKCgew6fKDcAg6W1j8t4x0Ta2TzUtCj%3DfSS7WoU9LegNUA%40mail.gmail.com.

[Dennis Ju]

Is there a ticket number or link for the issue w/ the UnpooledBytebufAllocator?