Monitoring direct memory allocation.

[Luke Daley]

Hi all, has anyone found a good way to manage the amount of direct memory that Netty has claimed?

I've set up measurement according https://blogs.oracle.com/alanb/entry/monitoring_direct_buffers - but the numbers don't seem right, and my reading of the code seems to indicate that Netty doesn't alloc via a path that is measured by this stuff.

AFAICT, that method only applies to java.nio.ByteBuffer.allocateDirect() usage as the values are based on counters in java.nio.Bits which Netty does not use to allocate.

The best lead I have found is PooledByteBufAllocator.directArenas() which provides PoolArenaMetric instances. However, I’m not sure how to use this (if indeed it is the right thing to be using) to determine how much memory Netty has indeed claimed for direct buffers. 

I’m operating in a memory constrained environment and need to understand what memory is being used.

Thanks in advance for any assistance.

LD.

[Leonardo Gomes]

I haven't tried JMC, but maybe it could help (if you're running Oracle JDK 8): http://www.oracle.com/technetwork/java/javaseproducts/mission-control/java-mission-control-1998576.html

Or maybe visualgc http://www.oracle.com/technetwork/java/visualgc-136680.html

Cheers,

Leo.

--
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/etPan.56de0f4a.1f9e1dda.80b1%40Trajan.local.
For more options, visit https://groups.google.com/d/optout.

[Norman Maurer]

Netty uses ByteBuffer.allocateDirect() so it should work

--

[Ian Barfield]

Are you sure Netty doesn't use Bits? I know there's eventually some internalish class or other with a long that gets updated. However, that would be all off heap memory -- not just Netty. Those metric instances sound like good bets unless they contain no useful looking methods. I'd poke around those.

I'll check the Bits thing again and reply of it is some other class.

--

[Norman Maurer]

Netty is using Bits :)

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CAGGLr%3D_%3Dc60Eze28RVVCiAPW97MU82BUQQRzVrj8Tu3rTp7A9g%40mail.gmail.com.

[Ian Barfield]

Thanks, Norman. I let that promise get away from me for a few days. I was pretty sure though cause had traced OOM errors that don't trip the OnOOM flag through Netty back to there once a long while ago.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/28D2150C-2183-42BA-818E-611FEA9F6078%40googlemail.com.

[Luke Daley]

Hi,

Some further digging revealed that you're all indeed correct; that counter does include Netty's allocations.

What tripped me up was that the memory is being free'd far more aggressively than I expected. The monitoring solution I was using had a 10 second granularity, and the entire 2MB request was being consumed and free'd between samples. 

I am using the pooled allocator here. Does anyone have any explanation as to why the direct memory allocation count shrinks so rapidly after the buffer is released? What should I go looking at?

[Ian Barfield]

I believe the pooled allocator only does a substantial amount of freeing (to the OS) when the pool (or sub components) are over their max configuration (adjusting various sub page breakdowns may help avoid that). In any event, Netty is probably using the majority of your off heap memory? There is a jmx viewable metric for that. Or just sample more frequently?

--

You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/d19a5756-42e1-424a-a05e-2f033708ceb7%40googlegroups.com.