Issues with Vert.x (and Netty) memory usage

[Odd Eirik Nes]

Hi,

I'm currently trying out Vert.x for a small application that records sensor data received from UDP datagrams. Most of it is working very well and I'm very pleased with how easy and powerful the Vert.x event bus is (we use the SockJS/EventBus bridge for communicating with a verticle via the web client). The incoming bytes are pretty much just sanity checked (valid sensor data packets) and then stored to disk in binary format, and we're only talking about ~1 MB/s or so in data, so the maximum memory required for the application shouldn't really have to be very large.

However, there seems to be an issue with memory usage, as the application balloons in size until the JVM runs out of memory and it all crashes and burns.

I enabled heap dump on OOME, and the biggest objects in the heap are all related to Netty:

Class statistics (byte[]is the "culprit" here, as expected):

I've done several heap dumps while running the application as well, and the Netty PoolArena/HeapArena just keeps increasing in size, seemingly without ever decreasing or freeing memory.

So, is this a possible bug in Netty or Vert.x, or is it just a user error and there's this really obvious setting or usage I haven't realized?

Environment:

Win 10 (dev) and Centos 5 (production) - The memory usage is a problem in both environments

JDK/JRE 8u102 64-bit (on both OSes)

Vert.x 3.3.2

[Michael Scholz]

When this happened to me it was a programming error, I was looping a vertx thread that needed to return to the pool to release the mem. I believe logging has since been added to alert such a misuse, have you checked/enabled enough logging?

[Odd Eirik Nes]

It could most definitely be a programming error! :-)

I'll enable (more) Netty logging and see if I can get anything useful from that. Thanks!

[Julien Viet]

Hi,

can you share some simplified version of your program (reproducer) so other can try it ?

> On Aug 1, 2016, at 9:05 PM, Odd Eirik Nes <odde...@gmail.com> wrote:
>
> It could most definitely be a programming error! :-)
>
> I'll enable (more) Netty logging and see if I can get anything useful from that. Thanks!
>

> --
> You received this message because you are subscribed to the Google Groups "vert.x" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
> Visit this group at https://groups.google.com/group/vertx.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/890ac90b-9936-429d-94a1-0677fd089674%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ajay Garg]

I agree with Michael's diagnosis.

In my case, it turned out to be the usage of PooledBytebufAllocator.

Try replacing the occurrences of PooledByteBufAllocator with UnpooledBytebufAllocator in your application-code. It should probably do the trick.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/365A4B7E-9E2E-4F6C-A5AC-8484E9CB4BFB%40julienviet.com.

[Odd Eirik Nes]

I am not using Netty directly, only via Vert.x, so I don't myself use neither PooledByteBufAllocator nor UnpooledByteBufAllocator. I see that there is a usePooledBuffers option for things like NetClient, HttpClient/HttpServer and EventBus, but there doesn't seem to be anything like this for DatagramOptions.

I just got in the office, so I'll enable debug logging for Netty and look into that.

Julien: Not really easy to create a simplified version but the general gist of the application is something like this:

One verticle acts as a web server, serving a simple web interface to the user where he can send a few commands to deploy a verticle and set it up for sensor acquisition. The web interface also makes use of a SockJS/EventBus bridge to receive status from the other verticle (basically whether it's running or not).

The verticle responsible for acquisition is started by the web interface and sets up an UDP socket, listening for incoming packets, verifies them and stores them to disk.

Thanks for the feedback so far!

[Odd Eirik Nes]

Ok, I enabled debug logging for netty and at startup I see this:

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.bytebuf.checkAccessible: true

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 10

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 10

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG io.netty.channel.DefaultChannelId - -Dio.netty.processId: 22084 (auto-detected)

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: pooled

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.threadLocalDirectBufferSize: 65536

2016-08-02 10:14:22 [vert.x-eventloop-thread-1] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.maxThreadLocalCharBufferSize: 16384

So is there some configuration in Vert.x I can set or change to use UnpooledByteBufAllocator instead? And will this also be set for any DatagramSockets I use?

On Tuesday, August 2, 2016 at 8:01:53 AM UTC+2, Ajay Garg wrote:

[Julien Viet]

can you tell what are the exact problem you are seing ?

because you only mention “Issues” in the title and you don’t explain what is going wrong.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/67beb9c6-9f45-4cfa-abc1-d74c964d0016%40googlegroups.com.

[Odd Eirik Nes]

Apologies if it wasn't clear enough what the issue is. As I mentioned in the first post the application will eventually run out of memory, so that is the issue here. Judging by the heap dumps using VisualVM it seems to be because netty just keeps allocating memory (io.netty.buffer.PoolArena$HeapArena and io.netty.buffer.PoolChunks are the largest objects in the heap after running for a short while) until the JVM heap is exhausted. At no point does it seem to want to release or free this allocated memory, so the time it takes for the application to run out is mostly limited by what I set the JVM heap size to be.

The "lifetime" of these incoming datagrams are fairly short; they're checked briefly and then written to disk and that's it. So while there is a little bit of buffering required to handle the incoming datagrams (which is usually limited at ~1 MB/s), the total buffer size required shouldn't have to be much more than maybe some tens of megabytes.

[Odd Eirik Nes]

Well, I may have figured it out (still need to do some long-term testing), and I guess I'm the idiot (that's usually the case anyway ;-) ). As someone above here mentioned: use unpooled buffers. Well, turns out there is a parameter in Netty to do this (which you guys probably knew): -Dio.netty.allocator.type=unpooled.

I ran the application with this argument and memory usage is WAY down and most importantly: it's stable. CPU usage is every so slightly higher (which is because we're doing direct.

Does this change affect Vert.x in any negative way? I guess there's higher CPU usage and allocating ByteBufs is slightly more expensive when they're unpooled? That's shouldn't be a big problem for this application at least.

[Julien Viet]

no you can use this change safely.

however I believe that Vert.x should create unpooled buffer by default and you should not have to set this parameter.

I’ve created this issue : https://github.com/eclipse/vert.x/issues/1554

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f072a63f-a1b7-4da9-91b3-090ab66987a2%40googlegroups.com.

[Odd Eirik Nes]

Great, looking forward to the Vert.x release with this as the default setting! Thanks!