Memory issues with recycler in 4.1.3.Final

[Julien Viet]

Hi,

we are seing OOM "GC overhead limit exceeded" issue in Vert.x test suite on slow machines like Cloudbees or using virtualization on a laptop.

This issue means that the VM spend more time in GC than the rest of the VM (98%), but I’m pretty sure everyone here knows that already.

In the case of Vert.x testsuite, it happens because we create and destroy many Vertx instances and therefore many event loops / threads during the tests.

I spent quite some time on the issue and I found that it happens because the ThreadDeatchWatcher's Recycler DELAYED_RECYCLE fast thread local is a WeakHashMap<Stack, WeakOrderQueue) used by the ThreadPoolCache grows and retains many entries (up to 2000). This WeakHashMap contains recycled objects with large footprint and also the Stack itself reference its Thread that has a large footprint.

I am not saying it is a leak per se, but the maps grows and takes time to be garbaged. This does not prevent the testsuite to run on a laptop but it slows it down and on a slow machines it prevents running the testsuite entirely.

I believe this behavior was introduced by this commit : https://github.com/netty/netty/commit/afafadd3d7caf1e4b346da049baab0afeae0a4bc

The change that makes the whole difference is : https://github.com/netty/netty/commit/afafadd3d7caf1e4b346da049baab0afeae0a4bc#diff-23eafd00fcd66829f8cce343b26c236aR226

The introduction of the field stack in the WeakOrderQueue keeps a reference on the Stack objects, however Stack instances are also weak keys of the fast thread local WeakHashMap and therefore defeats the purpose of the WeakHashMap. Not entirely as it is GC’ed but much less often and increase the memory footprint.

let me know what you think

Julien

[Norman Maurer]

Yes this sounds not right… So what are the steps to reproduce ? Fire up a vm, checkout vert.x master and run the build ?

> --
> You received this message because you are subscribed to the Google Groups "Netty discussions" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/netty/2A547499-D279-4286-AA95-3447EE2B621E%40julienviet.com.
> For more options, visit https://groups.google.com/d/optout.

[Julien Viet]

Yes with branch netty-4.1.3.Final

> To view this discussion on the web visit https://groups.google.com/d/msgid/netty/D5C167E1-D3C3-4B8F-B4B8-FD4D1793BC1F%40googlemail.com.

[Norman Maurer]

Could you test :

https://github.com/netty/netty/pull/5569

?

> To view this discussion on the web visit https://groups.google.com/d/msgid/netty/14FFE4AD-8D8F-4721-8A1E-C763E1F187DE%40julienviet.com.

[Julien Viet]

Can you deploy a snapshot in sonatype repi with a specific classifier? that would help for testing in cloudbees

> To view this discussion on the web visit https://groups.google.com/d/msgid/netty/BC8153A8-4705-438A-BB3F-A5F099FDEAE8%40googlemail.com.

[Norman Maurer]

Nope :(

> To view this discussion on the web visit https://groups.google.com/d/msgid/netty/EB11694F-03CB-4318-842B-620EF5BFC662%40julienviet.com.

[Alexander Lehmann]

I will try to set up a cloudbees build using the netty pr (should work without snapshot version)

[Norman Maurer]

thanks let me know once you know more….

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/ba8ac728-0a91-438f-bf14-bf1080acc760%40googlegroups.com.

[Julien Viet]

Thanks Alex

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/ba8ac728-0a91-438f-bf14-bf1080acc760%40googlegroups.com.

[julie...@gmail.com]

perhaps you should add a comment saying that no reference to Stack objects should be kept.

[Norman Maurer]

Isn’t this what I did ?

https://github.com/netty/netty/pull/5569/files#diff-23eafd00fcd66829f8cce343b26c236aR238

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/943e0707-fb77-4013-b7c7-14ec42b8e7ac%40googlegroups.com.

[Julien Viet]

I missed it :-)

You received this message because you are subscribed to a topic in the Google Groups "Netty discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netty/wJ-fzhsSMx8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/B99EF8AA-91FB-471C-80ED-5FE51643392C%40googlemail.com.