Strategies to deal with a blocking garbage collector?

[Oliver Rolle]

Hi community,

I am writing a framework for event processing and I hit a performance barrier because of jvms garbage collector. The problem appears if I have short-lived state-full components. I did some tuning of the gc via command line args but it does not help. The problem is that if too many objects are created while the gc is cleaning up than the gc blocks the whole programm and the perf degenerates from 100k+ events per second to 0 events per second for a few seconds. This is highly problematic, because the processing latency than explodes from a few ms to a couple seconds. This behavior is ok for batch processing but not ok if you want to controll machinery in human-scale real-time. I prevent this behavior if I do not use more than 5% of the cpu, but I waste 95% of my expensive cloud cpu ressources :(

Any idea to solve this perf. degeneration?

* Manual memory management? DirectBuffers, sun.misc.Unsafe
* Object recycling? Use constructor once, reset and reuse object multiple times

PS: If I have a few strategies, I can write a blog article with example code.

[Jordan Halterman]

I think you're on the right track. The way Netty (and lots of other performant systems) deal with garbage collection is via some form of reference counting. Basically, you implement some sort of interface to handle reference counting (perhaps via an AtomicInteger) and an object pool for reusing those objects. Rather than creating a new object, you acquire an object from the pool, increment its references, and set primitive fields on that object. When a thread is done with the object, decrement its references. Once its references have returned to zero, its placed back in the queue.

The way this is done in Hadoop is RecordReader implementations provide instances of their output for each thread. When Hadoop requests the next() record, the instance is passed back in to the RecordReader. The RecordReader sets fields on that instance rather than creating a new instance.

Perhaps in Vert.x 3 you could implement such a pattern of reuse via codecs. Rather than constructing a new object when reading it from the wire, reuse an object from a pool. But pools are really only necessary if your objects are crossing threads. If you know your objects will only be used in one thread, you can reuse a single instance as Hadoop does.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jochen Mader]

Unsafe, ObjectPools ... are all very advanced things one only deals with if you have to.

The GC-numbers are already giving you a good hint:

Excessive Allocation.

Grab a profile (Yourkit, JProfiler, VisualVm,...) and check what objects are being allocated and what part of your program is doing it.

Try to reduce those numbers.

You might also be able to reduce stop time by switching to G1, but the GC can only do so much if you are allocating that many objects.

sun.misc.Unsafe and ObjectPools should always be tools of last resort as they will impact your application design quite a lot. Especially Unsafe is a very dangerous thing, there are no bound-checks and you can easily get into a C-like debugging hell.

I know, both look fancy and like a quick win, but in the end it's far more important to understand your application and the best way to get there is via a profiler.

--

Jochen Mader | Lead IT Consultant

codecentric AG | Elsenheimerstr. 55a | 80687 München | Deutschland
tel: +49 89 215486633 | fax: +49 89 215486699 | mobil: +49 152 51862390
www.codecentric.de | blog.codecentric.de | www.meettheexperts.de | www.more4fi.de

Sitz der Gesellschaft: Düsseldorf | HRB 63043 | Amtsgericht Düsseldorf
Vorstand: Michael Hochgürtel . Mirko Novakovic . Rainer Vehns
Aufsichtsrat: Patric Fedlmeier (Vorsitzender) . Klaus Jäger . Jürgen Schütz

[Tim Yates]

Unsafe is not going to be available in java 9 afaik

[Jordan Halterman]

I think the complications of object pooling are a bit overstated here - it's just another data structure - but it does impose limitations on usage. Object pools and reference counting make memory management more risky; users become responsible for releasing objects back to the pool. If objects aren't released back to the pool, presumably the pool will grow until you run out of memory, so they're just best used for low level internal code.

Also, there are bigger issues with Unsafe that weren't mentioned - compatibility across JVMs. As a demonstration, just look at Netty's PlatformDependent class:

https://github.com/netty/netty/blob/master/common/src/main/java/io/netty/util/internal/PlatformDependent.java

But on a side note, I thought there was talk about making Unsafe a public API in Java 9?

[Jochen Mader]

On Unsafe: I thought there was already a JSR in place but I can't find it ...

On Object pools: I am not against object pools per se, I just see them as a measure applied when everything else has been done. I used them just a handful of times in my career, in all other cases we were able to remove excessive allocation by improving application design.

[Tim Fox]

On 22/05/15 06:30, Jochen Mader wrote:

On Unsafe: I thought there was already a JSR in place but I can't find it ...

On Object pools: I am not against object pools per se, I just see them as a measure applied when everything else has been done. I used them just a handful of times in my career, in all other cases we were able to remove excessive allocation by improving application design.

Indeed. Object pooling may improve things in some situations but its very dependent on the use case.

Bear in mind that in some respects generational garbage collection can be thought of as a (complex) object pool. What makes you think your hand-rolled object pool is going to be faster than the JVMs own one? ;)

[Oliver Rolle]

Thanks for all suggestions! First, I will do some application profiling, looking for low hanging fruits. Maybe later I have to change my architecture to have a faster memory management.