Akka 2.4+, Java 8 low latency micro-services JVM GC options

[Guido Medina]

Hi,

I'm wondering if anyone has dealt with the pain of tuning the JVM 8 GC for Akka micro-services where the heap size isn't more than 1GB per JVM, here is what I have come up with:

-server
-Xms256m
-Xmx256m
-XX:NewSize=64m
-XX:MaxNewSize=64m
-XX:MaxTenuringThreshold=2
-XX:+UseConcMarkSweepGC
-XX:+ParallelRefProcEnabled
-XX:+CMSClassUnloadingEnabled
-XX:+CMSParallelRemarkEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=80

That's low latency and many small messages pattern, my micro-services will vary from heap sizes of 256m, 512m, 768m and 1g.

Any suggestion or discuss further will be very welcome, Akka team should know better the GC pattern of using akka-cluster, etc, but I don't think that is going to be too relevant considering the amount of the messages the application itself will generate.

Note: Some old JVM options for Java 8 are not default to true or what should be obvious like -XX:+UseNewParGC for your generation.

Regards,

Guido.

[Guido Medina]

Typos correction, I'm a bit sleepy sorry:

Note: Some old JVM options for Java 8 are now default to true or what should be obvious like -XX:+UseNewParGC for young generation.

[Konstantin Shaposhnikov]

Out of curiosity what are the average and max GC pause times you observing with these settings?

[Guido Medina]

I started testing today so I still have no data, I barely finished tuning the parameters base on research, but facts still zero, nada :(

[Guido Medina]

My new G1GC parameters:

-server
-Xms256m
-Xmx256m
-XX:+AlwaysPreTouch
-XX:+UseG1GC
-XX:MaxGCPauseMillis=10
-XX:+PerfDisableSharedMem
-XX:+ParallelRefProcEnabled

I found an interesting article about a major Linux related GC bug which is causing long pauses:

http://www.evanjones.ca/jvm-mmap-pause.html

PerfDisableSharedMem can be used with G1GC and CMS but your VisualVM won't be able to see these Java processes but that's OK for production, you will have to explicitly bind JMX to monitor the JVM.

Enjoy,

Guido.

On Monday, February 1, 2016 at 7:11:09 PM UTC, Konstantin Shaposhnikov wrote:

[Guido Medina]

For what I could gather and after 2012 bug fixed on the Linux kernel for NUMA it is safe and profitable to use NUMA, though I'm not fully aware if G1GC is NUMA-aware, at least JVM is not complaining about it:

• http://docs.oracle.com/javase/8/docs/technotes/guides/vm/performance-enhancements-7.html

I also learned that Eden and Young Generation for G1GC are adjusted by the max pause in millis target hence I got mine to a low value to avoid full GC.

Some options are on by default specially in Java 8 like TieredCompilation if you specify -server, etc.

Remember the GC I/O bug (http://www.evanjones.ca/jvm-mmap-pause.html) while writing stats to /tmp/hsperfdata_${user} which it can be resolved with either of the following options:

• Map /tmp to RAM
• Disable writing of stats which is OK for production but tools like VisualVM won't automatically discover your JVM.

New fill listing of GC settings:

-server
-Xms256m
-Xmx256m
-XX:+UseNUMA
-XX:+UseG1GC
-XX:+AlwaysPreTouch

-XX:MaxGCPauseMillis=10
-XX:+PerfDisableSharedMem
-XX:+ParallelRefProcEnabled

Enjoy,

Guido.

[Viktor Klang]

Do you have a jmh bench for testing this?

--
Cheers,
√

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Guido Medina]

This is not something easily testable with JMH as you don't measure performance but consistent performance, I have the application I'm working on logging times and I what I expect is to not see GC hiccups or longer pauses that you can clearly see the average time jump because of a full GC pause.

[Guido Medina]

It looks like for Java 8u72 G1GC is pretty much fixed, again I'm using it for low heap/latency micro-services, each micro-service won't exceed 1GB heap size.

I had to remove the MaxGCPauseMillis=10 because was making GC too costly slowing down my application and causing some 10+ ms pauses, and after; I haven't seen a pause for the last 24 hours so far for a 256m heap.

I read that some people configure MaxGCPauseMillis between 200 and 500 target which as side effect extends the Young and Eden making copying less frequent; again I left at its default of 200ms.

I even tried -XX:+AggressiveOpts and -XX:UseFastAccessorMethods but that was also slowing down the application, weird; so here is what I think will be my final setup for low heap/latency micro-services:

-server
-Xms256m (or 512m or 768m or 1g depending on your HEAP needs)
-Xmx256m (make it always equal to Xms for better GC)

-XX:+UseNUMA
-XX:+UseG1GC
-XX:+AlwaysPreTouch

-XX:+PerfDisableSharedMem
-XX:+ParallelRefProcEnabled

Enjoy,

Guido.

[Guido Medina]

Tests on Solr with JDK 8u72 corroborates my statement of G1GC looking good:

https://wiki.apache.org/solr/ShawnHeisey

[Patrik Nordwall]

Thanks a lot for sharing, Guido!

--

[Guido Medina]

No problem, I bother you guys a lot, I can only contribute from time to time in order to payback somehow to you and the the community, specially the painful task of tuning a JVM...