Setting CMS or G1 as the default GC algorithm for Jenkins?

[Samuel Van Oort]

Hi guys, 

I'd like to propose that we explicitly set the Java args for the Jenkins packages to use either Concurrent Mark Sweep or G1 as the default GC algorithm. 

The reason for this is that Jenkins is generally used as a long-running server process, and large stop-the-world GC pauses can pose a problem for stability and user experience.   The default Java GC algorithm (ParallelGC) is tuned to optimize throughput at the expense of potentially multi-second major GC pauses with large heaps, which is obviously not a good fit for this case. This is why Oracle is moving to G1 as the default as of Java 9.

I would suggest either CMS or G1 make good defaults for normal Jenkins users, because they greatly reduce average and maximum GC pause times.

Thoughts?

Regards,

Sam Van Oort

[nicolas de loof]

+1

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/c009a299-1b23-4cb8-97ef-c6718fc279e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kanstantsin Shautsou]

Applying theory to practise could be not proved without facts. I would suggest collect GC monitoring from jenkins on jenkins (or whatever it called now) and compare. Then it will be absolutely clear what is better.

G1 as default for java9... depends on how much patches and differences are between java8 and java9 and especially oracle vs openjdk.

[Jesse Glick]

On Thu, Oct 6, 2016 at 5:29 PM, Samuel Van Oort <svan...@cloudbees.com> wrote:
> I'd like to propose that we explicitly set the Java args for the Jenkins
> packages to use either Concurrent Mark Sweep or G1 as the default GC
> algorithm.

Fine insofar as that same package is pinning the Java version. But if
the user is selecting a newer JRE than you expected, it would be
better to let the JRE’s own defaults apply.

[Samuel Van Oort]

Kanstanstin, 

> suggest collect GC monitoring 

I'm glad you asked! I actually spent yesterday digging into problems with two high-load, large Jenkins instances which were experiencing issues, and for which we had GC logs available.  In both cases, we observed multi-second pause-the-world full GC cycles with the default parallel GC -- please find attached a screenshot from one of the GC analyses.

> G1 as default for java9... depends on how much patches and differences are between java8 and java9 and especially oracle vs openjdk.

Indeed.  It's pretty contentious for Java 9 as well.

The benchmarks I've seen for G1 on JDK 7 have been consistently poor, so best to avoid it.

On Java 8, where G1 has gotten a lot of optimizations over time, it seems to trade back and forth with CMS if you care about pause time, depending on situation:

http://eivindw.github.io/2016/01/08/comparing-gc-collectors.html

http://blog.dripstat.com/java-g1-gc-review/

CMS is harder to tune, and falls back to longer stop-the-world pauses eventually in some cases, but tends to chew a bit less CPU.  

G1 seems to have an advantage at >4 GB heaps based on benchmarks and tuning advice I've seen (though I haven't run it in a production setting yet). 

I think for *most* users running CI/CD systems with generally-spiky load, generally very low CPU load (or at least some unloaded cores), and multicore systems with 2-8 GB heap *probably* G1 is the best overall solution.  CMS has some advantages if your system tends to run highly loaded (better GC throughput). 

Maybe you've got experiences to contribute though? 

@jglick:

 > if the user is selecting a newer JRE than you expected, it would be better to let the JRE’s own defaults apply. 

Point.  My expectation would be: java 7 --> CMS, Java 8 --> probably G1 (maybe CMS), Java 9+ --> default for platform. 

[Baptiste Mathus]

+1. On the mid-size cluster I worked on in the past we did that quite a long time ago and it did improve the user experience. Clearly less pauses.

(As said on Jira, while doing so I would definitely take the opportunity to also set up gc logs enabled by default (with rotation). There's no noticeable impact on perf even under high load, and this is invaluable information when things go wrong.)

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/11b2a288-119f-4f8c-a87a-b0ce13b8aa26%40googlegroups.com.

[Samuel Van Oort]

Baptiste, I completely agree -- GC logs are a gimme for any production setting IMO (and since it's in the JIRA, it won't be missed).

As a point of reference, I'd reached out to my friends (and former colleagues) at a multi-billion dollar company known for open source, and CMS is still the preference there for production app servers.  It kept request times stable on the business-critical services layer I owned, and prevented errors/hangups due to timeouts from GC pauses.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

[Samuel Van Oort]

For interested parties, this is tracked in https://issues.jenkins-ci.org/browse/JENKINS-38802