Multiple Instances on one server with Java 11

[Matt Wilson]

For a few years I've been running multiple (independent) Jenkins instances on one server.  Each server runs under its own apache instance.

SiteA

SiteB

SiteC

This has worked perfectly fine for a few years with no problems.

Last week I upgraded all three servers to 2.346.1.  two of the three servers had been updated regularly so this wasn't a huge jump, that third server has a year and a bit behind so it was a bigger jump (siteB).

All three sites upgraded fine.  

All three sites got upgraded to Java 11.  That went fine.

All three sites run independently with no problems.

When I start all three sites, I have one site that crashes.  The crash seems to be triggered in particular when you access the management page on one particular server.

so like this

Site A is fine no matter what

Site B crashes when site C accesses its manage Jenkins page

I've double checked my settings, I can't find any port conflicts or Jenkins home conflicts.

I've tried running all three sites on separate Java installs.  Still crashes

The only thing that seems to stop the crash so far is if I run site C back on Java 8.

Any thoughts on what I could be missing here config wise?

[Alex Earl]

Is there any crash log in either the apache or jenkins logs? I would look for exception dumps in the logs, it might help narrow down where the issue is occurring.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/de1049d2-cbe0-462e-9bce-1f9656ba6018n%40googlegroups.com.

--

Website: http://earl-of-code.com

[Matt Wilson]

Sorry, should have mentioned that.

This is running on RHEL 7, and there are no indications of anything in the logs.  Just an abrupt stop.  I could probably turn on some more verbose logging though.

[Ivan Fernandez Calvo]

If there are three different Apache Tomcat instances running three different services on Java 11, the only explanation to make one crash that comes to my mind is the resources of the machine, memory in particular.

The recommendation is to not use more than 75% of the memory of the host to the JVM, in your case you have 3 JVM running so each should not use more than 25% of the memory of the host, if you use more than that the JVM will fight for the host resources and one will lose crashing. Check the values on your `xms` and `xmx` Apache Tomcat parameters.

[Matt Wilson]

ooo... you may have nailed it there.  I was definitely over  the 75%.  I bumped up my overall system RAM, and then locked in the one site that isn't used too much to a lower value.  Its early, but so far no crashes.  Interesting that Java 8 never had an issue with this, but 11 clearly is.

Hopefully this holds as a solution.  Thanks for the tip!

[Iván Fernández Calvo]

If you issue is the memory, it is time to check the garbage collector you are using, check that you are not using deprecated JVM flags, IIRC the garbage collector used by JDK 11 is G1 and should perform better than default JDK8 defaults.

Regards

Ivan Fernandez Calvo

El 11 jul 2022, a las 19:48, Matt Wilson <mwil...@gmail.com> escribió:

ooo... you may have nailed it there.  I was definitely over  the 75%.  I bumped up my overall system RAM, and then locked in the one site that isn't used too much to a lower value.  Its early, but so far no crashes.  Interesting that Java 8 never had an issue with this, but 11 clearly is.

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/mwvN_V-_C0o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ff1e0be4-a93c-48fd-aacd-3df27b92cb97n%40googlegroups.com.