Master + slave + job JVM memory tuning...

[Jeff]

I'm trying to optimize how much memory I allocate to the master application (aka - tomcat), slave processes, maven tasks, etc. and going crazy.

I've been unsuccessful in finding a guideline to determine how much memory I ought to allocate or if I should just let the defaults work.

All machines are single processor, dual core with 4 GB ram.  

I've typically set tomcat to use 2048MB + 512MB permgen space and set my maven process to use 384MB max.  Is this reasonable?

My slaves are set to run 4 executors.  Jobs use either svn or git to check out code and Maven to build with a SONAR post-build analysis.  We also do Maven releases at times.

How much does the core slave process need?  Is there a "rule-of-thumb" based on the number of executors?

Any perspective/considerations/resources/links/pointers are much appreciated.

--

Jeff Vincent
preda...@gmail.com
See my LinkedIn profile at:
http://www.linkedin.com/in/rjeffreyvincent

I ♥ DropBox !! 

[Nord, James]

Have enough RAM on the slave that you can run your jobs plus Jenkins + some for overhead.

(e.g. 1GB of Jenkins slave, + #executors * ( 2GB for Maven +2GB for unit tests in forked VM) + OS overhead)

 Although this amount is mostly an overkill setting it too close to the threshold will slow your jobs down due to paging at best, or mean you get intermittent failures due to OOM at worst – (memory is generally cheaper that human time taken to investigate issues!)  (our maven jobs can be very memory hungry so your figures will vary)  (Note I recommend a single executor per slave)

 

The master depends more on how many jobs / reports per jobs / job history you have etc – we use 10GB (but not tomcat).

 

If you’re in the UK and there is any spaces left I will cover some things like this next week at the Jenkins User Event.

 

/James

---

**************************************************************************************
This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postm...@nds.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.

NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************

[Jeff]

Thanks...I unfortunately can't be in the UK for the event. 

So if the amount of RAM is fixed (4GB), what is a good rule of thumb for determining what to allocate to the slave process, how many executors it can reasonably support, etc? 

I realize there are a lot of other variables but any starting point and things to consider or factor in that may not be obvious could help.

Or is it just a matter or monitoring and tuning?

[teilo]

I would never advise more than one executor per slave :-o

 At some point there will be unit test failures in a project as it has badly written unit tests that blindly assume they can use port 12345 (and it will be in use for a different job - and then fight each other).  And it will be the CI systems fault and come at a time that the most awkward for you (the project is trying to release, and you are about to leave for the weekend!)

The exception to this would be where the jenkins job is acting more as a controller and the work is being performed on some other system.

/James

[Johno Crawford]

As James mentioned it really depends on how many jobs you have and what you are actually building, project size etc. Off the top of my head here are some tips:

If you do not require more than 2GB on Windows or 3GB on Linux I recommend using a 32-bit JVM (64-bit JVM uses double the space for every pointer), if 32-bit JVM is not an option you may be interested in investigating the CompressedOops JVM switch [1].

Enable the com.sun.management.jmxremote JVM switch and connect to your JVMs with VisualVM [2]. Doing so will allow you to monitor how much heap and perm gen your JVM is occupying, from this it should be straight forward to tune the values.

Cheers,

Johno

[1] https://wikis.oracle.com/display/HotSpotInternals/CompressedOops
[2] http://visualvm.java.net/

-- 
:: Johnathan Crawford
:: Senior Software Developer

Sulake | Porkkalankatu 1 | 00180 Helsinki, Finland
http://www.sulake.com | tel +358 10 656 7000 | fax +358 10 656 7010