Specs for new Master server
David Brooks
My team currently runs a few Jenkins stand-alone build servers, lots of jobs each, some CI jobs, not a lot of concurrency in builds, but some.
I have been asked to migrate our environment to a master/slave setup so the job management can be run from one server. My questions is this: what kind of hardware would a master-only machine need to be successful? The current stand-alone servers will become the slaves.
So here are the questions I am starting with:
- It seems from what I have read in this and other forums that it is a good idea to have one CPU core per executor. Is this true for the master as well as the server doing the builds? Or can we scale back?
- I have also read that the master can become I/O bound while moving the logs and build results around. (I think I have that right but some threads suggest that Jenkins can become CPU bound as a result of being I/O bound, an interaction that I don't fully understand.) The network backbone isn't an issue but I can spec out a variety of different storage solutions. I would obviously like to avoid spending money on storage we don't need but I need to build something that will function.
- How much RAM should this master-only machine have?
- All of our servers are virtual. That helps in that they can be rescaled if needed, but are there any special considerations that a virtual master introduces that I should know about?
- Is there anything that I should be thinking about when converting the stand-alone servers to slaves? I have already modeled the process of getting a Jenkins server to be both a stand-alone server and a slave so that I can do the migration without bringing the servers down. That works fine. But is there anything about being a slave server that would change the system requirements?
Thanks for any suggestions!
~ Dave
Ginga, Dick
Dave, I am not as informed as others in this group, but my experience has been that the master does very little work as compared to the slaves. Yes, build logs and archived files are uploaded from a slave to the master, but I have not run into any issues with IO performance.
I run a Windows master as a VM running Server 2008 R2 Enterprise, 16 Gb RAM, 4 CPUs, and a 250G disk. I do have to monitor the disk usage by setting build retention values and sizes of artifacts archived.
We run 22 slaves (Mac and Windows) and perhaps another 20 on again, off again slaves and, I don’t know, 100’s of jobs. You will need to set executors numbers on your new slave build machines to match their capacity.
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Rob Mandeville
If you move most/all of the actual jobs off to slave nodes on other machines, then the Jenkins server host needs to:
· Interact with the user (minimal hardware requirements if your users don’t use auto-refresh; auto-refresh could greatly increase CPU needs)
· Retrieve data, artifacts, and logs from the slave machines (needs good network connectivity and large, fast storage capacity)
· Hold data for every build (but not artifacts and logs) in core (requires great gobs of memory)
You certainly don’t need one core per slave executor on the master, unless you expect all your jobs to be “chatty” all the time. Generally, capturing the data, logs, and artifacts from a build requires one or two orders of magnitude less CPU time than actually running the build. You’re likely to get bigger speed gains by increasing the speed of your disk than that of your CPU.
Every build that your Jenkins server “remembers” will have its data stored on disk and in memory, and its artifacts and logs will remain on disk until and unless requested by a user or another job. If this was the whole story, then memory and disk needs would increase without bounds. However, you can set jobs to only remember a certain number of builds and/or a certain number of days of builds. You can separately limit how many logs and artifacts it will keep. Use this feature judiciously, preferably on every job you define. If you have a need to store results “since the beginning of time”, you have a big problem. At a previous job, I had that and had to parse my build logs and write the data required to a database, only using Jenkins to run builds and not to analyze the results.
I currently run Jenkins servers and clients on VMs, and don’t have any problems with them. This might make it easier to do whole-host backups of your Jenkins system (if you have the space for that) for disaster recovery.
Any combined master/slave machine can be converted to a slave-only machine without more hardware; you might even be able to increase the number of executors. In terms of server CPU and network use, adding executors to an existing node is slightly less expensive than adding nodes.
Finally, you have said that you are running a small number of Jenkins standalone servers. I would suspect that a master-only machine needs no more CPU or disk than a standalone host, and may just need more disk capacity. You should be able to sacrifice one standalone system to serve as the master, and the rest of them can be converted to slave-only hosts.
From: jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com]
On Behalf Of David Brooks
Sent: Wednesday, November 12, 2014 8:44 AM
To: jenkins...@googlegroups.com
Subject: Specs for new Master server
I have searched the archives and don't see a direct answer to this question. I apologize if this has already been asked and answered.
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Click here to report this email as spam.
This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.
Thank you.
Please consider the environment before printing this email.
David Brooks
Excellent information, exactly what I was hoping to see.
For the RAM requirements, would it be safe to estimate the required RAM on the master by looking at the combined utilization of the build servers? For instance, I see that the Jenkins and Java processes combined are using about 8GB of ram on one of our build servers. Would this this translate directly to the master utilization? So just add up the utilization for the build servers and that is the baseline for the master?
If so then this machine is going to need TONS of ram. Or I am going to have to start pruning how much Jenkins 'remembers' about builds. That may not go over very well with the teams.
Thanks,
~ Dave
Les Mikesell
> Hi Rob, rginga -
>
> Excellent information, exactly what I was hoping to see.
>
> For the RAM requirements, would it be safe to estimate the required RAM on
> the master by looking at the combined utilization of the build servers? For
> instance, I see that the Jenkins and Java processes combined are using about
> 8GB of ram on one of our build servers. Would this this translate directly
> to the master utilization? So just add up the utilization for the build
> servers and that is the baseline for the master?
>
> If so then this machine is going to need TONS of ram. Or I am going to have
> to start pruning how much Jenkins 'remembers' about builds. That may not go
> over very well with the teams.
the number of builds retained for each, but won't be equivalent to
adding the whole JVM usage across multiple machines. But if you are
using VMs you shouldn't have to be that precise in your initial setup
as long as you don't overcommit the physical host. I'd just pick one
of the existing servers as the master, move the jobs there and connect
the others as slaves. You can do that piecemeal and adjust resources
if you see a problem. And I'd recommend not running any jobs on the
master itself, just on general principles.
--
Les Mikesell
lesmi...@gmail.com
Rob Mandeville
Also (and you probably know this) you may have to explicitly set the memory settings on the JVM as well as the VM; Java won't take all your memory by default, and you might want to leave a GB or so free so that the OS can do its OS-y things. Basically, if the JVM gives an out of memory complaint, think about the JVM settings first--adding 64GB to your host won't help if Java is only taking 4GB of it.
--Rob
Les Mikesell
<rmand...@dekaresearch.com> wrote:
> There are some hairy reasons for running a job on the master, specifically if you need to mess with Jenkins internals. If you don't need to do that (and if you have to ask, you don't), just reduce the number of executors on the master node to zero and see what breaks ;^>.
'system' groovy script could do that from a slave anyway. Is that not
the case? I did at one point set up an executor on the master with
the intent of forcing build_flow jobs there but haven't used it yet.
All of our jobs are restricted by label so nothing runs there.
> Also (and you probably know this) you may have to explicitly set the memory settings on the JVM as well as the VM; Java won't take all your memory by default, and you might want to leave a GB or so free so that the OS can do its OS-y things. Basically, if the JVM gives an out of memory complaint, think about the JVM settings first--adding 64GB to your host won't help if Java is only taking 4GB of it.
>
processes. Mine shows 2.3g as RES for jenkins but it also shows 4.9g
as SWAP for the process while at the same time showing 0 Swap used in
the global part. Does that mean it has memmap'd some other file but
not paged it in - or is it in the 'casched' portion of swap?
--
Les Mikesell
lesmi...@gmail.com