Jenkins running large multi-CPU memory-intensive jobs

[mike crowley]

Hi,

I'm trying to use Jenkins to automate some hardware build and sim jobs. One of these jobs is to build an FPGA. I use Jenkins to call an FPGA build tool. The FPGA build tool runs processes in parallel (~10 jobs each hogging a CPU and ~10GB of RAM) in order to speedup build time. When I run the tool standalone it runs in ~10 hours. On Jenkins it can run for >24 hours (which is no use, I need this to run overnight). If I only have 1 run it runs OK in 10 hours.

When I look at what is going on it seems that some of the "parallel" jobs stall. There is a timeout (within the FPGA build tool - TCL internal to the tool) which isn't getting hit so it looks like Jenkins somehow stalls these jobs.

I'm running the job on the same server that the Jenkins instance is running on.

Does Jenkins somehow throttle jobs that consume large amount of memory? I mean is there some limit that I can change within Jenkins?

There is one thing within the parallel builds in that when 1 finishes the stdout from the FPGA build tool switches from 1 of the parallel builds to another. There is also quite a lot of console output 10MB or so.

Any help/thoughts appreciated,

Mike.

[Artur Szostak]

What size machine are you trying to run this on?
From your description if you are trying to run 10 jobs each using 1 CPU and 10GB of RAM you need a machine that has > 10 cores and > 100GB of RAM. If your machine too small then you will likely experience thrashing and stalls.
Also, default setups of Jenkins have scalability issues. For scalable Jenkins, never run the builds on the master, only on build slaves (unless you have carefully calculated and considered the required resources)

---

From: jenkins...@googlegroups.com [jenkins...@googlegroups.com] on behalf of mike crowley [michael....@gmail.com]
Sent: 03 March 2017 11:03
To: “jenkins...@googlegroups.com”
Subject: Jenkins running large multi-CPU memory-intensive jobs

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAK2oGcrmAsW%2BUe1MJk%3Dk6PcisUMNYuaWSuWUVZCTYgX0R7EH0Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[mike crowley]

Hi Artur,

Thanks for reply.

The machine is a server with 256G RAM and 48 cores I think. The FPGA build application runs standalone fine so I don't think it's a machine problem.

I tried running on a slave initially (with similiar large specs) but this also ran slowly.

The first 5 or so builds of 10 in parallel seems to work fine, the other 5 don't seem to be "getting in" (as they don't timeout).

I'm wondering is it something to do with logging that after a whole the logging slows things down. The log file is ~10MB.

Mike.

On Friday, March 3, 2017 at 11:56:34 AM UTC, Artur Szostak wrote:

What size machine are you trying to run this on?
From your description if you are trying to run 10 jobs each using 1 CPU and 10GB of RAM you need a machine that has > 10 cores and > 100GB of RAM. If your machine too small then you will likely experience thrashing and stalls.
Also, default setups of Jenkins have scalability issues. For scalable Jenkins, never run the builds on the master, only on build slaves (unless you have carefully calculated and considered the required resources)

---

From: jenkins...@googlegroups.com [jenkins...@googlegroups.com] on behalf of mike crowley [michael....@gmail.com]
Sent: 03 March 2017 11:03
To: “jenkins...@googlegroups.com”
Subject: Jenkins running large multi-CPU memory-intensive jobs

Hi,

I'm trying to use Jenkins to automate some hardware build and sim jobs. One of these jobs is to build an FPGA. I use Jenkins to call an FPGA build tool. The FPGA build tool runs processes in parallel (~10 jobs each hogging a CPU and ~10GB of RAM) in order to speedup build time. When I run the tool standalone it runs in ~10 hours. On Jenkins it can run for >24 hours (which is no use, I need this to run overnight). If I only have 1 run it runs OK in 10 hours.

When I look at what is going on it seems that some of the "parallel" jobs stall. There is a timeout (within the FPGA build tool - TCL internal to the tool) which isn't getting hit so it looks like Jenkins somehow stalls these jobs.

I'm running the job on the same server that the Jenkins instance is running on.

Does Jenkins somehow throttle jobs that consume large amount of memory? I mean is there some limit that I can change within Jenkins?

There is one thing within the parallel builds in that when 1 finishes the stdout from the FPGA build tool switches from 1 of the parallel builds to another. There is also quite a lot of console output 10MB or so.

Any help/thoughts appreciated,

Mike.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.