Jenkins with Parallel Builds?

413 views
Skip to first unread message

David Aldrich

unread,
Jan 15, 2020, 9:38:47 AM1/15/20
to jenkins...@googlegroups.com
Hi

We build large C++ projects using Jenkins for Linux platforms.  These build jobs currently run on a single executor.  I want 'make -j' or ninja to compile concurrently on several cores to speed up the build time.  How can I cause a job to utilise multiple executors on an agent in this way?

I saw a similar question discussed in a StackOverflow question: https://stackoverflow.com/questions/20003919/jenkins-with-parallel-builds. The answer given there was to use the Heavy Job Plugin. But that plugin seems quite old. Is it still the best solution?

Best regards
David

Dirk Heinrichs

unread,
Jan 15, 2020, 9:48:06 AM1/15/20
to jenkins...@googlegroups.com
Am Mittwoch, den 15.01.2020, 14:38 +0000 schrieb David Aldrich:

We build large C++ projects using Jenkins for Linux platforms.  These build jobs currently run on a single executor.  I want 'make -j' or ninja to compile concurrently on several cores to speed up the build time.  How can I cause a job to utilise multiple executors on an agent in this way?

These are two completely different things. The number of executers defines how man Jenkins jobs an agent can run in parallel. If you want to run your build (as executed by ONE job) as a parallel build, then you just need to specify the level of concurrency inside that single job, just like you would do if it was executed manually from within a shell (don't know whether Jenkins has a "make" plugin).

This is also how we do it. All build nodes only have one Jenkins executor to ensure we only run one job at a time on any given node. But the builds executed by these jobs can then utilize all the CPU cores in that node.

HTH...

Dirk
-- 
Dirk Heinrichs
Senior Systems Engineer, Delivery Pipeline
OpenText ™ Discovery | Recommind
Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach
Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, Registernummer HRB 10646
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.

Simon Richter

unread,
Jan 15, 2020, 9:56:05 AM1/15/20
to jenkins...@googlegroups.com
Hi,

On Wed, Jan 15, 2020 at 02:38:26PM +0000, David Aldrich wrote:

> We build large C++ projects using Jenkins for Linux platforms. These build
> jobs currently run on a single executor. I want 'make -j' or ninja to
> compile concurrently on several cores to speed up the build time. How can
> I cause a job to utilise multiple executors on an agent in this way?

If you want to go for a 1:1 executor to CPU mapping, then using -j inside
the build and combining it with the Heavy Job plugin to allocate more CPUs
is indeed the way to go.

The downside is that it will wait until that many executors are available,
and IIRC there is an anti-starvation mechanism that also blocks other jobs
from executing on the chosen node.

Depending on your setup, you might as well have the best result with a
setup like "four executors, and all jobs use "-j$((($(nproc)+3)/4)" or
something similar.

For the most flexible result, I suggest defining an environment variable in
the agent config that defines the -j option to be used on that node, then
you have both the number of executors and the number of threads per
executor in the agent config, and if you see that it doesn't work
optimally, you can tune there.

Two or four CPUs per executor are good if you have many builds, while two
or four executors with half/quarter of the CPUs each are good if you build
a massive project seldom.

The Heavy Jobs plugin is for mixing strategies -- e.g. you'd define a -j4
job for incremental builds that run quickly either way except if there is a
big change, and a heavyweight job that takes over half the node for the
daily full rebuild/warning generation/... task.

Simon
Reply all
Reply to author
Forward
0 new messages