Question about Maven "Build modules in parallel" setting

[David Alayachew]

Hello Jenkins Team,

I have a GitHub repo with a Maven project with over 50 sub-modules, all in the same repo. All of those sub-modules are under a single aggregate pom file.

I created a single Maven job pointed to the aggregate pom, and on the first run, all sub-modules were discovered as expected.

Now, I want these jobs to build in parallel. So, I checked the checkbox that says "Build modules in parallel".

But when I looked at the nodes used, all of the builds occurred on a single node, even though there are many other nodes available. And to be clear, it was using all executors of that node, but still limiting itself to a single node.

So, that meant that, even though the queue had 40+ jobs queued up, once all the executors on that single node were occupied, it didn't matter if any of the other nodes were free, no work would be picked up by them.

On a second attempt, a different node was used, but the same behaviour -- all sub-modules only executed on this second node.

Is this behaviour overridable? Or am I potentially doing things wrong? Again, I am only using the Jenkins Maven Job, not a Jenkinsfile or anything like that.

And as a workaround, I could just make a new Maven Job for each Maven sub-module. Doing it this way, now I do actually run on many free nodes.

But that has the downside of increasing the maintenance burden immensely. Remember, I have over 50 sub-modules. If I need to add new functionality, that's 50 Jenkins jobs that I need to update. Not ideal at all.

Thank you for your time and help.

[muni bhasker]

Hi 

You have slack group so I cloud join.

Thank you 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/jenkinsci-users/CAA9v-_M1Ex2E%2Bn2MGwH6RfT3ek0T23QX6VyQKcmRfbUhJ6N79w%40mail.gmail.com.

[David Alayachew]

Hello,

I do not understand your question. What do you mean by Slack group? I do not know of any Slack group, so if there is one, I cannot grant you access to it.

If you are asking about having the ability to join this google groups discussion, it appears that you have already successfully done so.

To view this discussion visit https://groups.google.com/d/msgid/jenkinsci-users/CAO54Onviqjzw__p6MnVw7yG6LF0khG7QONxbYRFRCYdYvh_Hfw%40mail.gmail.com.

[David Alayachew]

Hello Jenkins Team,

I still have not heard back regarding my query last week about the Maven Jenkins build options. Could someone take a look and respond?

Thank you for your time and consideration.

David Alayachew

[Mark Waite]

On Tue, 23 Sep 2025 at 7:05 PM, David Alayachew wrote:

Hello Jenkins Team,

I have a GitHub repo with a Maven project with over 50 sub-modules, all in the same repo. All of those sub-modules are under a single aggregate pom file.

I created a single Maven job pointed to the aggregate pom, and on the first run, all sub-modules were discovered as expected.

 

Now, I want these jobs to build in parallel. So, I checked the checkbox that says "Build modules in parallel".

But when I looked at the nodes used, all of the builds occurred on a single node, even though there are many other nodes available. And to be clear, it was using all executors of that node, but still limiting itself to a single node.

So, that meant that, even though the queue had 40+ jobs queued up, once all the executors on that single node were occupied, it didn't matter if any of the other nodes were free, no work would be picked up by them.

On a second attempt, a different node was used, but the same behaviour -- all sub-modules only executed on this second node.

Is this behaviour overridable?

No, that is not overridable as far as I know.

 

Or am I potentially doing things wrong? Again, I am only using the Jenkins Maven Job, not a Jenkinsfile or anything like that.

I think it is a mistake to use the Maven job type in Jenkins.  Use a Pipeline.  It gives you more precise control and allows you to choose techniques that better suit your specific needs.

Refer to the "Risks" section of the Maven job type documentation for a detailed description why you should use a Pipeline job instead of the Maven job type.

 

And as a workaround, I could just make a new Maven Job for each Maven sub-module. Doing it this way, now I do actually run on many free nodes.

But that has the downside of increasing the maintenance burden immensely. Remember, I have over 50 sub-modules. If I need to add new functionality, that's 50 Jenkins jobs that I need to update. Not ideal at all.

 

Yes, if you need fine-grained control of job parallelism and then you'll need to manage that fine-grained control.

Mark Waite 

[David Alayachew]

Thanks for the response Mark.

Yes, me and my whole team had a nice sitdown going through the exact Risks section you are talking about, making sure that this is what we want to do. After a fairly long back and forth, we felt that the risks were worth putting up with in order to get the benefits.

As for the Jenkinsfile, that is the last resort. I have to clarify, the example I gave, of 50 sub-modules, was a severe understatement. We actually have nearly 200 modules total, each with their own specific build logic. A big part of the reason why we still wanted the Maven Build Job even in spite of the scary failures is because the Maven Build Job is documented to handle each of our use cases out-of-the-box. This one about the nodes being used is literally the only thing that is not running as expected. A Jenkinsfile, in comparison, will take significantly more effort. And while we're willing to if needed, this first pathway was enticing enough that it's worth considering for us.

But based on what you are saying, it kind of looks like we are going to be making a Jenkinsfile. As a last ditch effort, I see links to the Jira at the top of the link you shared. Assuming that I don't find any linked issues that are already describe our feature request, do you think they would be fine if I submitted one?

Thanks again for your time, help, and speedy response.

David Alayachew

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/jenkinsci-users/b59c26a3-c445-4759-8016-4515c13b3803n%40googlegroups.com.

[David Alayachew]

And to be clear, I am more than happy to make the change myself and offer a complete pull request.

I am moreso making sure that a request and offer like this would even be welcome to the team. It seems yes, but while I have you here, I figured I'd ask.

[Mark Waite]

On Wednesday, October 1, 2025 at 6:30:28 AM UTC-6 David wrote:

And to be clear, I am more than happy to make the change myself and offer a complete pull request.

If you're willing to do the work yourself, then you can immediately use the result of that work on your installation, whether or not it is ever included in a release of the Maven plugin.

 

I am more so making sure that a request and offer like this would even be welcome to the team. It seems yes, but while I have you here, I figured I'd ask.

The Maven integration plugin is not being actively enhanced.  I doubt that your pull request will be merged, but even if it is not merged, you can use the incremental build of your pull request and allow others to have access to that incremental build as well.

Mark Waite

[David Alayachew]

Thank you very much!

I just scoured all the way to page 16 out of 20 (chronologically, so the remaining tickets are <=2011), and the only one that even appeared relevant was https://issues.jenkins.io/browse/JENKINS-11825?jql=Component%3D16033.

I'll go ahead and start looking into making the change myself, submit the pr, and then see if the incremental build is good enough to share. Best case scenario, it is, and we can merge it in so that everyone who uses the Maven Plugin will be able to benefit.

I'll post my PR to this thread if/once I get it done.

Thanks again for your time and help!

David Alayachew

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/jenkinsci-users/2344e980-b5c9-4e5a-88b1-b255f56fcbdcn%40googlegroups.com.

[David Alayachew]

Thanks again for the help Mark.

Here is my Feature Request -- https://issues.jenkins.io/browse/JENKINS-76168

I will now start working on the PR.

I will read through this thoroughly, then reference it when building my PR -- https://github.com/jenkinsci/maven-plugin?tab=contributing-ov-file

I understand that my PR is unlikely to be accepted. Nonetheless, if you have any suggestions on what to watch out for or things to improve my submission (to better my chances of getting it approved), I would appreciate it. At the end of the day, I think having this feature will improve the Maven plugin in a fundamental way. And since the feature request only simplifies what is already possible (creating multiple jobs manually), I think there is a decent chance that this might be seen favorably.

Thanks yet again for the help thus far.

[David Alayachew]

This goes to anyone else as well -- if anyone has any feedback or criticisms, I would appreciate it. For both my created Feature Request, as well as the soon-to-be-created PR.

[David Alayachew]

Hello all,

After giving a pretty thorough review of the code base, it's pretty clear that the assumption that the build occurs on a single node is assumed everywhere throughout the codebase. Updating the logic to work on multiple nodes would involve weaving through a lot of implicit assumptions, and would require me to make a lot of fairly intrusive changes in a bunch of places in the form of new parameters and value weaving.

Sadly, I've come to the same conclusion as the maintainers -- it's not worth the effort. Even if I did put in the effort, there's an extremely high chance I'd introduce a large number of bugs. It's just not wise to do. Maybe someone more intimately familiar than me can give it a shot, but considering that the maintainers themselves seem to be putting their efforts into the Jenkinsfile path, it's pretty clear that that is unlikely.

For the maintainers, feel free to point back to this discussion thread in case anyone else makes the same feature request in the future. My maven jobs involve a lot of brittle, complex, and ugly logic, and yet, trying to get it done via the Jenkinsfile is honestly not that bad. Plus, since that logic is repeated in a lot of places, I can extract that into files, and use those to create simple, modular calls.

I'll close the Jira ticket, and point back to this response as the close reason.

Thanks again for the time and help everyone.

David Alayachew