Concurrent builds with SCM polling

[David Shaw]

Hello,

Here's what I'm trying to do: I have a project stored in Mercurial and I want to have Jenkins check every 30 minutes for changes (build trigger to poll the SCM every 30 minutes), and if it finds any changes, to build that project. That part works fine. However, the project takes a long time to build, and for some reason Jenkins won't build the project a second time while there is a build in progress.

So this is what happens, starting at 12 noon:

12:00 Jenkins sees a change in Mercurial. Starts build #1.
12:15 Another change made in Mercurial
12:30 Jenkins can't manage to poll for a change. The Mercurial Polling Log just shows "Started on (date) 12:30" and never progresses past that single line.
13:20 Build #1 finishes. Suddenly the polling is unblocked and Jenkins sees the change made at 12:15. Kicks off build #2.

To answer the obvious question first, concurrent builds are definitely enabled. Concurrent builds work fine if I hit the "build now" link while a build is taking place. It only doesn't work with Mercurial polling.

Having read that some plugins can cause this problem, I've gone so far as to install a brand new Jenkins box with no additional plugins aside from the Mercurial one, and created a job description that has no build steps other than "sleep 120", and no post-build actions. I've tried it with both a single Jenkins box as well as a master/slave architecture. I've walked the Jenkins and Mercurial plugin version up and down as well. No dice with any of these.

Any advice would be very welcome.

David

[Mandeville, Rob]

I believe that the behavior you are seeing is by design. The concurrent builds checkbox allows you to run multiple instances of that job at all, but the SCM poller will only launch one at a time. Basically, with the checkbox you could kick off as many builds as you wanted manually.

If you really want to be able to launch a build at 12:30 in the below case, you need two jobs.

Job A monitors SCM, and when it sees an SCM change, it launches job B.

Job B doesn't monitor SCM at all. It lets job A kick it off, grabs sources from SCM, and goes to work on it. You will need the concurrent build flag on this one.

I have something similar in my own setup. Job A doesn't have any meaningful build actions, but has a post-build action of "trigger parameterized build on other projects" (IIRC, you need the Parameterized Trigger plugin). Tell job A to launch job B in that post-build action and pass any SCM information (such as branch) as a parameter. So now, the timeline goes as:

12:00 Jenkins Project A polls Mercurial and sees a change. Starts build #A-1
12:02 Jenkins A-1 launches job against project B (build #B-1) and exits out. Job #B-1 runs for the next 80 minutes
12:15 Another change is made in mercurial.
12:30 Jenkins job A polls Mercurial (since it isn't running a build on project A) and sees a change, starting build #A-2
12:32 Jenkins A-2 launches #B-2, which runs for the next 80 minutes. A-1 exits out.
13:20 Jenkins job B-1 finishes
13:50 Jenkins job B-2 finishes

--Rob

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.

[Dean Yu]

On Thursday, May 9, 2013 1:02:58 PM UTC-7, Mandeville, Rob wrote:

I believe that the behavior you are seeing is by design.  The concurrent builds checkbox allows you to run multiple instances of that job at all, but the SCM poller will only launch one at a time.  Basically, with the checkbox you could kick off as many builds as you wanted manually.

This isn't quite right. SCM polling is independent of individual builds. The fact that SCM polling is blocked while a build is in progress sounds like a bug in the Mercurial plugin.

What should be happening is that every half hour, Jenkins will poll your configured source location in Mercurial for changes. If there are changes, a new build is scheduled into the build queue. If the project is not marked as able to run concurrently, and there is no in progress build of the job, and there is an available executor, then the new build will start. If there is a build of the job in progress, the new build will wait in the queue until the previous build completes. If, at the next polling interval, there is yet another change detected, and the first build is still in progress, the build in the queue is updated to check out from the new revision. In other words, there is normally only one outstanding build of the project in the queue at a time.

If the project is configured to run concurrently, Jenkins will assign the build to an available executor, even if another build of the same project is still in progress.

Yet another possibility is if you have some mutex or semaphore configured (via Throttle Concurrent Builds, Port Allocator, Locks and Latches, or some similar plugin), and that is preventing multiple builds from running at once.

  -- Dean

[David Shaw]

On May 9, 2013, at 4:02 PM, "Mandeville, Rob" <rmand...@litle.com> wrote:

> I believe that the behavior you are seeing is by design. The concurrent builds checkbox allows you to run multiple instances of that job at all, but the SCM poller will only launch one at a time. Basically, with the checkbox you could kick off as many builds as you wanted manually.
>
> If you really want to be able to launch a build at 12:30 in the below case, you need two jobs.
>
> Job A monitors SCM, and when it sees an SCM change, it launches job B.
>
> Job B doesn't monitor SCM at all. It lets job A kick it off, grabs sources from SCM, and goes to work on it. You will need the concurrent build flag on this one.

Thank you very much for this workaround! It solves the problem. I wonder though - is this behavior recent? Unless I'm quite confused, having multiple concurrent builds triggered by polling worked in the past.

David

[David Shaw]

On May 10, 2013, at 5:39 PM, Dean Yu <dea...@gmail.com> wrote:

> On Thursday, May 9, 2013 1:02:58 PM UTC-7, Mandeville, Rob wrote:
> I believe that the behavior you are seeing is by design. The concurrent builds checkbox allows you to run multiple instances of that job at all, but the SCM poller will only launch one at a time. Basically, with the checkbox you could kick off as many builds as you wanted manually.
>
> This isn't quite right. SCM polling is independent of individual builds. The fact that SCM polling is blocked while a build is in progress sounds like a bug in the Mercurial plugin.
>
> What should be happening is that every half hour, Jenkins will poll your configured source location in Mercurial for changes. If there are changes, a new build is scheduled into the build queue. If the project is not marked as able to run concurrently, and there is no in progress build of the job, and there is an available executor, then the new build will start. If there is a build of the job in progress, the new build will wait in the queue until the previous build completes. If, at the next polling interval, there is yet another change detected, and the first build is still in progress, the build in the queue is updated to check out from the new revision. In other words, there is normally only one outstanding build of the project in the queue at a time.
>
> If the project is configured to run concurrently, Jenkins will assign the build to an available executor, even if another build of the same project is still in progress.

Unfortunately, this isn't what is happening. The job is configured to run concurrently, but the second build isn't queued until the first build completes.

> Yet another possibility is if you have some mutex or semaphore configured (via Throttle Concurrent Builds, Port Allocator, Locks and Latches, or some similar plugin), and that is preventing multiple builds from running at once.

I'm not using any of those plugins. To test, I set up a Jenkins install with no plugins (aside from Mercurial and the ones that are shipped with Jenkins). It still happens.

Any thoughts on how to debug this further? I'm okay with the workaround that Rob Mandeville suggested, but if it's a bug, better to get it fixed than rely on a workaround forever.

David

[Dean Yu]

Next time this is happening, capture a thread dump of the master and all
the slaves by going to the URL:

http://<yourJenkinsServer>/threadDump

We should be able to tell from the thread dump what the SCM polling thread
is getting blocked on.

-- Dean