Git based builds; one commit at a time

[Steve K]

Let's say that, since my most recent build, there have been three (3) new commits.
Rather than having one build that includes all three commits, I would like to have three separate builds; one for each of the three in sequence.

Is anyone doing something like that?  If so, how are you doing it?

Thanks in advance.

[Dirk Heinrichs]

Not exactly something like that, but similar, I think. We develop everything on change branches, where each branch is directly related to one ticket in our tracking system. Jenkins is our gatekeeper for merging those branches back to master (or the maintenance branch for a specific release). So we use some scripting in our Jenkins jobs to clone/pull the repo, merge one branch (given as parameter), run build and tests and commit and push the result only if everything was OK. In case of failure in any of these steps, the developer is informed by email and the merge result is thrown away.

HTH...

    Dirk

--

Dirk Heinrichs, Senior Systems Engineer, Engineering Solutions
Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach
Tel: +49 2226 1596666 (Ansage) 1149
Email: d...@recommind.com
Skype: dirk.heinrichs.recommind
www.recommind.com

[Rob Mandeville]

It can be (mostly) done, but you are intentionally defeating a Jenkins feature and you need to take that into account.

 

When a job is monitoring a Git repository, there are two problems.  First off, by default it polls.  If two commits get in within the same poll interval, you’re getting both.  Secondly, you can only have one job in the queue with a given name and a given set of parameters.  Duplicate jobs in the queue get silently discarded.  This is a feature, not a bug, so that five commits don’t cause five builds.  It sounds like you explicitly don’t want this behavior.

 

To do what you want, build two jobs, a “poller” and a runner.  The “polling” job starts by polling Git (it’s in quotes for reasons below).  Its only purpose is to launch the runner job with the ${GIT_REVISION} as its only argument.

 

The runner doesn’t poll at all, it just pulls from the Git repository, explicitly getting the version passed in by the poller.  By making the revision an argument, this allows five commits to trigger five different jobs, and allows all five jobs to remain in the queue (as they have different arguments).

 

The above still has the problem of handling multiple commits in one polling interval.  In theory, you could poll once a minute, but that’s expensive and still not going to get you what you want.  There is no way to poll quickly enough to insure one job per commit.  You’re going to have to do something other than polling.

 

In my shop, we use a local GitLab installation for our source control.  We installed the Gitlab Hook Plugin, and set our jobs to only poll once a day.  The plugin allows GitLab to use a Web hook, so that whenever it receives a commit, it tells Jenkins “Poll on this repository NOW”.  Jenkins will then launch a poll on every job which is polling that Git repository.  Polling once a day satisfies that requirement, and our jobs launch within seconds of getting a commit.

 

It may still be possible to get multiple commits to come in fast enough so that the forced poll catches multiple commits.  If someone merges in a branch with five commits on it, I suspect that all five commits are considered simultaneous, and you will only poll once and build once.

 

If you absolutely, positively need one job per commit, including the case above (or cases where multiple people commit faster than Jenkins can even run its poll), now you’re going to have to get tricky.  Now you need to work in the poller job.  It will have to behave like so:

1.       Determine what the last commit built was.  Maybe it can use Jenkins internals, or get the parameters from the last build of the runner via HTTP to the job’s /api/xml page.

2.       Get the history of the repository or branch

3.       Find all the commits done after the last commit built

4.       In order, launch the runner for each of those commits.

 

--Rob

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/60bb2099-c4c1-4630-874c-339fc27c82b0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Click here to report this email as spam.

---

This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.

Thank you.

Please consider the environment before printing this email.

[Mark Waite]

You might also consider writing your own plugin as an extension of the git plugin.  

The git plugin already has the ability to schedule multiple runs of the same job from a single polling result.  It schedules more than one execution of the same job when it detects more than one matching new branch have been added since the last poll.

The plugin does not provide a direct way to do what you want, but I think you might be able to extend the existing behavior to meet your needs.

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0A40042D85E7C84DB443060EC44B3FD36D9A9B890A%40dekaexchange07.deka.local.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Jason Pyeron]

> -----Original Message-----
> From: Rob Mandeville
> Sent: Wednesday, December 31, 2014 8:57
>
[reformatted to make sense...]
>> From: Steve K

>> Sent: Tuesday, December 30, 2014 3:21 PM
>>

>> Let's say that, since my most recent build, there have been
>> three (3) new commits.
>> Rather than having one build that includes all three commits,
>> I would like to have three separate builds; one for each of
>> the three in sequence.
>>
>> Is anyone doing something like that? If so, how are you doing it?
>>
>> Thanks in advance.
>>

> It can be (mostly) done, but you are intentionally defeating
> a Jenkins feature and you need to take that into account.
>

This sounds like you should be using a hook to ensure no commits are recevied (added to the repo) unless they pass all the tests. Steve, can you confirm that intention?

<snip/>

> If you absolutely, positively need one job per commit,
> including the case above (or cases where multiple people
> commit faster than Jenkins can even run its poll), now you're
> going to have to get tricky. Now you need to work in the
> poller job. It will have to behave like so:
>
> 1. Determine what the last commit built was. Maybe it
> can use Jenkins internals, or get the parameters from the
> last build of the runner via HTTP to the job's /api/xml page.
>
> 2. Get the history of the repository or branch
>
> 3. Find all the commits done after the last commit built
>
> 4. In order, launch the runner for each of those commits.
>


--

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
- -
- Jason Pyeron PD Inc. http://www.pdinc.us -
- Principal Consultant 10 West 24th Street #100 -
- +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
- -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.

[Steve K]

Jason, Yes. True. Development wants to ensure no commits are received (added to the repo) unless they pass all the tests

[Mark Waite]

If that's the case, you should probably consider the pre-tested commit plugin that is part of the Cloudbees commercial offering.

On Wed, Dec 31, 2014 at 7:48 AM, Steve K <Steve.K...@carestream.com> wrote:

Jason, Yes. True. Development wants to ensure no commits are received (added to the repo) unless they pass all the tests

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/398cb510-7500-43a4-98ed-27e9321ad6ac%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Dirk Heinrichs]

Am 31.12.2014 um 15:59 schrieb Mark Waite:

If that's the case, you should probably consider the pre-tested commit plugin that is part of the Cloudbees commercial offering.

Or use a branch based development approach and some lines of script code to let Jenkins do the pull - merge - build - test - push cycle. No rocket science, really, but may need some time to convince people.

Bye...

[James Nord]

Or just use Gerrit and the Gerrit trigger plugin :-)

--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.