Issues with a job configured to build any git 'feature' branch.

[Jim Lloyd]

Our workflow using git is similar to the GitFlow model that is now commonly used by many developer teams. In particular, we create feature branches that use the prefix feature/, and we have a Jenkins job configured to build any push to any branch matching the pattern 'origin/feature/**'. Developers work on these branches in parallel, so the order they are committed is not in any logical sequence. That is, it doesn't make sense for the jenkins job to consider the change set for the job to be the difference between the commit being built, and the commit built in the last successful run of the job. Rather, we'd like the set of changes to simply be the changes contained in the push. Is there any way to support this model?

A different but related concern is what the job for building feature branches should do when it cannot keep up with multiple independent pushes of feature branches happening in quick succession. We currently have to throttle these jobs to run just one job at a time, and the jobs take approximately an hour. I think we'd like to ensure that every one of these pushes is eventually built, even if it results in a backlog that is only eventually cleared in the middle of the night. We are working on ways to allow jobs to run in parallel, but may have to live with the current setup for a month or two. In the meantime, is there a way to add jobs to the queue immediately, with the exact commit to build specified in the job?

[Mark Waite]

I thought I remembered a pull request submitted to the git plugin to allow user definition of the comparison base.  In your case, there is a "base branch" against which you want all comparisons, no matter the results of the last job.  Unfortunately, I didn't find it in the current pull requests for the git plugin, though I thought it was there.  Others may be able to find it and consider if it should be merged.  If you can locate that pull request, you could consider building an experimental version of the git plugin with that change included, then could report the results of your testing back to the pull request.

Can't you obtain the results you want (on your different, but related concern) by asking your developers to push a new branch for changes which are to be built independently, rather than pushing to an existing branch?  I believe that when pushes are detected to a new branch, a new build is queued.  If multiple pushes to an existing branch are detected, they are all batched together into a single job.

Mark Waite

On Tue, Jul 22, 2014 at 12:28 PM, Jim Lloyd <jim....@gmail.com> wrote:

Our workflow using git is similar to the GitFlow model that is now commonly used by many developer teams. In particular, we create feature branches that use the prefix feature/, and we have a Jenkins job configured to build any push to any branch matching the pattern 'origin/feature/**'. Developers work on these branches in parallel, so the order they are committed is not in any logical sequence. That is, it doesn't make sense for the jenkins job to consider the change set for the job to be the difference between the commit being built, and the commit built in the last successful run of the job. Rather, we'd like the set of changes to simply be the changes contained in the push. Is there any way to support this model?

A different but related concern is what the job for building feature branches should do when it cannot keep up with multiple independent pushes of feature branches happening in quick succession. We currently have to throttle these jobs to run just one job at a time, and the jobs take approximately an hour. I think we'd like to ensure that every one of these pushes is eventually built, even if it results in a backlog that is only eventually cleared in the middle of the night. We are working on ways to allow jobs to run in parallel, but may have to live with the current setup for a month or two. In the meantime, is there a way to add jobs to the queue immediately, with the exact commit to build specified in the job?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Jim Lloyd]

Hi Mark, thanks for the reply.

On Tuesday, July 22, 2014 12:39:52 PM UTC-7, Mark Waite wrote:

I thought I remembered a pull request submitted to the git plugin to allow user definition of the comparison base.  In your case, there is a "base branch" against which you want all comparisons, no matter the results of the last job.  Unfortunately, I didn't find it in the current pull requests for the git plugin, though I thought it was there.  Others may be able to find it and consider if it should be merged.  If you can locate that pull request, you could consider building an experimental version of the git plugin with that change included, then could report the results of your testing back to the pull request.

I spent some time reviewing the existing open PRs. There are a lot of them, and several seem like they are related in some way. Unfortunately, I don't have the luxury to spend time testing & refining any of these now, though we have an open req for a build engineer. I'm hopeful we can find someone who could do this kind of work. 

Can't you obtain the results you want (on your different, but related concern) by asking your developers to push a new branch for changes which are to be built independently, rather than pushing to an existing branch?  I believe that when pushes are detected to a new branch, a new build is queued.  If multiple pushes to an existing branch are detected, they are all batched together into a single job.

Most of the pushes are for new branches. A specific branch may have commits pushed to it multiple times, but the goal is that for most feature work, a developer pushes a feature/<unique-feature-name> branch only when they have completed the work on the feature, after already running unit tests on their workstation. If the Jenkins job passes, and no changes are requested by the reviewer in the of the Pull Request code review, then the feature branch can be merged and deleted with no additional commits. I believe the problem we have is that the Git plugin assumes that a Jenkins job is used for one specific branch, and is therefore partially confused when a job is configured to match a wildcard pattern like 'feature/**'.

-Jim

[Mark Waite]

On Tue, Jul 22, 2014 at 5:30 PM, Jim Lloyd <jim....@gmail.com> wrote:

Hi Mark, thanks for the reply.

On Tuesday, July 22, 2014 12:39:52 PM UTC-7, Mark Waite wrote:

I thought I remembered a pull request submitted to the git plugin to allow user definition of the comparison base.  In your case, there is a "base branch" against which you want all comparisons, no matter the results of the last job.  Unfortunately, I didn't find it in the current pull requests for the git plugin, though I thought it was there.  Others may be able to find it and consider if it should be merged.  If you can locate that pull request, you could consider building an experimental version of the git plugin with that change included, then could report the results of your testing back to the pull request.

I spent some time reviewing the existing open PRs. There are a lot of them, and several seem like they are related in some way. Unfortunately, I don't have the luxury to spend time testing & refining any of these now, though we have an open req for a build engineer. I'm hopeful we can find someone who could do this kind of work. 

Great.  We're always looking for help to improve things.

 

Can't you obtain the results you want (on your different, but related concern) by asking your developers to push a new branch for changes which are to be built independently, rather than pushing to an existing branch?  I believe that when pushes are detected to a new branch, a new build is queued.  If multiple pushes to an existing branch are detected, they are all batched together into a single job.

Most of the pushes are for new branches. A specific branch may have commits pushed to it multiple times, but the goal is that for most feature work, a developer pushes a feature/<unique-feature-name> branch only when they have completed the work on the feature, after already running unit tests on their workstation. If the Jenkins job passes, and no changes are requested by the reviewer in the of the Pull Request code review, then the feature branch can be merged and deleted with no additional commits. I believe the problem we have is that the Git plugin assumes that a Jenkins job is used for one specific branch, and is therefore partially confused when a job is configured to match a wildcard pattern like 'feature/**'.

The git plugin tries to handle the workflow you're describing.  If it detects changes on multiple branches, it schedules a separate build for each branch.

The git plugin does not handle the presentation of differences from a base branch nearly as well as I'd like.  I'm still not sure of the best way to envision how it should do that.  I think "difference to the base branch" is the model that best fits how I envision it, but I'm open to other ideas. 

You mentioned that you'd like the set of changes to simply be the changes contained in the push.  That seems like a model which even git does not fully support.  There is no record in the git repository of which push submitted a particular change.  If a branch contains 5 commits, those commits could have been pushed in a single "git push" or in 5 separate calls to "git push" or some combination.  Aren't you really wanting the presented differences to be the differences between the base branch and the feature branch being evaluated with that job?

Mark Waite

 

-Jim

On Tue, Jul 22, 2014 at 12:28 PM, Jim Lloyd <jim....@gmail.com> wrote:

Our workflow using git is similar to the GitFlow model that is now commonly used by many developer teams. In particular, we create feature branches that use the prefix feature/, and we have a Jenkins job configured to build any push to any branch matching the pattern 'origin/feature/**'. Developers work on these branches in parallel, so the order they are committed is not in any logical sequence. That is, it doesn't make sense for the jenkins job to consider the change set for the job to be the difference between the commit being built, and the commit built in the last successful run of the job. Rather, we'd like the set of changes to simply be the changes contained in the push. Is there any way to support this model?

A different but related concern is what the job for building feature branches should do when it cannot keep up with multiple independent pushes of feature branches happening in quick succession. We currently have to throttle these jobs to run just one job at a time, and the jobs take approximately an hour. I think we'd like to ensure that every one of these pushes is eventually built, even if it results in a backlog that is only eventually cleared in the middle of the night. We are working on ways to allow jobs to run in parallel, but may have to live with the current setup for a month or two. In the meantime, is there a way to add jobs to the queue immediately, with the exact commit to build specified in the job?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Jim Lloyd]

On Tue, Jul 22, 2014 at 5:37 PM, Mark Waite <mark.ea...@gmail.com> wrote:

On Tue, Jul 22, 2014 at 5:30 PM, Jim Lloyd <jim....@gmail.com> wrote:

Can't you obtain the results you want (on your different, but related concern) by asking your developers to push a new branch for changes which are to be built independently, rather than pushing to an existing branch?  I believe that when pushes are detected to a new branch, a new build is queued.  If multiple pushes to an existing branch are detected, they are all batched together into a single job.

Most of the pushes are for new branches. A specific branch may have commits pushed to it multiple times, but the goal is that for most feature work, a developer pushes a feature/<unique-feature-name> branch only when they have completed the work on the feature, after already running unit tests on their workstation. If the Jenkins job passes, and no changes are requested by the reviewer in the of the Pull Request code review, then the feature branch can be merged and deleted with no additional commits. I believe the problem we have is that the Git plugin assumes that a Jenkins job is used for one specific branch, and is therefore partially confused when a job is configured to match a wildcard pattern like 'feature/**'.

The git plugin tries to handle the workflow you're describing.  If it detects changes on multiple branches, it schedules a separate build for each branch.

The git plugin does not handle the presentation of differences from a base branch nearly as well as I'd like.  I'm still not sure of the best way to envision how it should do that.  I think "difference to the base branch" is the model that best fits how I envision it, but I'm open to other ideas. 

Yes, I think you're right, difference from the base branch would be the best solution.

 

You mentioned that you'd like the set of changes to simply be the changes contained in the push.  That seems like a model which even git does not fully support.  There is no record in the git repository of which push submitted a particular change.  If a branch contains 5 commits, those commits could have been pushed in a single "git push" or in 5 separate calls to "git push" or some combination.  

Good point! I see I was making a naive assumption.

 

Aren't you really wanting the presented differences to be the differences between the base branch and the feature branch being evaluated with that job?

Yes, you're right. If the base branch is known, 'git show-branch' can provide the commits. We currently have a situation where feature branches can be made off one of several long-lived branches. We could possibly create one job per long-lived branch, and change our feature branch naming convention so that each job would only build feature branches appropriate for it. But I wonder if it might be reasonable for us to enter all of the base branches that exist in the configuration, and then try 'git show-branch' against each base branch, choosing the one with the shortest list of commits?

But perhaps the right way to do this is to instead configure the job to trigger builds on pull requests. In that scenario, we know exactly the set of commits. The problem with this is we currently benefit by making it possible to verify the feature branch builds before the PR is submitted.

Jim

Mark Waite

 

-Jim

On Tue, Jul 22, 2014 at 12:28 PM, Jim Lloyd <jim....@gmail.com> wrote:

Our workflow using git is similar to the GitFlow model that is now commonly used by many developer teams. In particular, we create feature branches that use the prefix feature/, and we have a Jenkins job configured to build any push to any branch matching the pattern 'origin/feature/**'. Developers work on these branches in parallel, so the order they are committed is not in any logical sequence. That is, it doesn't make sense for the jenkins job to consider the change set for the job to be the difference between the commit being built, and the commit built in the last successful run of the job. Rather, we'd like the set of changes to simply be the changes contained in the push. Is there any way to support this model?

A different but related concern is what the job for building feature branches should do when it cannot keep up with multiple independent pushes of feature branches happening in quick succession. We currently have to throttle these jobs to run just one job at a time, and the jobs take approximately an hour. I think we'd like to ensure that every one of these pushes is eventually built, even if it results in a backlog that is only eventually cleared in the middle of the night. We are working on ways to allow jobs to run in parallel, but may have to live with the current setup for a month or two. In the meantime, is there a way to add jobs to the queue immediately, with the exact commit to build specified in the job?

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/lLYJ5oZrFhc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.

[Mark Waite]

If you're willing to help with the evaluation of a pull request, I believe that https://github.com/jenkinsci/git-plugin/pull/231 is the proposed pull request that is most likely to meet your needs.

I could build a pre-release of a git plugin which includes that pull request, and could upload it to a shared location from which you could pull it.  You'd need to install it and evaluate it for operational compatibility.  I'd need to evaluate the source code of the change for other possible incompatibilities.

Let me know if you're interested.

Mark Waite

[Jim Lloyd]

Sure Mark, let's do that. I'll email you directly to arrange the details.