[JIRA] (JENKINS-37727) Multibranch pipeline project jobs are not deleted when branches are deleted

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite updated an issue

Jenkins / JENKINS-37727 Multibranch pipeline project jobs are not deleted when branches are deleted Change By: Mark Waite Summary: Jobs of a multibranch Multibranch pipeline project jobs are not deleted when branches are deleted Add Comment

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite edited a comment on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted Thanks for catching that bug before release! The master branch adds command line git support for multi-branch pipelines, and makes command line git the default (just as command line git is the default in other cases).  Unfortunately, it appears there may be some implicit dependency on JGit behavior in the AbstractGitSCMSource or GitSCMSource implementation which causes branches to remain even though they were deleted from the remote repository. I was able to duplicate the problem as reported.  It is included in my [bug verification docker instance|https://github.com/MarkEWaite/docker/blob/lts-with-plugins/ref/jobs/jenkins-bugs-multibranch-pipeline/config.xml] and references the [JENKINS-37727 branch of my jenkins-bugs repository|https://github.com/MarkEWaite/jenkins-bugs/tree/JENKINS-37727]. If I change the job definition from using the command line git implementation to use the jgit implementation, the jobs which correspond to deleted branches are removed, so long as the local branch extension is not enabled (create a branch for each build).  If the local branch extension is enabled, or if I switch from jgit to the command line implementation, the jobs which correspond to deleted branches are not removed. I suspect there may be two problems here.   # If the local branch extension is enabled, I suspect that the cache repository (used to extract the Jenkinsfile) performs a checkout to a named branch matching the target branch.  That checkout of a named branch in the cache repository is necessary (to extract the Jenkinsfile), but it does not need to checkout as a named branch.  It can use a detached head checkout. # Command line git branch pruning doesn't seem to be working in the cache I marked this as critical, since users will not want to modify their multi-branch job definitions to use JGit, especially since there are authentication cases which command git can handle that JGit cannot. I've found what appears to be a [clean fix for the problem|https://github.com/MarkEWaite/git-plugin/pull/2].  I've not yet completed the testing, and still need to write a unit test to show the problem, but it seems sufficient to resolve  the issue, replaces a call to a deprecated method, and simplifies the code.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite edited a comment on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

Thanks for catching that bug before release!

The master branch adds command line git support for multi-branch pipelines, and makes command line git the default (just as command line git is the default in other cases).  Unfortunately, it appears there may be some implicit dependency on the calling sequence which worked for JGit behavior (fetch() followed by prune()) behaves better in JGit than CliGit).  Changing the AbstractGitSCMSource or GitSCMSource implementation which causes branches calling sequence to remain even though they were deleted from fetch_().prune().execute() fixes the remote repository issue for both JGit and CliGit) .

I was able to duplicate the problem as reported.  It is included in my [bug verification docker instance|https://github.com/MarkEWaite/docker/blob/lts-with-plugins/ref/jobs/jenkins-bugs-multibranch-pipeline/config.xml] and references the [JENKINS-37727 branch of my jenkins-bugs repository|https://github.com/MarkEWaite/jenkins-bugs/tree/JENKINS-37727]. If I change the job definition from using the command line git implementation to use the jgit implementation, the jobs which correspond to deleted branches are removed, so long as the local branch extension is not enabled (create a branch for each build).  If the local branch extension is enabled, or if I switch from jgit to the command line implementation, the jobs which correspond to deleted branches are not removed. I suspect there may be two problems here.   # If the local branch extension is enabled, I suspect that the cache repository (used to extract the Jenkinsfile) performs a checkout to a named branch matching the target branch.  That checkout of a named branch in the cache repository is necessary (to extract the Jenkinsfile), but it does not need to checkout as a named branch.  It can use a detached head checkout. # Command line git branch pruning doesn't seem to be working in the cache I marked this as critical, since users will not want to modify their multi-branch job definitions to use JGit, especially since there are authentication cases which command git can handle that JGit cannot. I've found what appears to be a [clean fix for the problem|https://github.com/MarkEWaite/git-plugin/pull/2].  I've not yet completed the testing, and still need to write a unit test to show the problem, but it seems sufficient to resolve  the issue, replaces a call to a deprecated method, and simplifies the code.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite edited a comment on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

Thanks for catching that bug before release!

The master branch adds command line git support for multi-branch pipelines, and makes command line git the default (just as command line git is the default in other cases).  Unfortunately, the calling sequence which worked for JGit (fetch() followed by prune()) behaves better in JGit than CliGit ) .  Changing the calling sequence to fetch_().prune().execute() fixes the issue for both JGit and CliGit).

I was able to duplicate the problem as reported.  It is included in my [bug verification docker instance|https://github.com/MarkEWaite/docker/blob/lts-with-plugins/ref/jobs/jenkins-bugs-multibranch-pipeline/config.xml] and references the [JENKINS-37727 branch of my jenkins-bugs repository|https://github.com/MarkEWaite/jenkins-bugs/tree/JENKINS-37727]. If I change the job definition from using the command line git implementation to use the jgit implementation, the jobs which correspond to deleted branches are removed, so long as the local branch extension is not enabled (create a branch for each build).  If the local branch extension is enabled, or if I switch from jgit to the command line implementation, the jobs which correspond to deleted branches are not removed. I suspect there may be two problems here.   # If the local branch extension is enabled, I suspect that the cache repository (used to extract the Jenkinsfile) performs a checkout to a named branch matching the target branch.  That checkout of a named branch in the cache repository is necessary (to extract the Jenkinsfile), but it does not need to checkout as a named branch.  It can use a detached head checkout. # Command line git branch pruning doesn't seem to be working in the cache I marked this as critical, since users will not want to modify their multi-branch job definitions to use JGit, especially since there are authentication cases which command git can handle that JGit cannot. I've found what appears to be a [clean fix for the problem|https://github.com/MarkEWaite/git-plugin/pull/2].  I've not yet completed the testing, and still need to write a unit test to show the problem, but it seems sufficient to resolve  the issue, replaces a call to a deprecated method, and simplifies the code.

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

Code changed in jenkins User: Mark Waite Path: src/main/java/jenkins/plugins/git/AbstractGitSCMSource.java src/test/java/jenkins/plugins/git/AbstractGitSCMSourceTest.java http://jenkins-ci.org/commit/git-plugin/0197c85c64807fa4f6acdab5c04c02bb36e33c42 Log: Prune during AbstractGitSCMSource fetch Need for separate prune is removed by switching from deprecated fetch() method to fetch_() method. [Fix JENKINS-37727] CliGit leaves deleted branches in multi-branch jobs

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite edited a comment on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

Thanks for catching that bug before release!

The master branch adds command line git support for multi-branch pipelines, and makes command line git the default (just as command line git is the default in other cases).  Unfortunately, the calling sequence which worked for JGit (fetch() followed by prune()) behaves better in JGit than CliGit.  Changing the calling sequence to fetch_().prune().execute() fixes the issue for both JGit and CliGit).

I was able to duplicate the problem as reported.  It is included in my [bug verification docker instance|https://github.com/MarkEWaite/docker/blob/lts-with-plugins/ref/jobs/jenkins-bugs-multibranch-pipeline/config.xml] and references the [JENKINS-37727 branch of my jenkins-bugs repository|https://github.com/MarkEWaite/jenkins-bugs/tree/JENKINS-37727]. If I change the job definition from using the command line git implementation to use the jgit implementation, the jobs which correspond to deleted branches are removed, so long as the local branch extension is not enabled (create a branch for each build).  If the local branch extension is enabled, or if I switch from jgit to the command line implementation, the jobs which correspond to deleted branches are not removed. I suspect there may be two problems here.   # If the local branch extension is enabled, I suspect that the cache repository (used to extract the Jenkinsfile) performs a checkout to a named branch matching the target branch.  That checkout of a named branch in the cache repository is necessary (to extract the Jenkinsfile), but it does not need to checkout as a named branch.  It can use a detached head checkout. # Command line git branch pruning doesn't seem to be working in the cache I marked this as critical, since users will not want to modify their multi-branch job definitions to use JGit, especially since there are authentication cases which command git can handle that JGit cannot.

I've found what appears to be a [clean fix for the problem|https://github.com/MarkEWaite/git-plugin/pull/2].   I've not yet completed the testing,   Unit test and still need integration test are both available to write a confirm the problem is fixed.  The unit test could use some further work to show assure it tests both the problem CliGit and JGit implementations , but it seems sufficient that will need to resolve  the issue, replaces a call to a deprecated method, and simplifies the code wait for another time .

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite started work on JENKINS-37727

Change By: Mark Waite Status: Open In Progress

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite resolved as Fixed

Included in git plugin 2.6.0, released 2 Sep 2016. Was never visible in any released version of the plugin.

Jenkins / JENKINS-37727

Multibranch pipeline project jobs are not deleted when branches are deleted

Change By: Mark Waite Status: In Progress Resolved Resolution: Fixed

Add Comment

[assafl1616@gmail.com (JIRA)]

Assaf Leibovitch commented on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted Hi, I just installed 2.7.3 , using Git plugin 2.6.0, configured a multi-branch job, included feature* branches where I have 3 of them. 3 jobs were created as expected. I then deleted one branch but I still see it in the indexing log... Seen branch in repository origin/feature/XAVI_MULTI_Demo1 Seen branch in repository origin/feature/XAVI_MULTI_Demo2 Seen branch in repository origin/feature/XAVI_MULTI_Demo3 While on git there's no such branch $ git branch -a|grep -i demo feature/XAVI_MULTI_Demo1 feature/XAVI_MULTI_Demo2 remotes/origin/feature/XAVI_MULTI_Demo1 remotes/origin/feature/XAVI_MULTI_Demo2 On the master I see the workspace folders feature%2FXAVI_MULTI_Demo1@script feature%2FXAVI_MULTI_Demo2@script feature%2FXAVI_MULTI_Demo3@script and the deleted branch folder is empty... Is it related to this issue as well?

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

Leibo, that sounds like the same condition which shows the problem (sometimes) in my Docker test instance. Could you try switching that specific pipeline job definition from using the "Default" git implementation to using "jgit" as the implementation? Prior to git plugin 2.6.0, the only implementation available to the pipeline branch indexing was the "jgit" implementation and it seemed well behaved unless "checkout to local branch" was specified. The switch from "Default" to "jgit" should be available in the Jenkins job definition with the label "Git executable". Choices in that pick list should be "Default" or "jgit". Could you also try without the "Checkout to local branch" extension enabled?

Add Comment

[assafl1616@gmail.com (JIRA)]

Leibo edited a comment on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

Hi, I just installed 2.7.3 , using Git plugin 2.6.0, configured a multi-branch job, included feature* branches where I have 3 of them. 3 jobs were created as expected. I then deleted one branch but I still see it in the indexing log... Seen branch in repository origin/feature/XAVI_MULTI_Demo1 Seen branch in repository origin/feature/XAVI_MULTI_Demo2 Seen branch in repository origin/feature/XAVI_MULTI_Demo3 While on git there's no such branch

{code}

$ git branch -a|grep -i demo   feature/XAVI_MULTI_Demo1   feature/XAVI_MULTI_Demo2   remotes/origin/feature/XAVI_MULTI_Demo1   remotes/origin/feature/XAVI_MULTI_Demo2

{code}

On the master I see the workspace folders

{code} feature%2FXAVI_MULTI_Demo1@script feature%2FXAVI_MULTI_Demo2@script feature%2FXAVI_MULTI_Demo3@script {code}

and the deleted branch folder is empty...

After I removed the git-xxxxx folder from Jenkins caches folder - reindexing was ok.

Is it related to this issue as well?

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite edited a comment on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

[~assafl], that sounds like the same condition which shows the problem (sometimes) in my Docker test instance.

Could you try switching that specific pipeline job definition from using the "Default" git implementation to using "jgit" as the implementation?  Prior to git plugin 2.6.0, the only implementation available to the pipeline branch indexing was the "jgit" implementation and it seemed well behaved unless "checkout to local branch" was specified.  The switch from "Default" to "jgit" should be available in the Jenkins job definition with the label "Git executable".  Choices in that pick list should be "Default" or "jgit". Could you also try without the "Checkout to local branch" extension enabled?

Removing the git-xxxx folder from the Jenkins cache directory will (at least temporarily) fix the issue, since that will remove the deleted branch from the cache repository.  However, it seems likely that the problem will reappear the next time a branch is deleted, since the prune() call in the cached repository seems to not be removing that deleted branch.

Add Comment

[assafl1616@gmail.com (JIRA)]

Leibo commented on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

I don't see any options to change the definition... for the 'Add source' config item I can only pick from Git, Github, Single repo or subversion

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

I you don't see the phrase "Git executable" in your job definition page (it is inside the Git configuration section), then that means you haven't yet enabled "jgit" as a globally available git implementation in your Jenkins server. Add jgit as an available global git implementation in your Jenkins server by clicking "Manage Jenkins", then "Global Tool Configuration". In the "Git" section of that page, click the "Add" button and select "jgit" from the dropdown list.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite edited a comment on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

I If you don't see the phrase "Git executable" in your job definition page (it is inside the Git configuration section of the job definition page ), then that means you haven't yet enabled "jgit" as a globally available git implementation in your Jenkins server.

Add jgit as an available global git implementation in your Jenkins server by clicking "Manage Jenkins", then "Global Tool Configuration".  In the "Git" section of that page, click the "Add" button and select "jgit" from the dropdown list.

Add Comment

[assafl1616@gmail.com (JIRA)]

Leibo commented on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

Hi, Added the JGit as explained but I might be missing something because I get the following error: ... FATAL: Failed to recompute children of XAVI_Multibranch java.lang.IllegalStateException: Cannot open session, connection is not authenticated. at com.trilead.ssh2.Connection.openSession(Connection.java:1127) Tried to add a simple free style job with JGit - also get authentication error. We use SSH keys to authenticate, the key is in Jenkins .ssh folder. I saw some issues regarding this but nothing I could get help with. BTW, After reverting to use Git - the next reindexing deleted the branch and job but only once... if I create/delete another one we still have the same issue.

Add Comment

[assafl1616@gmail.com (JIRA)]

Leibo edited a comment on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

Hi, Added the JGit as explained but I might be missing something because I get the following and got authentication error : {code} ...

FATAL: Failed to recompute children of XAVI_Multibranch java.lang.IllegalStateException: Cannot open session, connection is not authenticated. at com.trilead.ssh2.Connection.openSession(Connection.java:1127)

{code} Tried to add after some issues searching I created a simple free style job with JGit - also get new user based on our SSH private key authentication error. and it seems working!!!

We use SSH keys to authenticate, the key is in Jenkins .ssh folder.

So we're good I saw some issues regarding this but nothing I could get help with hope . BTW, After reverting to use Git - Thanks a lot for the next reindexing deleted the branch and job but only once... if I create/delete another one we still have the same issue. fast response :-)

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

I'm glad it is working for you, though I'm skeptical that the issue is fully resolved. I don't yet understand the conditions which caused you (and me) to see the issue, while Quentin Dufour and I to not see the issue. I realize it is strange that I've both seen the problem, and not seen the problem, but that just means I don't really understand the circumstances which show the bug.

Add Comment

[gelefisk@gmail.com (JIRA)]

Øystein Kjærnet commented on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

I experience exactly the same as Leibo - old jobs from deleted branches not being deleted, even though I've selected "Discard old items" in the build config - choosing JGit as suggested by Mark fixed this. Jenkins version 2.20, Git plugin 3.0.0, Git client plugin 2.0.0, Build Pipeline 1.5.4, Pipeline: Multibranch 2.8, git version on system: 1.7.1.

Add Comment

[gelefisk@gmail.com (JIRA)]

Øystein Kjærnet edited a comment on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

I experience exactly the same as Leibo - old jobs from deleted branches not being deleted, even though I've selected "Discard old items" in the build config - choosing JGit as suggested by Mark fixed this. Jenkins version 2.20, Git plugin 3.0.0, Git client plugin 2.0.0, Build Pipeline 1.5.4, Pipeline: Multibranch 2.8, git version on system: 1.7.1. I'd be glad to help debugging this in any way...

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

@gelefisk can you confirm that in your environment git plugin 2.5.3 does not show the problem, while git plugin 2.6.0 and git plugin 3.0.0 show the problem? Can you also explore further to see if you can detect the set of steps which would show the problem on a fresh installation? If that works, then could you explore placing that fresh set of steps into a job on a docker instance (preferably as a pull request to the bugs folder of my lts-with-plugins branch)?

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite edited a comment on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

@ [~ gelefisk ] can you confirm that in your environment git plugin 2.5.3 does not show the problem, while git plugin 2.6.0 and git plugin 3.0.0 show the problem?

Can you also explore further to see if you can detect the set of steps which would show the problem on a fresh installation?

If that works, then could you explore placing that fresh set of steps into a job on a docker instance (preferably as a pull request to the [bugs folder|https://github.com/MarkEWaite/docker/tree/lts-with-plugins/ref/jobs/bugs/jobs] of my [lts-with-plugins branch|https://github.com/MarkEWaite/docker/tree/lts-with-plugins])?

Add Comment

[marius.balteanu@zitec.ro (JIRA)]

Marius Balteanu commented on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

I've the same issue and downgrading the Git plugin to 2.5.3 didn't solve the issue. I'll try the jgit workaround. git client plugin 2.1.0 git plugin 2.5.3 / 3.0.0 pipeline multibranch 2.9.2 git version on the server 2.9.2

Add Comment

[marius.balteanu@zitec.ro (JIRA)]

Marius Balteanu edited a comment on JENKINS-37727

Re: Multibranch pipeline project jobs are not deleted when branches are deleted

I've the same issue and downgrading the Git plugin to 2.5.3 didn't solve the issue. I'll try the jgit workaround.

* git client plugin 2.1.0 * git plugin 2.5.3 / 3.0.0 * pipeline multibranch 2.9.2 * git version on the server 2.9.2 Please ignore my above comment because my problem was from the job configuration.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite closed an issue as Fixed

Jenkins / JENKINS-37727

Multibranch pipeline project jobs are not deleted when branches are deleted

Change By: Mark Waite Status: Resolved Closed

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)