[JIRA] (JENKINS-42519) Closed pull requests, branch jobs are not cleaned up

38 views
Skip to first unread message

stephenash@gmail.com (JIRA)

unread,
Mar 6, 2017, 3:11:01 PM3/6/17
to jenkinsc...@googlegroups.com
Stephen Ash created an issue
 
Jenkins / Bug JENKINS-42519
Closed pull requests, branch jobs are not cleaned up
Issue Type: Bug Bug
Assignee: Unassigned
Components: github-branch-source-plugin
Created: 2017/Mar/06 8:10 PM
Environment: - Jenkins 2.32.2 LTS
- Branch API Plugin 2.0.7
- Git plugin 3.1.0
- GitHub Branch Source Plugin 2.0.3
- Pipeline: Multibranch 2.13
Priority: Minor Minor
Reporter: Stephen Ash

When I am building a multibranch pipeline job, I am noticing that a lot of jobs in Jenkins are still showing and that their workspaces can still be found on disk (wasting space). It seems like when a github webhook event is received that a PR is closed or a branch is deleted, then those should be cleaned up automatically.

 

Steps:

  1. Create multi branch pipeline.
  2. Set branch source to GitHub repo.
  3. Configure to build all branches and PRs.
  4. Create PR, confirm that job runs successfully, etc.
  5. Merge PR.
    ==- From "Scan Repository > Events"
    [Mon Mar 06 19:49:54 UTC 2017] Received org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl REMOVED event from 10.50.226.16 ⇒ https://<jenkins-host>/github-webhook/ with timestamp Mon Mar 06 19:49:49 UTC 2017
    ==-

Results:

  • PR job shows as disabled (strikethrough) in Jenkins
  • Workspace is still on disk

Expected:

  • PR job is no longer shown in Jenkins
  • Workspace is cleaned up.

A similar behavior is also seen when a branch is deleted outside of a PR.

Workarounds:
If I manually trigger "Scan Repository > Run Now", I see the expected results. There may also be a workaround to have scan run automatically (scheduled), but I'm not sure how to do that yet.

 

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

ryan.campbell@gmail.com (JIRA)

unread,
Mar 15, 2017, 6:45:01 PM3/15/17
to jenkinsc...@googlegroups.com
recampbell commented on Bug JENKINS-42519
 
Re: Closed pull requests, branch jobs are not cleaned up

What is your orphaned item strategy set to?

stephenash@gmail.com (JIRA)

unread,
Mar 15, 2017, 7:07:04 PM3/15/17
to jenkinsc...@googlegroups.com

"Discard old items" is checked.

Days to keep old items = 0

Max # of old items to keep = 0

 

If I have "Periodically if not otherwise run" set to some value it will also clean things up, but that was trying to avoid using that because of issues described in JENKINS-37491.

stephen.alan.connolly@gmail.com (JIRA)

unread,
Mar 16, 2017, 7:02:01 AM3/16/17
to jenkinsc...@googlegroups.com

You need to run periodically, if even once a week

stephenash@gmail.com (JIRA)

unread,
Mar 16, 2017, 1:07:02 PM3/16/17
to jenkinsc...@googlegroups.com

You need to run periodically, if even once a week

Maybe I'm confused about the role of branch indexing. Since Jenkins got the webhook about the merged PR (or a deleted branch), why wouldn't the related jobs and workspaces be removed right then? Deferring that action until branch indexing runs (and that may not run if "Periodically if not otherwise run" is unchecked by the user) seems wrong to me.

stephen.alan.connolly@gmail.com (JIRA)

unread,
Mar 16, 2017, 1:26:02 PM3/16/17
to jenkinsc...@googlegroups.com

Well if your orphaned item strategy says keep nothing, you might not have that issue... but when the orphaned item strategy says keep the last 3 branches...

  • There are two branches that are already orphaned.
  • You get three events for branch deletions.
  • The events are processing in parallel.
  • How does each event magically know that there are two other events in flight and hence the two existing orphaned branches are now pushed over the edge and need to be deleted?

We may look into augmenting the orphaned branch strategy to allow it to be run post-event, but for now the reality is that the only way it can correctly run is when it knows a consistent complete list of all live branches (and consequently has a consistent complete set of all dead branches)

stephen.alan.connolly@gmail.com (JIRA)

unread,
Mar 16, 2017, 1:28:01 PM3/16/17
to jenkinsc...@googlegroups.com

Also, consider the use case of users wanting to trigger a rebuild of a PR.

The way that happens right now is you close the PR and open it again.

If we deleted the branch immediately on the close event then the PR build history would be destroyed so that when re-opened it would start from build 1

acejam@gmail.com (JIRA)

unread,
Mar 15, 2018, 10:00:01 AM3/15/18
to jenkinsc...@googlegroups.com

I would like to bump this back up. Orphaned jobs should be deleted when an incoming webhook is received if the item strategy is set to 0. I also agree with Stephen Ash about JENKINS-37491. If a user wants to use GitHub Organization folders via the branch source plugin and build "merged" PR's, they either have to deal with build storms (by having "Periodically if not otherwise run" turned on) or deal with orphaned jobs. (By having "Periodically if not otherwise run" turned off, to prevent build storms)

acejam@gmail.com (JIRA)

unread,
Mar 15, 2018, 10:02:02 AM3/15/18
to jenkinsc...@googlegroups.com
Joshua Noble reopened an issue
 
Change By: Joshua Noble
Resolution: Not A Defect
Status: Resolved Reopened
Assignee: Stephen Connolly

stephen.alan.connolly@gmail.com (JIRA)

unread,
Mar 15, 2018, 5:44:03 PM3/15/18
to jenkinsc...@googlegroups.com
Stephen Connolly resolved as Not A Defect
 

It is not possible with the design of the API to know if the event means that the job should be deleted or if a lower priority source in the multibranch project has already received an event for the same branch being created because the branch is being moved from one source to another.

e.g.

if you configure two sources in the same multibranch:

  • g...@git.example.com:/development.git
  • g...@git.example.com:/production.git

Now suppose you do something like:

$ git remote add origin g...@git.example.com:/development.git
$ git remote add production g...@git.example.com:/production.git
$ git fetch origin feature-1
$ git push production feature-1
$ git push origin :feature-1

So what happens if we follow Joshua Noble's suggestion is

  1. Jenkins gets the event from "production" saying "new branch 'feature-1'" and we say: "cool, but development is higher in the list of sources and already owns that branch name... I'll just drop that event on the floor"
  2. Jenkins gets the event from "development" saying "branch 'feature-1' has been deleted" and we'd delete the job and its history.

So you would lose all the job history of feature-1 being developed.

What happens currently is as follows:

  1. Jenkins gets the event from "production" saying "new branch 'feature-1'" and we say: "cool, but development is higher in the list of sources and already owns that branch name... I'll just drop that event on the floor"
  2. Jenkins gets the event from "development" saying "branch 'feature-1' has been deleted" and we mark the branch as dead.
  3. If there is a subsequent event from "production" we let that event "take over" the branch, and all is good, history retained.
  4. When the scan comes along, we get the full list of branches from each source in priority order and we assign the branch to "production" as a live branch, because the revision is the same, we don't trigger a rebuild
Change By: Stephen Connolly
Status: Reopened Resolved
Resolution: Not A Defect

stephen.alan.connolly@gmail.com (JIRA)

unread,
Mar 15, 2018, 5:47:03 PM3/15/18
to jenkinsc...@googlegroups.com
$ git remote add origin g...@git.example.com:/development.git
$ git remote add production g...@git.example.com:
/production.git
$ git fetch origin feature-1
$ git push origin :feature-1
$ git push production feature-1

In this case, it is even more critical that we do not delete the branch immediately as if we followed Joshua Noble's suggestion the job history would be erased by the origin branch being deleted and then the production branch would be starting from build #1 again.

 

If the user actually wants the build history reset, then all they need to do is use the Jenkins UI to delete the dead branch before doing the push of the branch to production

acejam@gmail.com (JIRA)

unread,
Mar 16, 2018, 10:34:02 AM3/16/18
to jenkinsc...@googlegroups.com

Isn't the whole point of a multibranch pipeline to build multiple branches of the same origin? I don't see how adding multiple origins is a valid example.

acejam@gmail.com (JIRA)

unread,
Mar 16, 2018, 10:37:02 AM3/16/18
to jenkinsc...@googlegroups.com
Joshua Noble edited a comment on Bug JENKINS-42519
Isn't the whole point of a multibranch pipeline (within the context of the GitHub Branch Source plugin) to build multiple branches of the same origin? I don't see how adding multiple origins is a valid example.

stephen.alan.connolly@gmail.com (JIRA)

unread,
Mar 16, 2018, 3:22:01 PM3/16/18
to jenkinsc...@googlegroups.com

The original design of the API was that pull requests would be handled as a separate source in the same multibranch... similarly you can add forks of the same repo... now in retrospect it one could argue that it was a bad design decision to do it that way in 2013... but there are users who use the multiple sources, so we cannot remove it... which means a branch being deleted in one source does not mean the branch job should be deleted.

anton.doar@gmail.com (JIRA)

unread,
Nov 27, 2019, 1:20:04 AM11/27/19
to jenkinsc...@googlegroups.com
Anton Yurchenko updated an issue
 
Change By: Anton Yurchenko
Attachment: image-2019-11-27-08-19-32-533.png
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

anton.doar@gmail.com (JIRA)

unread,
Nov 27, 2019, 1:41:05 AM11/27/19
to jenkinsc...@googlegroups.com
Anton Yurchenko commented on Bug JENKINS-42519
 
Re: Closed pull requests, branch jobs are not cleaned up

Please re-open. We still encounter that issue on Jenkins 2.190.3.

 

/var/lib/jenkins/jobs/Projects/jobs/<project>/branches> $ du -h --max-depth=1 | grep "PR-"
11M	./PR-406
21M	./PR-403
11M	./PR-404
11M	./PR-405

(those sizes are small as we dynamically provision build nodes and nothing gets executed on master. much more space consumed on actual build nodes.)

 

"Scan Repository Now" cleans those PR's.

"Scan Repository Log":

 

Evaluating orphaned items in Project » <project>
Will remove PR-406
Will remove PR-405
Will remove PR-404
Will remove PR-403
Finished: SUCCESS

Relevant logs from /var/log/jenkins/jenkins.log:

 

 

2019-11-27 06:28:59.389+0000 [id=234]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/ct_<project>_PR-403@tmp on <slave-name>
2019-11-27 06:28:59.468+0000 [id=233]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/ct_<project>_PR-403@libs on master
2019-11-27 06:28:59.470+0000 [id=256]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/ct_<project>_PR-404@libs on master
2019-11-27 06:28:59.578+0000 [id=249]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/ct_<project>_PR-404@tmp on <slave-name>
2019-11-27 06:28:59.668+0000 [id=247]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/ct_<project>_PR-405@libs on master
2019-11-27 06:28:59.927+0000 [id=264]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/ct_<project>_PR-406@libs on master
2019-11-27 06:29:00.084+0000 [id=239]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/ct_<project>_PR-406@tmp on <slave-name>
2019-11-27 06:29:00.145+0000 [id=252]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/ct_<project>_PR-405@tmp on <slave-name>
2019-11-27 06:29:00.638+0000 [id=233]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/<omitted>_fix_recommendations@libs on master
2019-11-27 06:29:00.780+0000 [id=239]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/<omitted>_fix_recommendations@tmp on <slave-name>
2019-11-27 06:29:01.113+0000 [id=239]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/<omitted>_fix_uninstall@libs on master
2019-11-27 06:29:01.154+0000 [id=229]   INFO    j.b.MultiBranchProject$BranchIndexing#run: Project/<project> #20191127.062845 branch indexing action completed: SUCCESS in 15 sec
2019-11-27 06:29:01.259+0000 [id=264]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/<omitted>_fix_uninstall@tmp on <slave-name>

 

 

anton.doar@gmail.com (JIRA)

unread,
Nov 27, 2019, 1:42:05 AM11/27/19
to jenkinsc...@googlegroups.com
Anton Yurchenko edited a comment on Bug JENKINS-42519
Please re-open. We still encounter that issue on Jenkins 2.190.3.

Is there a proper workaround?

!image-2019-11-27-08-19-32-533.png|width=1190,height=174!

 
{noformat}

/var/lib/jenkins/jobs/Projects/jobs/<project>/branches> $ du -h --max-depth=1 | grep "PR-"
11M ./PR-406
21M ./PR-403
11M ./PR-404
11M ./PR-405{noformat}

(those sizes are small as we dynamically provision build nodes and nothing gets executed on master. much more space consumed on actual build nodes.)

 

"*Scan Repository Now*" cleans those PR's.

"*Scan Repository Log*":

 
{noformat}

Evaluating orphaned items in Project » <project>
Will remove PR-406
Will remove PR-405
Will remove PR-404
Will remove PR-403
Finished: SUCCESS{noformat}
Relevant logs from */var/log/jenkins/jenkins.log*:

 

 
{noformat}

2019-11-27 06:28:59.389+0000 [id=234]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/ct_<project>_PR-403@tmp on <slave-name>
2019-11-27 06:28:59.468+0000 [id=233]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/ct_<project>_PR-403@libs on master
2019-11-27 06:28:59.470+0000 [id=256]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/ct_<project>_PR-404@libs on master
2019-11-27 06:28:59.578+0000 [id=249]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/ct_<project>_PR-404@tmp on <slave-name>
2019-11-27 06:28:59.668+0000 [id=247]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/ct_<project>_PR-405@libs on master
2019-11-27 06:28:59.927+0000 [id=264]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/ct_<project>_PR-406@libs on master
2019-11-27 06:29:00.084+0000 [id=239]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/ct_<project>_PR-406@tmp on <slave-name>
2019-11-27 06:29:00.145+0000 [id=252]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/ct_<project>_PR-405@tmp on <slave-name>
2019-11-27 06:29:00.638+0000 [id=233]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/<omitted>_fix_recommendations@libs on master
2019-11-27 06:29:00.780+0000 [id=239]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/<omitted>_fix_recommendations@tmp on <slave-name>
2019-11-27 06:29:01.113+0000 [id=239]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/<omitted>_fix_uninstall@libs on master
2019-11-27 06:29:01.154+0000 [id=229]   INFO    j.b.MultiBranchProject$BranchIndexing#run: Project/<project> #20191127.062845 branch indexing action completed: SUCCESS in 15 sec
2019-11-27 06:29:01.259+0000 [id=264]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/<omitted>_fix_uninstall@tmp on <slave-name>{noformat}
 


 

anton.doar@gmail.com (JIRA)

unread,
Nov 27, 2019, 1:44:06 AM11/27/19
to jenkinsc...@googlegroups.com
Anton Yurchenko edited a comment on Bug JENKINS-42519
2019-11-27 06:29:00.638+0000 [id=233]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/<omitted> _fix_recommendations @libs on master
2019-11-27 06:29:00.780+0000 [id=239]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/<omitted>
_fix_recommendations @tmp on <slave-name>
2019-11-27 06:29:01.113+0000 [id=239]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/<omitted>
_fix_uninstall @libs on master

2019-11-27 06:29:01.154+0000 [id=229]   INFO    j.b.MultiBranchProject$BranchIndexing#run: Project/<project> #20191127.062845 branch indexing action completed: SUCCESS in 15 sec
2019-11-27 06:29:01.259+0000 [id=264]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/<omitted> _fix_uninstall @tmp on <slave-name>{noformat}
 

anton.doar@gmail.com (JIRA)

unread,
Nov 27, 2019, 1:53:07 AM11/27/19
to jenkinsc...@googlegroups.com
2019-11-27 06:29:00.638+0000 [id=233]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/<omitted>@libs on master
2019-11-27 06:29:00.780+0000 [id=239]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/<omitted>@tmp on <slave-name>
2019-11-27 06:29:01.113+0000 [id=239]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /var/lib/jenkins/workspace/<omitted>@libs on master

2019-11-27 06:29:01.154+0000 [id=229]   INFO    j.b.MultiBranchProject$BranchIndexing#run: Project/<project> #20191127.062845 branch indexing action completed: SUCCESS in 15 sec
2019-11-27 06:29:01.259+0000 [id=264]   INFO    j.b.WorkspaceLocatorImpl$Deleter$CleanupTask#run: deleting obsolete workspace /home/user/workspace/<omitted>@tmp on <slave-name>{noformat}
 
Space consumed on master before re-scanning all projects:
{noformat}
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1p1  150G   86G   65G  58% /var/lib/jenkins{noformat}
Space consumed on master after re-scanning all projects:
{noformat}
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1p1  150G   68G   83G  45% /var/lib/jenkins{noformat}
You may clearly see 20GB of junk in just 2 month on master node, when nothing gets build on it. All our pipeline Stages contain:
{noformat}
pipeline {
    agent none
    [...]    
Stages {
        [...]
        Stage('Build) {
            agent { label 'slave' }
            [...]
        }
        [...]
    }
}{noformat}
Much more wasted space on build nodes.

anton.doar@gmail.com (JIRA)

unread,
Nov 27, 2019, 1:53:07 AM11/27/19
to jenkinsc...@googlegroups.com
You may clearly see 20GB of junk in just 2 month on master node, when nothing gets build on it. All our pipeline Stages contain pipelines contains :

{noformat}
pipeline {
    agent none
    [...]    
Stages {
        [...]
        Stage('Build) {
            agent { label 'slave' }
            [...]
        }
        [...]
    }
}{noformat}
Much more wasted space on build nodes.

anton.doar@gmail.com (JIRA)

unread,
Nov 27, 2019, 1:54:02 AM11/27/19
to jenkinsc...@googlegroups.com
You may clearly see 20GB of junk in just 2 month on master node, when nothing gets build on it. All our pipelines contains:

{noformat}
pipeline {
    agent none
    [...]    
Stages {
        [...]
        Stage('Build) {
            agent { label 'slave' }
            [...]
        }
        [...]
    }
}{noformat}
Much more wasted space wasted on build nodes.
Reply all
Reply to author
Forward
0 new messages