4 month old deleted branches still not removed

[Michael Neale]

Using multibranch and git/github, I am seeing projects that are created for a branch be apparently marked for deletion, but never removed. 

There is one that hasn't been around for 4 months and shows up like this: 

Is there something wrong with my config that makes it retain this in this fashion? Shouldn't it be deleted? 

https://stackoverflow.com/questions/44419841/jenkins-multi-branch-pipeline-not-pruning-branches-deleted-from-remote

[Mark Waite]

There is a configuration setting, "Orphaned Item Strategy", in the multi-branch pipeline job definition and in the GitHub Organization Folders definition which sets a retention period for jobs associated with deleted branches.  Mine is set to Discard old items, with a retention period of 0 days.

Is theirs enabled?

Does it have the retention period they want?

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/094aab14-3cc4-43a9-92d0-c5ca1b796ea6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Neale]

Yes I suspected that - it is the default that you get when you create a new project (see below screen shot). Is this is what yours is? 

This multibranch project has been around for a while - and it only recently started showing this behavior (recently as 4 months) so I suspect something changed in a plugin version... 

[Mark Waite]

Yes, that's how mine is configured.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e885e143-f109-4937-97c3-bda7f51faedc%40googlegroups.com.

[Michael Neale]

and obviously you aren't seeing deleted branches still show up without being cleaned up? 

[Mark Waite]

Correct.  I have some test jobs which intentionally create and destroy branches as part of their testing, and those branches correctly disappear shortly after the branch deletion is detected.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/45b529a1-edd0-43ae-82bb-f4c57a7e89c4%40googlegroups.com.

[Michael Neale]

github? polling or webhooks? 

The "scan respotory log" doesn't show anything recent (but new branches are still being built) - does yours show anything different? 

This seems a recent thing (well a change in last few months) or else we would have > 4 month old dead branches. 

[Stephen Connolly]

On Wed 21 Jun 2017 at 06:17, Michael Neale <mne...@cloudbees.com> wrote:

github? polling or webhooks? 

The "scan respotory log" doesn't show anything recent (but new branches are still being built) - does yours show anything different? 

How often is a full scan?

Orphaned items are only removed on a full scan.

New items are added by events, so adding new branches will not be a sign that you are getting full scans

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2b4c91a8-1819-431a-84a3-3e21566d92f7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Sent from my phone

[Michael Neale]

On Wednesday, June 21, 2017 at 3:50:06 PM UTC+10, Stephen Connolly wrote:

How often is a full scan?

Is that is from the scan log link - then not for 2 months or so. What triggers that? All the settings are stock. 

 

Orphaned items are only removed on a full scan. 

New items are added by events, so adding new branches will not be a sign that you are getting full scans

What decides how often there is a full scan?  

[Stephen Connolly]

On Wed 21 Jun 2017 at 06:54, Michael Neale <mne...@cloudbees.com> wrote:

On Wednesday, June 21, 2017 at 3:50:06 PM UTC+10, Stephen Connolly wrote:

How often is a full scan?

Is that is from the scan log link - then not for 2 months or so. What triggers that? All the settings are stock. 

Periodic triggers

you want "periodically unless otherwise run" to be something like once a day/week depending on how long you can handle a missed event (which should be unlikely)

I recommend somewhere between 1 day and 2 weeks

 

Orphaned items are only removed on a full scan. 

New items are added by events, so adding new branches will not be a sign that you are getting full scans

What decides how often there is a full scan?  

User configuration 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/25632742-1301-4478-9d8d-7178d496e876%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Stephen Connolly]

On Wed 21 Jun 2017 at 07:18, Stephen Connolly <stephen.al...@gmail.com> wrote:

On Wed 21 Jun 2017 at 06:54, Michael Neale <mne...@cloudbees.com> wrote:

On Wednesday, June 21, 2017 at 3:50:06 PM UTC+10, Stephen Connolly wrote:

How often is a full scan?

Is that is from the scan log link - then not for 2 months or so. What triggers that? All the settings are stock. 

Periodic triggers

you want "periodically unless otherwise run" to be something like once a day/week depending on how long you can handle a missed event (which should be unlikely)

I recommend somewhere between 1 day and 2 weeks

 

Orphaned items are only removed on a full scan. 

New items are added by events, so adding new branches will not be a sign that you are getting full scans

What decides how often there is a full scan?  

User configuration 

Was this job configuration created by BO?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/25632742-1301-4478-9d8d-7178d496e876%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sent from my phone

[Michael Neale]

On Wednesday, June 21, 2017 at 4:19:58 PM UTC+10, Stephen Connolly wrote:

Periodic triggers

you want "periodically unless otherwise run" to be something like once a day/week depending on how long you can handle a missed event (which should be unlikely)

I recommend somewhere between 1 day and 2 weeks

Ah gotcha. 

 

 

Was this job configuration created by BO?

No - but I will check what it does and what default does - as the default should be to do 1 to 2 weeks right?  

[Michael Neale]

OK on looking further - MBPs created by classic don't have this set - which seems wrong - worthy of a ticket? 

Ones created by blue ocean do have it set (1 day, as it turns out) via the github org folder. 

So this explains it, but leaves the issue of the default for creation being wrong. 

to be clear it is the following right? 

[Michael Neale]

reading the docs for folders, perhaps there is a bug from a bad assumption. a github event will not trigger a scan like the author of the folders plugin implies.... so it still needs to have this set even for github

[Stephen Connolly]

On 21 June 2017 at 00:19, Michael Neale <mne...@cloudbees.com> wrote:

On Wednesday, June 21, 2017 at 4:19:58 PM UTC+10, Stephen Connolly wrote:

Periodic triggers

you want "periodically unless otherwise run" to be something like once a day/week depending on how long you can handle a missed event (which should be unlikely)

I recommend somewhere between 1 day and 2 weeks

Ah gotcha. 

 

 

Was this job configuration created by BO?

No - but I will check what it does and what default does - as the default should be to do 1 to 2 weeks right?  

for GitHub / Bitbucket, their events are reasonably reliable, so I think 1 week would be fine... 1 day also

for plain Git, it can be harder for people to set up webhooks, so I would think 8h or 1 day as the default

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/25632742-1301-4478-9d8d-7178d496e876%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sent from my phone

--

Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6198792c-e664-40f8-95c3-f3650ba74d38%40googlegroups.com.

[Stephen Connolly]

On 21 June 2017 at 01:04, Michael Neale <mne...@cloudbees.com> wrote:

reading the docs for folders, perhaps there is a bug from a bad assumption. a github event will not trigger a scan like the author of the folders plugin implies.... so it still needs to have this set even for github

Link to the docs please. (would like to correct those... probably want a JIRA for that anyway)

The intent is that periodically unless otherwise run should be acting as a backup... can probably make it default to on. Can you file a JIRA

 

On Wednesday, June 21, 2017 at 6:03:03 PM UTC+10, Michael Neale wrote:

OK on looking further - MBPs created by classic don't have this set - which seems wrong - worthy of a ticket? 

Ones created by blue ocean do have it set (1 day, as it turns out) via the github org folder. 

So this explains it, but leaves the issue of the default for creation being wrong. 

to be clear it is the following right? 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/70f52071-5f57-4205-835b-8bb27c21d1ba%40googlegroups.com.

[Michael Neale]

Mark do you have a scan interval set? 

Should the github events include removing of branch projects? as that doesn't seem to be happening lately (since a few months). 

On Wednesday, June 21, 2017 at 6:13:32 PM UTC+10, Stephen Connolly wrote:

On 21 June 2017 at 00:19, Michael Neale <mne...@cloudbees.com> wrote:

On Wednesday, June 21, 2017 at 4:19:58 PM UTC+10, Stephen Connolly wrote:

Periodic triggers

you want "periodically unless otherwise run" to be something like once a day/week depending on how long you can handle a missed event (which should be unlikely)

I recommend somewhere between 1 day and 2 weeks

Ah gotcha. 

 

 

Was this job configuration created by BO?

No - but I will check what it does and what default does - as the default should be to do 1 to 2 weeks right?  

for GitHub / Bitbucket, their events are reasonably reliable, so I think 1 week would be fine... 1 day also

for plain Git, it can be harder for people to set up webhooks, so I would think 8h or 1 day as the default

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/25632742-1301-4478-9d8d-7178d496e876%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sent from my phone

--

Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

[Stephen Connolly]

On 21 June 2017 at 01:16, Michael Neale <mne...@cloudbees.com> wrote:

Mark do you have a scan interval set? 

Should the github events include removing of branch projects? as that doesn't seem to be happening lately (since a few months). 

So the multibranch API does not allow us to know if an event about a branch being removed from one source means that the job is now orphaned.

You can have multiple sources, in which case removing the branch in one source may mean that a lower-priority source now claims ownership of the branch.

If we have one and only one source (ACK that this is likely the majority case) then and only then can we know that an event removing a branch has orphaned an item... but even that doesn't get us far.

The orphaned item strategy may say keep the last 10... so now, in an event we actually have to go and collect all the currently orphaned items, sort them by last active and then trim the oldest... which makes the event code cease being single responsibility as the branch the event was for may or may not correspond to the job being removed... perhaps the solution is that we limit the event to deleting its own job if it is not in the last 10...

Oh but it gets worse... the strategy mat say keep for at least 7 days... so now, in an event, we should not delete the orphaned item, rather we should queue up a task to run in 7 days time that would then delete the item (not quite true, the current implementation uses the time of last build... but it should be the time of last event, and I will probably fix that soon)

So in reality, you just need to run the periodic indexing... we could tweak that periodic indexing to have a two layer support, e.g. if there are orphaned items or no events then run every 8h otherwise run every 7 days... the UI would need thinking but a two layer support might not be a bad idea

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/45dd6d78-e268-480c-af88-d70e9c279695%40googlegroups.com.

[Michael Neale]

I think it is fine to expect some infrequent scanning - it probably needs to be the default though as otherwise people wire up a webhook - and end up never deleting branches.