‘Discard Old Items’ not removing all directories for multi-branch pipeline

[james....@secondmind.ai]

I have a multi-branch pipeline with ‘Discard old Items’ set up to get rid of old PRs, but I’ve noticed that the directories aren’t all being removed completely from the controller’s filesystem. These directories have no confg.xml and they disappear from the UI but it leads to warning messages in the log saying the config.xml couldn’t be found for the PR job. They all seem to have the remnants of a single build that wasn’t removed correctly, which includes a directory named ‘workflow-fallback’. What remains looks like:

$JENKINS_HOME/jobs/<my job>/branches/<old pr>/builds/<build id>/

    build.xml

    workflow-fallback/

         <number>.xml

This workflow-fallback directory doesn’t currently exist for any build of a PR that hasn’t been removed, so I’m unsure where it come from. Given the time stamps of these directories this was happening over a wide span of versions up to and including 2.401.2.

Can anyone explain what’s happening and is there anything I can do to prevent this?

[jn...@cloudbees.com]

Looking at the code it appears as though the pipeline was running but when Jenkins was restarted it could not find its data. (https://github.com/jenkinsci/workflow-cps-plugin/blob/a6ac2775ac1767f68ddabcff14364235a61e8bdf/plugin/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L659)

Perhaps you had some dirty shutdown of Jenkins and are using a non survivable performace strategy rather than  MAX_SURVIVABILITY   ? https://www.jenkins.io/doc/book/pipeline/scaling-pipeline/

/James

[James Robson]

Looking at the timestamps and the logs, most but not all of these 'workflow-fallback' directories get created after reapplying Configuration as Code not after a restart and it doesn't look like the builds are being interrupted. Though for some reason there's a large delay between the original build and the attempt at restarting it. For example one build I looked at had succeeded at 2023-07-06 13:50 GMT, but the attempt to restart came on 2023-07-14 despite the configuration as code being applied every day. I also don't see any log messages that refer to one of the affected builds between it's completion and the attempt to restart it so I can't see what changed to trigger the restart attempt.

We are also using 'Max survivability' as the default and don't override this anywhere.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/Kf5_PUHFBBM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/91073651-8d0d-4260-9de2-fac469cbd963n%40googlegroups.com.