Clearing external workspaces during build discard using Fingerprints and Workspace Cleanup plugin

[Benjamin Beggs]

I am trying to write a feature update for the Enhanced Old Build Discarder plugin that will trigger the Workspace Cleanup plugin to clear external workspaces set up using the External Workspace Manager plugin by aiming to have it that:

• When all builds that used a given external workspace are deleted, the workspace is deleted as well.
• This is determined using build Fingerprints, which contain data indicating the workspace(s) a build made use of.
• Once no build Fingerprints indicate use of this given workspace, the Workspace Cleanup plugin is triggered to clean it.

These conditions aim to avoid workspace bloat, whereby upstream job workspaces kept for reuse in downstream jobs will not require manual maintenance to avoid indefinite growth in large projects.

I have experimented with monitoring the fingerprints for every individual build, but this has so far been ineffective. It requires that every build in every job in a Jenkins instance be scanned and indexed after every build discard, until it finally reaches the point where no build fingerprints indicate the usage of a given workspace, at which point it can be cleared. This is very resource intensive.

The current implementation I am experimenting with scans all artifact fingerprints after every build discard until it is observed that a given artifact is no longer seen to have been used in a build. Once all the artifacts in workspace have this property, that workspace is cleared with the Workspace Cleanup plugin. This is at least less resource intensive than the previous implementation, though it will still be costly in projects with many artifacts.

Does this implementation seem sensible? Does anyone know of a better one? Also, does anyone know of an existing plugin that scans artifact fingerprint data, and ideally looks at the usage of that artifact? I am finding lacking precedent for these kinds of operations via my own code searches.

[martinda]

Hi Ben,

It is indeed expensive to run a scan of fingerprints at the end of every run or a project, esp. when, in the end, no workspace meets the criteria for discard.

I recently found about a plugin called purge job history plugin. It looks like this plugin adds an action to a project, so it would not run at the end of each build, it would run on-demand. So I am thinking that Instead of running a scan of all fingerprints at the end of every run of a job, it may be possible to run the scan at a pre-determined time or on-demand instead. It could even be triggered from a REST API, say once per day or once per week.

Martin

[martinda]

There is a recent thread about this purge job history plugin: https://groups.google.com/forum/#!msg/jenkinsci-dev/OxSNmlyP1dY/PkKPxkvwDgAJ

Martin

[Benjamin Beggs]

Great, thank you. I was rethinking an implementation for this and I think the following may be most sensible:
Scan all artifact fingerprints after a regular time interval. If an artifact has no build usage history, it is deleted. If the working directory of the artifact is left empty after this, it is deleted as well.

Wouldn’t this in effect cleanup workspaces? If so it may be doable without a dependency on Workspace Cleanup plugin....

[martinda]

So you are thinking of letting the build discarder plugin do its job, and asynchronously to that, scan the fingerprints to find workspaces, and delete workspaces that no longer have builds. I think this would work.

Martin

[Benjamin Beggs]

I think this is the most resource efficient way of doing this.....I will experiment. It will function separate to the build discard process. It doesn't seem resource efficient to attach the feature to build discards, as far as I can see.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/5vp_JBye3Tk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/de33e71a-3378-4880-9e5b-e2801bf222f7%40googlegroups.com.