Reasons for workspace cleanup

[Fabian Cenedese]

Hi

I have a task that needs a lot of time. To not occupy Jenkins for
days I made it like this that the task is started via cron on the
agent, without Jenkins. In the end a Jenkins job is started to
just get the results which is a matter of seconds.
My problem is now that sometimes the workspace on the node
is completely wiped, sometimes even while the task is running
which of course then makes it fail. But I couldn't find a reason
why this would happen. It's exactly the workspace as defined
in the Jenkins job that is cleared, not just the (sub)directory
where my task is running.
What are the reasons that Jenkins might do that? I've seen it
happen before checking out from a version control system if
the workspace was not yet a checkout. But there's no checking
out in the job anymore. Could it be that if the connection to the
node is lost and reestablished that some cleaning or setting
up is done? Are there any other reasons that Jenkins (agent)
might clean the workspace, completely out of sync with any
running job?

Thanks

bye Fabi

[Dirk Heinrichs]

Am Dienstag, dem 05.04.2022 um 08:35 +0200 schrieb Fabian Cenedese:

My problem is now that sometimes the workspace on the node

is completely wiped, sometimes even while the task is running

which of course then makes it fail. But I couldn't find a reason

why this would happen.

There are various places in a job's configuration where workspace cleanup can be enabled, partly depending on installed plugins, for example:

• The SCM section
• Under "Build environment": Delete workspace before build starts
• Under "Post-build Actions": Delete workspace when build is done
• Might also be something in your build steps
• Workspaces can also be wiped manually from the job's page
• We have special cleanup jobs in place that periodically remove all workspaces
• ...

HTH...

Dirk

-- 

Dirk Heinrichs

Senior Systems Engineer, Delivery Pipeline

OpenText ™ Discovery | Recommind

Phone: +49 2226 15966 18

Email: dhei...@opentext.com

Website: www.recommind.de

Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach

Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, Registernummer HRB 10646

This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.

[Fabian Cenedese]

At 08:55 05.04.2022, you wrote:

>Am Dienstag, dem 05.04.2022 um 08:35 +0200 schrieb Fabian Cenedese:
>
>>My problem is now that sometimes the workspace on the node
>>is completely wiped, sometimes even while the task is running
>>which of course then makes it fail. But I couldn't find a reason
>>why this would happen.
>
>There are various places in a job's configuration where workspace cleanup can be enabled, partly depending on installed plugins, for example:
>

> * The SCM section
> * Under "Build environment": Delete workspace before build starts
> * Under "Post-build Actions": Delete workspace when build is done
> * Might also be something in your build steps
> * Workspaces can also be wiped manually from the job's page
> * We have special cleanup jobs in place that periodically remove all workspaces
Thanks for the ideas. However I think none of it applies here.
- I've removed all references to git and svn as I also assumed this to be the cause.
- It happens even when no job is running/has been running so the config
shouldn't matter here. But I have nothing of that sort anyway.
- I don't have anything deleting in my script
- As it also happened in the middle of the night where nobody is working
I rule manual out as well (and Jenkins is not accessible from outside)
- Are you talking about your own self-made cleanup jobs or is that
something builtin into Jenkins?

I checked Jenkins logfiles, system logs, virus scanners, job history...
but I still can't find a reason. It's also arbitrary, I couldn't see a pattern yet.

Thanks

bye Fabi

[Dirk Heinrichs]

Am Dienstag, dem 05.04.2022 um 09:38 +0200 schrieb Fabian Cenedese:

Are you talking about your own self-made cleanup jobs or is that

  something builtin into Jenkins?

Self-made.

Bye...

[Daniel Beck]

Workspaces Jenkins considers "unused" are wiped periodically. While this is intended to be safe, i.e., never delete the last workspace, or workspaces of running builds, it's not impossible that "active" workspaces are caught in rare circumstances. Check the workspace cleanup log in your Jenkins home to see whether this happens to you, or enable detailed (FINE) logging for the logger "hudson.model.WorkspaceCleanupThread".

(Code reference in case you're curious: https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/WorkspaceCleanupThread.java )

[Fabian Cenedese]

At 13:53 11.04.2022, you wrote:

>Workspaces Jenkins considers "unused" are wiped periodically. While this is intended to be safe, i.e., never delete the last workspace, or workspaces of running builds, it's not impossible that "active" workspaces are caught in rare circumstances. Check the workspace cleanup log in your Jenkins home to see whether this happens to you, or enable detailed (FINE) logging for the logger "hudson.model.WorkspaceCleanupThread".

Thanks for the info, that's what seems to be happening. I'll see if I can
check this in case it happens again. Is this something that can be
influenced in the settings or is it completely automatic? Would I need
to run a job more often to prevent this? Or I could put all my files in
a different directory outside the workspace and change into it in
the job. That way there's nothing to delete in the configured workspace.

Thanks

bye Fabi

[Daniel Beck]

This is completely automated and invisible beyond log entries (probably not great). While there are some possible approaches to stop Jenkins from doing that when it occurs with regular Jenkins jobs, your situation is different. In your particular case, you're using what amounts to a Jenkins-managed resource outside Jenkins. The cleanest solution is what you suggest: to stop doing that, run your cron job somewhere else, and limit Jenkins workspace use to actual Jenkins jobs.

[Fabian Cenedese]

Well, I do use Jenkins to collect the results of my otherwise started script,
keep the history, send emails and so on. But I understand that it knows
nothing of it until the job is started to get the results.
I'll rearrange my layout to have s separate directory for Jenkins.

Thanks for your help.

bye Fabi

[mar...@gmail.com]

When starting Jenkins you can change the default for the  hudson.model.WorkspaceCleanupThread.retainForDays which is 30 days.  Search for the property in Jenkins Features Controlled with system properties. What I have done is added a cron trigger to run the job at least every 20 days and when started by timer I simply exit. My job is a pipeline but freestylejob also have periodic triggering capabilities.