[JIRA] (JENKINS-53579) Asynchronous cleanup not removing renamed workspace directories on slaves

13 views
Skip to first unread message

bochenski.kuba+jenkins@gmail.com (JIRA)

unread,
Sep 14, 2018, 6:13:03 AM9/14/18
to jenkinsc...@googlegroups.com
Jakub Bochenski created an issue
 
Jenkins / Bug JENKINS-53579
Asynchronous cleanup not removing renamed workspace directories on slaves
Issue Type: Bug Bug
Assignee: Oliver Gondža
Attachments: Screen Shot 2017-01-09 at 3.46.57 PM.png
Components: ws-cleanup-plugin
Created: 2018-09-14 10:12
Environment: Jenkins 1.579
ws-cleanup 0.24
Priority: Major Major
Reporter: Jakub Bochenski

After upgrading to ws-cleanup 0.24 in order to get the asynchronous cleanup function we noticed the workspaces on our slaves getting renamed to the form of ${WORKSPACE}ws-cleanup${TIMESTAMP}. (ie, job1 would become job1_ws-cleanup_1411197183394). The expected behavior under ws-cleanup 0.24 is that these were temporary to support asynchronous processing and would be deleted. However, these directories never get removed from the slave. Over time, the slave hard drives filled up resulting in build failures.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

bochenski.kuba+jenkins@gmail.com (JIRA)

unread,
Sep 14, 2018, 6:14:01 AM9/14/18
to jenkinsc...@googlegroups.com
Jakub Bochenski updated an issue
Change By: Jakub Bochenski
Environment: Jenkins 1.579
 ws-cleanup 0. 24 34

bochenski.kuba+jenkins@gmail.com (JIRA)

unread,
Sep 14, 2018, 7:09:01 AM9/14/18
to jenkinsc...@googlegroups.com
Jakub Bochenski updated an issue
Change By: Jakub Bochenski
Environment: Jenkins ver. 2.121.3
 ws-cleanup 0.34

bochenski.kuba+jenkins@gmail.com (JIRA)

unread,
Sep 14, 2018, 7:12:01 AM9/14/18
to jenkinsc...@googlegroups.com
Jakub Bochenski updated an issue
After upgrading to ws-cleanup 0.24 in order to get the asynchronous cleanup function we We noticed the workspaces on our slaves getting renamed to the form of ${WORKSPACE}_ws-cleanup_${TIMESTAMP}.  (ie, job1 would become job1_ws-cleanup_1411197183394).   The expected behavior under ws-cleanup 0.24 is that these were temporary to support asynchronous processing and would be deleted.  However, these directories never get removed from the slave.  Over time, the slave hard drives filled up resulting in build failures.

o.perepelytsya@gmail.com (JIRA)

unread,
Nov 13, 2018, 5:00:02 AM11/13/18
to jenkinsc...@googlegroups.com
Oleksandr Perepelytsya commented on Bug JENKINS-53579
 
Re: Asynchronous cleanup not removing renamed workspace directories on slaves

We're experiencing same issue on Jenkins - v. 2.138.2 / WS-Cleanup plugin - v. 0.34

collin.reed@nteligen.com (JIRA)

unread,
Mar 6, 2019, 5:17:03 PM3/6/19
to jenkinsc...@googlegroups.com
Collin R commented on Bug JENKINS-53579

Same here Jenkins v. 2.140 / WS-Cleanup plugin v. 0.34. Seems that the ${WORKSPACE}ws-cleanup${TIMESTAMP} directory is somehow owned by root and therefore causes the async resource disposer to fail due to lack of permissions. Workaround is to add a cron job to chown ${WORKSPACE}ws-cleanup${TIMESTAMP} which allows async cleanup to occur.

collin.reed@nteligen.com (JIRA)

unread,
Mar 6, 2019, 5:17:08 PM3/6/19
to jenkinsc...@googlegroups.com
Collin R edited a comment on Bug JENKINS-53579
Same here on Jenkins v. 2.140 / WS-Cleanup plugin v. 0.34. Seems that the {{${WORKSPACE}_ws-cleanup_$\{TIMESTAMP}}} directory is somehow owned by root and therefore causes the async resource disposer to fail due to lack of permissions. Workaround is to add a cron job to chown {{${WORKSPACE}_ws-cleanup_$\{TIMESTAMP}}} which allows async cleanup to occur.

collin.reed@nteligen.com (JIRA)

unread,
Mar 6, 2019, 5:18:03 PM3/6/19
to jenkinsc...@googlegroups.com

collin.reed@nteligen.com (JIRA)

unread,
Mar 6, 2019, 5:18:08 PM3/6/19
to jenkinsc...@googlegroups.com
Collin R edited a comment on Bug JENKINS-53579
Same here on Jenkins v. 2.140 / WS-Cleanup plugin v. 0.34. Seems that the ${WORKSPACE} \ _ws- cleanup_ cleanup\_ $\{TIMESTAMP} directory is somehow owned by root and therefore causes the async resource disposer to fail due to lack of permissions. Workaround is to add a cron job to chown ${WORKSPACE} \ _ws- cleanup_ cleanup\_ $\{TIMESTAMP} which allows async cleanup to occur.

bochenski.kuba+jenkins@gmail.com (JIRA)

unread,
Mar 7, 2019, 9:09:10 AM3/7/19
to jenkinsc...@googlegroups.com
Jakub Bochenski updated an issue

On second thoughts I think this is an improvement, not a defect. Nobody promised async cleanup will handle ownership errors etc.
It would be nice to have support for it, but any solution will be hacky and
Change By: Jakub Bochenski
Issue Type: Bug Improvement

ogondza@gmail.com (JIRA)

unread,
Mar 11, 2019, 3:03:02 AM3/11/19
to jenkinsc...@googlegroups.com
Oliver Gondža commented on Improvement JENKINS-53579
 
Re: Asynchronous cleanup not removing renamed workspace directories on slaves

Agreed, in general, Jenkins does not promis it will delete anything not owned by the user running the agent. The correct course of action is to see, why are those directories owned by other users.

michaelaervin@lavorotechnologies.com (JIRA)

unread,
Aug 31, 2019, 12:23:03 PM8/31/19
to jenkinsc...@googlegroups.com

Why are the ${WORKSPACE}ws-cleanup${TIMESTAMP} directories owned by someone else?  AFAIK jenkins creates them so it should delete them.

michaelaervin@lavorotechnologies.com (JIRA)

unread,
Aug 31, 2019, 12:26:02 PM8/31/19
to jenkinsc...@googlegroups.com
Michaela Ervin edited a comment on Improvement JENKINS-53579
Why are the ${WORKSPACE}_ws-cleanup_${TIMESTAMP} directories owned by someone else?  AFAIK jenkins creates them so it should delete them.

 

Edit: I see, our scripts run some commands with sudo and thus stuff is owned by root.

Maybe I just need to run `chown -R jenkins:jenkins` on the directory at the end of my script.

michaelaervin@lavorotechnologies.com (JIRA)

unread,
Aug 31, 2019, 12:29:02 PM8/31/19
to jenkinsc...@googlegroups.com
Michaela Ervin edited a comment on Improvement JENKINS-53579
Why are the ${WORKSPACE}_ws-cleanup_${TIMESTAMP} directories owned by someone else?  AFAIK jenkins creates them so it should delete them.

 

Edit: I see, our scripts run some commands with sudo and thus stuff is owned by root.

Maybe I just need to run `chown -R jenkins:jenkins` on the directory at the end of my script.


 

What about an option to "try" to run the cleanup as root?

mark.earl.waite@gmail.com (JIRA)

unread,
Aug 31, 2019, 12:40:02 PM8/31/19
to jenkinsc...@googlegroups.com

I hope for security purposes that the Jenkins agent process cannot become the root user as a general pattern. I don't think the plugin has any way to ask the operating system to promote the cleanup operation to root permission. Even if it did, I don't think it would be wise to make such a request, since agents that can easily become the root user can then also be used to harm the operating system that is hosting the agent

michaelaervin@lavorotechnologies.com (JIRA)

unread,
Aug 31, 2019, 2:03:03 PM8/31/19
to jenkinsc...@googlegroups.com

Understandable!

 

I added the following to the end of my build script to test changing the perms.

 

LOGGER.info('Workspace directory is: {}'.format(workspace)) 
system('cd "{}" && sudo chown -R jenkins:jenkins *'.format(workspace))

It seems to have worked except a file still existed that I didn't realize was getting created.  Running again to see if it works.

michaelaervin@lavorotechnologies.com (JIRA)

unread,
Aug 31, 2019, 2:03:12 PM8/31/19
to jenkinsc...@googlegroups.com
Michaela Ervin edited a comment on Improvement JENKINS-53579
Understandable!

 

I added the following to the end of my build script to test changing the perms.

 
{code:java}
 workspace = getcwd()
LOGGER.info('Workspace directory is: {}'.format(workspace))
system('cd "{}" && sudo chown -R jenkins:jenkins *'.format(workspace))
{code}

It seems to have worked except a file still existed that I didn't realize was getting created.  Running again to see if it works.

gilles.dartiguelongu@rakuten.com (JIRA)

unread,
Apr 16, 2020, 6:57:05 AM4/16/20
to jenkinsc...@googlegroups.com

For some reason, here the problems appears on jobs that are using Docker images running root user inside. I would expect Jenkins to establish a permission mapping between its euid and container's root. What is even stranger is that not all files created in the relevant steps are surviving.
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

bochenski.kuba+jenkins@gmail.com (JIRA)

unread,
Apr 16, 2020, 11:16:05 AM4/16/20
to jenkinsc...@googlegroups.com

> I would expect Jenkins to establish a permission mapping between its euid and container's root.

Can you suggest a technical solution to this. It would be useful but I see no robust and easy way to do it

bochenski.kuba+jenkins@gmail.com (JIRA)

unread,
Apr 16, 2020, 11:18:06 AM4/16/20
to jenkinsc...@googlegroups.com
Jakub Bochenski edited a comment on Improvement JENKINS-53579
> I would expect Jenkins to establish a permission mapping between its euid and container's root.

Can you suggest a technical solution to this. It would be useful but I see no robust and easy way to do it
Reply all
Reply to author
Forward
0 new messages