[JIRA] (JENKINS-43269) workspace cleanup removes workspaces for running job

[alexey.bychko@percona.com (JIRA)]

Alexey Bychko created an issue

Jenkins / JENKINS-43269 workspace cleanup removes workspaces for running job Issue Type: Bug Assignee: Oliver Gondža Components: ws-cleanup-plugin Created: 2017/Mar/31 12:03 PM Environment: Jenkins ver. 2.32.3, workspace cleanup plugin 1.0.5 Priority: Critical Reporter: Alexey Bychko if we run a lot of copies of the same job with workspace cleanup enabled - every next job will clean up workspaces on slaves with already running jobs. jobs are executed in parallel, so perhaps it's better to check if workspace is used currently and belongs to running job before the cleanup. it's important for us because we have ENOSPACE with cleanup disabled and many failed jobs with cleanup enabled. running job copies one-by-one is not an option for us, because it will take a lot of time.   thanks in advance Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

[ogondza@gmail.com (JIRA)]

Oliver Gondža commented on JENKINS-43269

Re: workspace cleanup removes workspaces for running job This certainly should not happen. Please provide more information about the situation (reproducer ideally) as well as plugin version (ws-cleanup was never released in version 1.0.5 as stated). Do you use custom workspace on these jobs? Are you running parallel build on the same machine? Do you share the drive with workspaces between machines? Any erorrs/logs somewhere? Are the workspaces cleaned asynchronously? How do you know it is ws-cleanup that is deleting those directories?

Add Comment

[alexey.bychko@percona.com (JIRA)]

Alexey Bychko updated an issue

Jenkins / JENKINS-43269

workspace cleanup removes workspaces for running job

Change By: Alexey Bychko Attachment: Screen Shot 2017-03-31 at 15.27.28.png

Add Comment

[alexey.bychko@percona.com (JIRA)]

Alexey Bychko commented on JENKINS-43269

Re: workspace cleanup removes workspaces for running job we don't have any custom workspaces no. we have a lot of slaves, every slave is able to build the job no yes, from failed jobs. we have cleanup disabled at the moment, so no fresh logs. usually it looks like {code}+ mv /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/centos5-64/mysql-test/test_results.subunit /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/centos5-64 mv: cannot stat `/mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/centos5-64/mysql-test/test_results.subunit': No such file or directory{code} suddenly workspace disappears. workspaces set to be cleaned up at the start of build. so, it looks like next started copy will destroy already running jobs because such a failures disappeared after disabling the cleanup

Add Comment

[ogondza@gmail.com (JIRA)]

Oliver Gondža updated an issue

Jenkins / JENKINS-43269

workspace cleanup removes workspaces for running job

Change By: Oliver Gondža Component/s: hudson-wsclean-plugin Component/s: hudson-pview

Add Comment

[ogondza@gmail.com (JIRA)]

Oliver Gondža updated an issue

Jenkins / JENKINS-43269 workspace cleanup removes workspaces for running job

Change By: Oliver Gondža Component/s: hudson-pview Component/s: ws-cleanup-plugin

Add Comment

[ogondza@gmail.com (JIRA)]

Oliver Gondža commented on JENKINS-43269

Re: workspace cleanup removes workspaces for running job Uh, that is for https://wiki.jenkins-ci.org/display/JENKINS/Distributed+Workspace+Clean+plugin. I have fixed the component name and the wiki page so users can find what the component name is.

Add Comment

[ogondza@gmail.com (JIRA)]

Oliver Gondža assigned an issue to tspengler

Jenkins / JENKINS-43269

workspace cleanup removes workspaces for running job

Change By: Oliver Gondža Assignee: Oliver Gondža tspengler

Add Comment

[alexey.bychko@percona.com (JIRA)]

Alexey Bychko updated an issue

Jenkins / JENKINS-43269 workspace cleanup removes workspaces for running job

Change By: Alexey Bychko Attachment: Screen Shot 2017-03-31 at 15.42.40.png

Add Comment

[pjdarton@gmail.com (JIRA)]

pjdarton commented on JENKINS-43269

Re: workspace cleanup removes workspaces for running job I've hit this issue before myself. I think it's down to this line where it's only considering the build which is running that build step and not considering the possibility that Jenkins might be running multiple runs of that build job in parallel on different slave nodes. The repro case is fairly simple: Have multiple static slave nodes. Define a job with the "allow concurrent execution" flag set so you can have two (or more) builds of that job running at the same time, and ensure that the "restrict where build can run" labels are set to allow it to run on more than one of the static slave nodes. Trigger a build, wait until it's running, then trigger another. That'll result in two builds running in parallel, one on one slave node, one on another. When the first build completes, it'll run the cleanup phase and try to delete the workspace that's currently in use by the second build.

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[pjdarton@gmail.com (JIRA)]

pjdarton assigned an issue to pjdarton

Jenkins / JENKINS-43269

workspace cleanup removes workspaces for running job

Change By: pjdarton Assignee: tspengler pjdarton

Add Comment

[pjdarton@gmail.com (JIRA)]

pjdarton commented on JENKINS-43269

Re: workspace cleanup removes workspaces for running job I've created a PR that'll address this issue ... but until the plugin code has a Jenkinsfile (see PR#5) folks will have to build their own copy of the code.

Add Comment

[pjdarton@gmail.com (JIRA)]

pjdarton started work on JENKINS-43269

Change By: pjdarton Status: Open In Progress

Add Comment

[pjdarton@gmail.com (JIRA)]

pjdarton commented on JENKINS-43269

Re: workspace cleanup removes workspaces for running job

FAO tspengler and/or Arnaud Héritier - you're both listed (in the pom.xml file) as developers of this plugin - could you take a look at the PRs I've raised?

Add Comment

[pjdarton@gmail.com (JIRA)]

pjdarton updated an issue

Jenkins / JENKINS-43269

workspace cleanup removes workspaces for running job

Change By: pjdarton Attachment: hudson-wsclean-plugin.hpi

Add Comment

[aheritier@apache.org (JIRA)]

Arnaud Héritier commented on JENKINS-43269

Re: workspace cleanup removes workspaces for running job pjdarton I'm not active anymore on this plugin and many others. I won't have the time to look at it before at least a month. Could you consider to become a maintainer of this plugin ? I will +1 your candidature

Add Comment

[pjdarton@gmail.com (JIRA)]

pjdarton commented on JENKINS-43269

Re: workspace cleanup removes workspaces for running job

Alexey Bychko I've uploaded an enhanced hudson-wsclean-plugin.hpi file. Try ticking the "use history" check-box (in the "Advanced" config). That'll change the plugin's behavior from "hit all slaves that could run this project except the one running this build right now" to "hit all slaves that previously ran this build except any running this build right now". That should allow you to have concurrent builds without a problem. Note: By default, it'll continue to use the old "hit all slaves that could've run this project" logic which is largely unchanged and will still bork a concurrent build - you'll need to select the new behavior in your job config. If you still have issues, create a log (manage jenkins -> system log) that logs de.jamba.hudson.plugin.wsclean at level "ALL", collect logs from that to see what it's doing wrong (as this code now logs what it's doing to the Jenkins logger as well as to the builds), and let me know further details.

Add Comment

[pjdarton@gmail.com (JIRA)]

pjdarton commented on JENKINS-43269

Re: workspace cleanup removes workspaces for running job

Arnaud Héritier I guess I could, although I'd prefer not to stay a maintainer long-term (I seem to adopt them much faster than I can let them go, and my boss would prefer me to get on with other things...) So, sure, I'll take over long enough to kick out a new release with the functionality I want to add, but I don't want to stay involved if I can avoid doing so In the meantime, can you merge PR#5 ? It should just be a simple button-click to do that and that'll then get the jenkinsci build engine to notice the plugin and start doing CI builds of the PRs...

Add Comment

[pjdarton@gmail.com (JIRA)]

pjdarton commented on JENKINS-43269

Re: workspace cleanup removes workspaces for running job

Arnaud Héritier I've requested access. While I've linked back to your earlier approval in the comments here, if you could "approve" that PR then it may help speed things along. Similarly, if you can grant write access to the github area then that would also be helpful.

Add Comment

[pjdarton@gmail.com (JIRA)]

pjdarton commented on JENKINS-43269

Re: workspace cleanup removes workspaces for running job

Update for anyone watching this: There's now a Pull Request that contains a fix for this issue (plus other enhancements). Anyone can download the .hpi file of the bugfixed plugin from there and then upload that (Manage Jenkins -> Manage Plugins ->Advanced -> Upload plugin) to their own Jenkins server(s) to try it out. It would be very helpful if everyone watching this could download and test that these changes fix the issue (and don't introduce any other big problems). You're all welcome to take a look at the code changes and comment on there too, if you're so inclined. Once I'm confident that everything is OK then I'll merge those changes in and release the new plugin officially.

Add Comment

[pjdarton@gmail.com (JIRA)]

pjdarton resolved as Fixed

Fixed in version 1.0.6, which was released today.

Jenkins / JENKINS-43269

workspace cleanup removes workspaces for running job

Change By: pjdarton Status: In Progress Resolved Resolution: Fixed

Add Comment