[JIRA] [ws-cleanup-plugin] (JENKINS-12783) Delete workspace before build starts ignores Exlude patterns (for directories)

48 views
Skip to first unread message

james.pace@rightmove.co.uk (JIRA)

unread,
May 8, 2015, 10:55:01 AM5/8/15
to jenkinsc...@googlegroups.com
James Pace commented on Bug JENKINS-12783
 
Re: Delete workspace before build starts ignores Exlude patterns (for directories)

I am also having issues with the exclude pattern. I do not want to delete a directory called "node_modules", or any of its sub directories or files, in the root of the workspace.

I have added an exclude pattern "*/node_modules//" but it still gets deleted. It doesn't seem to matter if the delete directories checkbox is selected or not.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

david.w.ackerman@gmail.com (JIRA)

unread,
Jun 9, 2015, 7:46:02 PM6/9/15
to jenkinsc...@googlegroups.com

I had the same issue, and I think I figured out what was going wrong (after cloning the repo and adding enough logging statements).

Excluding (not-deleting) everything under a toplevel directory works fine, i.e "toplevel/*" excludes everything under the folder "$WORKSPACE/toplevel". However, excluding something in a subdirectory does not work - i.e. "toplevel/asubdir/*" ends up deleting everything. This happens when you have "apply pattern also on directories" turned on. The reason is that DirectoryScanner will find directories to delete that might be parents of files that you've excluded, but the plugin deletes those folders recursively (i.e. rm -rf). The way to fix this is to add "toplevel" explicitly as an exclude to prevent it from deleting the parent directory as well. The reason I have the directories checkbox turned on is because I don't want the ".git" directory (or any extra directories) lying around and confusing jenkins when it builds the next time.

My project looks something like this (in a multi-module maven project, where node_modules/ resides in several subdirectories).

  • folder-1
  • node_modules
  • folder-2
  • node_modules
  • folder-3
  • folder-4
  • node_modules

My Configuration looks like this:
EXCLUDE */node_modules/*
EXCLUDE folder-1
EXCLUDE folder-2
EXCLUDE folder-4

And that seems to work as expected. It seems like the plugin needs to be more careful about the directories it deletes recursively - it needs to avoid deleting a parent that would delete excluded files (or maybe just do "rm" instead of "rm -r" so it won't delete the directory if it has otherwise excluded files in it). I would offer to create a patch, but my Intellij doesn't seem to be resolving dependencies in the project properly (although maven commands work fine), and I am not sure what others would consider the ideal behavior. At the very least, I think more documentation in the plugin help and/or plugin page would save people a lot of time.

david.w.ackerman@gmail.com (JIRA)

unread,
Jun 9, 2015, 7:50:01 PM6/9/15
to jenkinsc...@googlegroups.com
David Ackerman edited a comment on Bug JENKINS-12783
I had the same issue, and I think I figured out what was going wrong (after cloning the repo and adding enough logging statements).

Excluding (not-deleting) everything under a toplevel directory works fine, i.e "toplevel/ \ * \ *" excludes everything under the folder "$WORKSPACE/toplevel".  However, excluding something in a subdirectory does _not_ work - i.e. "toplevel/asubdir/ \ * \ *" ends up deleting everything.  This happens when you have "apply pattern also on directories" turned on.  The reason is that DirectoryScanner will find _directories_ to delete that might be parents of files that you've excluded, but the plugin deletes those folders recursively (i.e. rm -rf).  The way to fix this is to add "toplevel" explicitly as an exclude to prevent it from deleting the parent directory as well.  The reason I have the directories checkbox turned on is because I don't want the ".git" directory (or any extra directories) lying around and confusing jenkins when it builds the next time.


My project looks something like this (in a multi-module maven project, where node_modules/ resides in several subdirectories).
{noformat}
* folder-1
  - node_modules
* folder-2
  - node_modules
* folder-3
* folder-4
  - node_modules
{noformat}

My Configuration looks like this:
{noformat}
EXCLUDE   \   * \ */node_modules/ \ * \ *

EXCLUDE folder-1
EXCLUDE folder-2
EXCLUDE folder-4
{noformat}

And that seems to work as expected.  It seems like the plugin needs to be more careful about the directories it deletes recursively - it needs to avoid deleting a parent that would delete excluded files (or maybe just do "rm" instead of "rm -r" so it won't delete the directory if it has otherwise excluded files in it).  I would offer to create a patch, but my Intellij doesn't seem to be resolving dependencies in the project properly (although maven commands work fine), and I am not sure what others would consider the ideal behavior.  At the very least, I think more documentation in the plugin help and/or plugin page would save people a lot of time.

david.w.ackerman@gmail.com (JIRA)

unread,
Jun 9, 2015, 7:50:02 PM6/9/15
to jenkinsc...@googlegroups.com

ophyte@gmail.com (JIRA)

unread,
Mar 11, 2016, 12:49:04 PM3/11/16
to jenkinsc...@googlegroups.com
O. phyTe stopped work on Bug JENKINS-12783
 
Change By: O. phyTe
Status: In Progress Open

ophyte@gmail.com (JIRA)

unread,
Mar 11, 2016, 12:49:04 PM3/11/16
to jenkinsc...@googlegroups.com
O. phyTe started work on Bug JENKINS-12783
 
Change By: O. phyTe
Status: Open In Progress

lsanders@gmail.com (JIRA)

unread,
Apr 14, 2016, 2:10:02 PM4/14/16
to jenkinsc...@googlegroups.com

I spent quite a bit of time on this very issue (ironically, also trying to avoid the node penalty).

Of course leaving empty directories laying around is problematic, specifically for .repository when using a private maven repository for your build.

So, my configuration ended up looking like:

monospaced text
EXCLUDE */node_modules/*
INCLUDE .repository/**

lsanders@gmail.com (JIRA)

unread,
Apr 14, 2016, 2:10:02 PM4/14/16
to jenkinsc...@googlegroups.com
Larry Sanders edited a comment on Bug JENKINS-12783
I spent quite a bit of time on this very issue (ironically, also trying to avoid the node penalty).

Of course leaving empty directories laying around is problematic, specifically for .repository when using a private maven repository for your build.

So, my configuration ended up looking like:

{{monospaced text}}
EXCLUDE  **/node_modules/**
INCLUDE   .repository/**
{{monospaced text}}

lsanders@gmail.com (JIRA)

unread,
Apr 14, 2016, 2:11:01 PM4/14/16
to jenkinsc...@googlegroups.com

lsanders@gmail.com (JIRA)

unread,
Apr 14, 2016, 2:54:01 PM4/14/16
to jenkinsc...@googlegroups.com
Larry Sanders edited a comment on Bug JENKINS-12783
I spent quite a bit of time on this very issue (ironically, also trying to avoid the node penalty).

Of course leaving empty directories laying around is problematic, specifically for .repository when using a private maven repository for your build.

So, my configuration ended up looking like:


    EXCLUDE  **/node_modules/**
    INCLUDE   .repository/**

spikeynick@gmail.com (JIRA)

unread,
Sep 15, 2016, 1:54:03 PM9/15/16
to jenkinsc...@googlegroups.com

I don't know if this applies to the OP, but I had a very similar issue. I specified patterns to exclude, but it would completely delete everything in the workspace. If I instead specified a pattern to include that would work as expected. In my case the exclude actually was working correctly. I verified this by adding a custom delete command which was just a simple python script that appended the name of the file to be deleted to a text file (so it didn't actually do any deleting). It turned out that I wasn't excluding the .git directory, so that was getting deleted. You can't clone into a non empty directory, so Jenkins had to remove the workspace completely in order to do a new clone. I haven't tried with .svn, but I would expect similar results. One other thing to note is that if that you don't specify an include pattern then the plugin automatically uses the pattern */ So if you have included anything that doesn't match the .git directory that will also spare it from deletion.

tl;dr Make sure you also exclude your .git directory

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

spikeynick@gmail.com (JIRA)

unread,
Sep 15, 2016, 1:54:03 PM9/15/16
to jenkinsc...@googlegroups.com
Nick Romanyshyn edited a comment on Bug JENKINS-12783
I don't know if this applies to the OP, but I had a very similar issue.  I specified patterns to exclude, but it would completely delete everything in the workspace.  If I instead specified a pattern to include that would work as expected.  In my case the exclude actually was working correctly.  I verified this by adding a custom delete command which was just a simple python script that appended the name of the file to be deleted to a text file (so it didn't actually do any deleting).  It turned out that I wasn't excluding the .git directory, so that was getting deleted.  You can't clone into a non empty directory, so Jenkins had to remove the workspace completely in order to do a new clone.  I haven't tried with .svn, but I would expect similar results.  One other thing to note is that if that you don't specify an include pattern then the plugin automatically uses the pattern \ **/*  So if you have included anything that doesn't match the .git directory that will also spare it from deletion.

*tl;dr Make sure you also exclude your .git directory*

spikeynick@gmail.com (JIRA)

unread,
Sep 15, 2016, 1:55:02 PM9/15/16
to jenkinsc...@googlegroups.com
Nick Romanyshyn edited a comment on Bug JENKINS-12783
I don't know if this applies to the OP, but I had a very similar issue.  I specified patterns to exclude, but it would completely delete everything in the workspace.  If I instead specified a pattern to include that would work as expected.  In my case the exclude actually was working correctly.  I verified this by adding a custom delete command which was just a simple python script that appended the name of the file to be deleted to a text file (so it didn't actually do any deleting).  It turned out that I wasn't excluding the .git directory, so that was getting deleted.  You can't clone into a non empty directory, so Jenkins had to remove the workspace completely in order to do a new clone.  I haven't tried with .svn, but I would expect similar results.  One other thing to note is that if that you don't specify an include pattern then the plugin automatically uses the pattern \ ** * /* *   So if you have included anything that doesn't match the .git directory that will also spare it from deletion.


*tl;dr Make sure you also exclude your .git directory*

pisalaun@bouyguestelecom.fr (JIRA)

unread,
Mar 1, 2017, 8:06:03 AM3/1/17
to jenkinsc...@googlegroups.com

Hi,
I'm having the same issue, Include work fine but not Exclude.
I have a maven project with custom workspace and a private maven repository local to workspace.
I want to keep the ".repository" directory but it is delete by the "Delete workspace before build starts" function.
I have add a Exclude ".respository/**" but it isn't work.
Can you help me?

grimnaut@gmail.com (JIRA)

unread,
Feb 21, 2018, 1:08:02 PM2/21/18
to jenkinsc...@googlegroups.com

Jenkins Versions: 2.60.3 & 2.73.3 (WAR deployments on WildFly)
Plugin Version: 0.34
Platform: Linux
Plugins:
Git client plugin 2.6.0
Git plugin 3.6.4

Details:
No issues when the SCM is CVS. However we are migrating from CVS to GIT.
When I configure a freestyle job to use GIT as the SCM and with "Delete workspace before build starts" checked it ignores any "Advanced" options I use (I'm trying to exclude ' .build. ' and a 'reports/**' directory) and just deletes everything in the entire workspace.

In troubleshooting I also testing the Post Build Action > "Delete workspace when build is done". This half works. I can tell it to "Exclude" specific files, but as soon as the "Apply pattern also on directories" is checked it goes back to the above behavior of deleting everything regardless of excludes.

Really hoping someone can have a look at this and find the root cause and a fix.

Thanks in advance!

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

grimnaut@gmail.com (JIRA)

unread,
Feb 21, 2018, 1:09:03 PM2/21/18
to jenkinsc...@googlegroups.com
Grim Reaper updated an issue
 
Jenkins / Bug JENKINS-12783
Change By: Grim Reaper
Environment: OP - Linux, Jenkins ver. 1.451, plugin 0.7, job running on master or slave
Grim -

grimnaut@gmail.com (JIRA)

unread,
Feb 21, 2018, 1:10:02 PM2/21/18
to jenkinsc...@googlegroups.com

grimnaut@gmail.com (JIRA)

unread,
Feb 21, 2018, 1:11:04 PM2/21/18
to jenkinsc...@googlegroups.com
Grim Reaper updated an issue
Change By: Grim Reaper
Environment:
OP - Linux, Jenkins ver. 1.451, plugin 0.7, job running on master or slave
Grim:
   Jenkins Versions: 2.60.3 & 2.73.3 (WAR deployments on WildFly)
   Plugin Version: 0.34
   Platform: Linux
   Plugins:
      Git client plugin 2.6.0
      Git plugin 3.6.4

grimnaut@gmail.com (JIRA)

unread,
Feb 21, 2018, 1:11:04 PM2/21/18
to jenkinsc...@googlegroups.com

grimnaut@gmail.com (JIRA)

unread,
Feb 21, 2018, 1:24:04 PM2/21/18
to jenkinsc...@googlegroups.com
Jenkins Versions: 2.60.3 & 2.73.3 (WAR deployments on WildFly)
Plugin Version: 0.34
Platform: Linux
Plugins:
Git client plugin 2.6.0
Git plugin 3.6.4

Details:
No issues when the SCM is CVS. However we are migrating from CVS to GIT.
When I configure a freestyle job to use GIT as the SCM and with "Delete workspace before build starts" checked it ignores any "Advanced" options I use (I'm trying to exclude ' * * .build.* * ' and a 'reports/**' directory) and just deletes everything in the entire workspace.


In troubleshooting I also testing the Post Build Action > "Delete workspace when build is done". This half works. I can tell it to "Exclude" specific files, but as soon as the "Apply pattern also on directories" is checked it goes back to the above behavior of deleting everything regardless of excludes.

Really hoping someone can have a look at this and find the root cause and a fix.

Thanks in advance!

grimnaut@gmail.com (JIRA)

unread,
Feb 22, 2018, 7:26:03 AM2/22/18
to jenkinsc...@googlegroups.com
Grim Reaper edited a comment on Bug JENKINS-12783
Jenkins Versions: 2.60.3 & 2.73.3 (WAR deployments on WildFly)
Plugin Version: 0.34
Platform: Linux
Plugins:
Git client plugin 2.6.0
Git plugin 3.6.4

Details:
No issues when the SCM is CVS. However we are migrating from CVS to GIT.
When I configure a freestyle job to use GIT as the SCM and with "Delete workspace before build starts" checked it ignores any "Advanced" options I use (I'm trying to exclude ' **.build.** ' and a 'reports/**' directory) and just deletes everything in the entire workspace.


In troubleshooting I also testing the Post Build Action > "Delete workspace when build is done". This half works. I can tell it to "Exclude" specific files, but as soon as the "Apply pattern also on directories" is checked it goes back to the above behavior of deleting everything regardless of excludes.

Really hoping someone can have a look at this and find the root cause and a fix.

Thanks in advance!


Edit:
I forgot to add that the GIT install is on a windows machine and the repo is located through a UNC share. Not that I think this has any bearing as the repo pull/clone works fine. Just trying to provide as much info as possible.

grimnaut@gmail.com (JIRA)

unread,
Feb 22, 2018, 8:06:02 AM2/22/18
to jenkinsc...@googlegroups.com

grimnaut@gmail.com (JIRA)

unread,
Feb 22, 2018, 8:08:02 AM2/22/18
to jenkinsc...@googlegroups.com
Grim Reaper assigned an issue to Oliver Gondža
 

Assigning to you as the plugins page indicates you are the ws-cleanup maintainer.
Also for visibility.

Change By: Grim Reaper
Assignee: Lucie Votypkova Oliver Gondža

grimnaut@gmail.com (JIRA)

unread,
Mar 7, 2018, 3:27:03 PM3/7/18
to jenkinsc...@googlegroups.com
Grim Reaper commented on Bug JENKINS-12783
 
Re: Delete workspace before build starts ignores Exlude patterns (for directories)

For those looking for a work around, I've use the following plugin:
https://plugins.jenkins.io/preSCMbuildstep

With the following Shell script to do the same thing as WS-cleanup.

echo "Deleting..."
ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' 
ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' | xargs rm -rf

 

The above deletes everything except the items in the "egrep" command (excludes).

timothy.bradt@dematic.com (JIRA)

unread,
Aug 6, 2019, 10:13:02 AM8/6/19
to jenkinsc...@googlegroups.com

This bug is unfortunate.  We're trying to preserve our IntelliJ project files so we don't have to restore project settings every time we build.  This should really be addressed.

+1 vote

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

grimnaut@gmail.com (JIRA)

unread,
Dec 4, 2019, 2:18:04 PM12/4/19
to jenkinsc...@googlegroups.com

Updated my test server to Jenkins 2.89.4 and we are now using "Gitea" as our backend SCM.

Still the same issue.

Also update on the work around. Found out it wasn't working as intended. So now I shuffle files around to make it work. This is all on Linux.

Steps to configure the job:

  1. Under "Source Code management" (Git), select "Wipe out repo & force clone" under "Additional behaviours".
  2. Under "Build Environment", note you still need to have the preSCMBuildStep plugin and check "Run buildstep before SCM runs" and add "Execute shell", then use the following to "Exclude files" (You may need to modify for your needs):
    # Move build files to temp location before workspace wipe
    ls -Al "${WORKSPACE}"
    [ ! -d "/tmp/${BUILD_TAG}/" ] && mkdir -p "/tmp/${BUILD_TAG}/"
    mv -v "${WORKSPACE}"/*.build.* "${WORKSPACE}"/suppressions.xml "/tmp/${BUILD_TAG}/"
    ls -Al "/tmp/${BUILD_TAG}/"
    
  1. Under the "Build" section of your job. Add "Execute shell" and move it to the top. In the textbox add the following script commands to move your files back into the workspace:
    # Move build files back to workspace before build runs 
    mv -v "/tmp/${BUILD_TAG}"/* "${WORKSPACE}" 
    ls -Al "${WORKSPACE}" 
    rm -rfv "/tmp/${BUILD_TAG}/"
    

Off to try the latest and greatest Jenkins (Hopefully) it can still build our legacy apps >.<).

 

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

grimnaut@gmail.com (JIRA)

unread,
Dec 4, 2019, 2:19:02 PM12/4/19
to jenkinsc...@googlegroups.com
Grim Reaper edited a comment on Bug JENKINS-12783
Updated my test server to Jenkins 2.89.4 and we are now using "Gitea" as our backend SCM.

Still the same issue.

Also update on the work around. Found out it wasn't working as intended. So now I shuffle files around to make it work. This is all on Linux.

Steps to configure the job:
# Under "Source Code management" (Git), select "Wipe out repo & force clone" under "Additional behaviours".
# Under "Build Environment", note you still need to have the preSCMBuildStep plugin and check "Run buildstep before SCM runs" and add "Execute shell", then use the following to "Exclude files" (You may need to modify for your needs):
{code:java}

# Move build files to temp location before workspace wipe
ls -Al "${WORKSPACE}"
[ ! -d "/tmp/${BUILD_TAG}/" ] && mkdir -p "/tmp/${BUILD_TAG}/"
mv -v "${WORKSPACE}"/*.build.* "${WORKSPACE}"/suppressions.xml "/tmp/${BUILD_TAG}/"
ls -Al "/tmp/${BUILD_TAG}/"
{code}

# Under the "Build" section of your job. Add "Execute shell" and move it to the top. In the textbox add the following script commands to move your files back into the workspace:
{code:java}

# Move build files back to workspace before build runs
mv -v "/tmp/${BUILD_TAG}"/* "${WORKSPACE}"
ls -Al "${WORKSPACE}"
rm -rfv "/tmp/${BUILD_TAG}/"
{code}

Off to try the latest and greatest Jenkins (Hopefully
) it can still build our legacy apps >.<).

 
Reply all
Reply to author
Forward
0 new messages