Still fighting with "tidy"

[Dan White]

Either puppet-resource-type "tidy" is busted or I do not properly understand its use.

Red Had Enterprise Linux 5.8 server (x86_64)
I am seeing this behavior on both Puppet 2.6.17 and 2.7.19

I want to "tidy" up a path -- let's say /volume/foo/bar/ (bar is a directory)

The resource definition is:

tidy { '/volume/foo/bar/':
age => '0',
recurse => true,
rmdirs => true,
}

To test it, I do the following

cd /volume/foo/bar/
mkdir 1 2 3 4 5 6
touch 1/a 2/b 3/c 4/d 5/e 6/f 6/g

then I run puppet.
It deletes all the "touched" files (a, b, c, d, e, f, g)

Now for The Problem:
When I run puppet again, I am expecting the empty directories ( 1/ 2/ 3/ 4/ 5/ 6/ ) to get deleted.
They are not removed. Even though the "rmdirs" parameter is set to true.

Is this the way "tidy" is supposed to work ?

“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”
Bill Waterson (Calvin & Hobbes)

[jcbollinger]

On Thursday, October 11, 2012 5:07:36 PM UTC-5, Ygor wrote:

Either puppet-resource-type "tidy" is busted or I do not properly understand its use.

Red Had Enterprise Linux 5.8 server (x86_64)
I am seeing this behavior on both Puppet 2.6.17 and 2.7.19

I want to "tidy" up a path -- let's say /volume/foo/bar/  (bar is a directory)

The resource definition is:

    tidy { '/volume/foo/bar/':
        age     => '0',
        recurse => true,
        rmdirs  => true,
    }

To test it, I do the following

cd /volume/foo/bar/
mkdir 1 2 3 4 5 6
touch 1/a 2/b 3/c 4/d 5/e 6/f 6/g

then I run puppet.
It deletes all the "touched" files (a, b, c, d, e, f, g)

Now for The Problem:
When I run puppet again, I am expecting the empty directories ( 1/ 2/ 3/ 4/ 5/ 6/ ) to get deleted.
They are not removed.  Even though the "rmdirs" parameter is set to true.

Is this the way "tidy" is supposed to work ?

That behavior is not consistent with my reading of the docs.  There is definitely a bug there.  I speculate that it might be related to the fact that recursing the subdirectories necessarily updates their atimes, which is what 'tidy' looks at by default to compute an age.  It still should work even then, however, because it should not be possible that way to produce an age less than zero.  You should file a ticket.

You could try to work around it by setting a negative 'age' parameter and / or by changing the 'type' to 'mtime' (last modification time) or even 'ctime' (creation time).  Note that using 'mtime' might have similar implications for nonempty directories as 'atime' may have for empty ones (removing contents should update directory mtimes, possibly producing an issue with 'age') but at least you would then expect the then-empty directories to be cleaned up on the next Puppet run.


John

[Dan White]

Thanks for responding, John.

I saw other discussions about "tidy" that mentioned the same thing, so I started tinkering.

If I use mtime or ctime, directories get deleted, which is great, but it also deleted the top of the tree -- /volume/foo/bar/ in my example.
I found a workaround, but I think it is ugly.

Looking at the output, tidy must generate temporary resources to delete the files and directories.

I see:
notice: /Stage[main]//Node[server.foo.org]/Tidy[/volume/foo/bar]: Tidying File[/volume/foo/bar/]
notice: /Stage[main]//Node[server.foo.org]/Tidy[/volume/foo/bar]: Tidying File[/volume/foo/bar/1]
notice: /Stage[main]//Node[server.foo.org]/Tidy[/volume/foo/bar]: Tidying File[/volume/foo/bar/1/a]
...
and then
notice: /File[/volume/foo/bar/1/a]/ensure: removed
notice: /File[/volume/foo/bar/1]/ensure: removed
...


If I declare a file resource for /volume/foo/bar/ with ensure => present, the puppet run output says:
info: /File[/volume/foo/bar]: Duplicate generated resource; skipping

Seems like a klooodjy way to do it but...?

[jcbollinger]

On Friday, October 12, 2012 10:27:11 AM UTC-5, Ygor wrote:

[...]

I saw other discussions about "tidy" that mentioned the same thing, so I started tinkering.

If I use mtime or ctime, directories get deleted, which is great, but it also deleted the top of the tree -- /volume/foo/bar/ in my example.
I found a workaround, but I think it is ugly.

That's not the behavior I would have expected from the docs, which, for one thing, use the example of tidying /tmp.  You would never want to delete that.

Is the behavior the same if you leave off the trailing '/' from the resource title?  What if you use "matches => ['[^.]*', '.[^.]*', '..?*']" (i.e. everything except . and ..)?  You shouldn't have to jump through such hoops, but I think it's clear that Puppet has some deficiencies here.  It the very minimum, the documentation should be improved.

 

Looking at the output, tidy must generate temporary resources to delete the files and directories.

Yes.  That's kinda strange, but documented.

 

If I declare a file resource for /volume/foo/bar/ with ensure => present, the puppet run output says:
info: /File[/volume/foo/bar]: Duplicate generated resource; skipping

As it should do.  Really, it is unwise anyway to use 'tidy' in a way that might affect an otherwise-managed file, but if you happen to do so then it is right for Puppet to complain instead of trying to guess which declarations represent your real intention.

 

Seems like a klooodjy way to do it but...?

I suspect the reason is exactly to catch conflicts, just as you observed it to do.  But, yeah, it does seem kludgy, or at least clunky.


John

[Dan White]

---

Good suggestion about the matches parameter.
I will try it and let you know.
I am fortunate in that the directory I am trying to tidy is NOT something important like "/tmp", but I share your concern.

I opened a bug/issue on this before -- http://projects.puppetlabs.com/issues/15904
I think I need to add this detail to that bug.

[Dan White]

>> Is the behavior the same if you leave off the trailing '/' from the resource title? What if you use "matches => ['[^.]*', '.[^.]*', '..?*']" (i.e. everything except . and ..)? You shouldn't have to jump through such hoops, but I think it's clear that Puppet has some deficiencies here. It the very minimum, the documentation should be improved.
>>

>> John

>>
> Good suggestion about the matches parameter.
> I will try it and let you know.
> I am fortunate in that the directory I am trying to tidy is NOT something important like "/tmp", but I share your concern.
>
> I opened a bug/issue on this before -- http://projects.puppetlabs.com/issues/15904
> I think I need to add this detail to that bug.

I tried the "matches" parameter and adding the trailing slash.
Neither preserved the top of my tree.

Using type atime does not remove directories.
Using types mtime or ctime will remove directories, but also deletes the directory named in the "path" parameter unless I use the hack of an explicit resource to protect it from being tidy-ed away.

I believe that "tidy" needs some work.

I thought of a less kludgy way to get what I want:

This is for user virtual user home folders for an FTP server. The policy is that files are removed after 5 days, so...

# /etc/puppet/modules/ftp-server/manifests/init.pp
class ftp-server ( $ftp-homedir="/tmp/default-ftp-home" ){

service { 'vsftp':
...
}

file { $ftp-homedir:
ensure => directory,
owner => root,
group => root,
mode => 775,
}

cron { 'ftp-remove-empty-dirs':
command => "/usr/bin/find ${ftp-homedir} -type d -depth -mindepth 2 -empty -exec rmdir {} \; "
user => root,
hour => 2,
minute => 10,
}
}

# /etc/puppet/modules/ftp-server/manifests/users.pp
define ftp-server::users ( $username ) {

file { "${ftp-homedir}/${username}":
ensure => directory,
owner => ftp,
group => ftp,
mode => 700;
}

tidy { "tidy-${username}":
path => "${ftp-homedir}/${username}",
age => '5d',
recurse => true,
}
}

This does what I wanted to do. The tidy resources on each virtual ftp home directory will expire the files as appropriate and the cron job will clean out the empty directories.

If anyone needs the details of the find command explained, just ask.