howto make file resource purge option aware of other puppet resources managed files

[piavlo]

Hi,

puppet file resources has an "purge" option - which will purge files
in the directory which are not managed by this specific file resource.
It will purge files managed by other puppet resources in the same
directory. What i need is a "purge" option to be aware of all all
puppet resources and not purge them.

Is there a trick/workaround to do that?

I know that I can make sure it gets applied prior to all other
resources - and purge all files in a directory and they will be
recreated again in the same puppet run - but it's not an acceptable
solution.

Thanks
Alex

[Daniel Pittman]

On Mon, Aug 29, 2011 at 07:43, piavlo <lolit...@gmail.com> wrote:

> puppet file resources has an "purge" option - which will purge files
> in the directory which are not managed by this specific file resource.
> It will purge files managed by other puppet resources in the same
> directory. What i need is a "purge" option to be aware of all all
> puppet resources and not purge them.

No, it shouldn't, at least in most cases. Can you post the specific
(and, ideally, minimal) set of file resource declarations that are
causing you trouble here?

I don't know what is going wrong, but it certainly wasn't the case
that the delete/recreate behaviour happened. We have code
specifically to find overlapping managed files in scenarios like this,
and to apply the more specific rules, such as "files with content
should exist".

Daniel
--
⎋ Puppet Labs Developer – http://puppetlabs.com
♲ Made with 100 percent post-consumer electrons

[piavlo]

Hi Daniel,

it looks indeed that it works in some/most cases - I tried to create a
minimal test class to check it using only file resources and it works
as you say - and as I want. So I have to post here one of the actual
not so minimal use case which depend on the expected behaviour.

---------------------------------------------
#Class to setup specific yum repos
class ssa::yum {

class { '::yum': }

class { 'yum::epel': enable => true }
class { 'yum::rpmforge': enable => true }
class { 'yum::nixval': enable => true }
class { 'yum::cloudera': enable => true }

class { 'yum::zabbix': enable => false }
class { 'yum::elff': enable => false }

}
---------------------------------------------
#Class which has yum::repo define
class yum {

file { "/etc/yum.repos.d":
ensure => directory,
purge => true,
recurse => true,
owner => 'root',
group => 'root',
mode => 0755,
source => "puppet:///yum/yum.repos.d"
}

define repo ( $descr='absent', $enabled='0', $baseurl='absent',
$mirrorlist='absent', $gpgcheck='absent', $gpgkey='absent',
$exclude='absent', $includepkgs='absent' ) {

if $gpgkey != 'absent' and ! defined(File["${gpgkey}"]) {

$key = "file://${gpgkey}"

file { "${gpgkey}":
owner => root,
group => root,
mode => 0444,
source => "puppet:///modules/yum${gpgkey}",
before => Yumrepo["${name}"]
}

exec { "rpm --import ${gpgkey}":
path => "/bin:/sbin:/usr/bin:/usr/sbin",
subscribe => File["${gpgkey}"],
refreshonly => true
}

}
else {
$key = 'absent'
}

yumrepo { "${name}":
descr => $descr,
enabled => $enabled,
baseurl => $baseurl,
mirrorlist => $mirrorlist,
failovermethod => 'priority',
gpgcheck => $gpgcheck,
gpgkey => $key,
exclude => $exclude,
includepkgs => $includepkgs,
require => File["/etc/yum.repos.d"]
}

}

}
----------------------------------------------
#Sample class that uses yum::repo define
class yum::epel( $enable = true ) {

if $enable { $enabled = '1' }
else { $enabled = '0' }

yum::repo { "epel":
descr => 'Extra Packages for Enterprise Linux 5 - $basearch',
enabled => $enabled,
mirrorlist => 'http://mirrors.fedoraproject.org/mirrorlist?
repo=epel-5&arch=$basearch',
gpgcheck => '1',
gpgkey => '/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL'
}

}
----------------------------------------------

In the log below you can clearly see that puppet first purges all
managed(and unmanaged) files from the /etc/yum.repos.d
and the populates it again with same managed files. The exact same
purge/populate happens on each run.
----------------------------------------------
[root@mon1a yum.repos.d]# puppet agent --verbose --no-daemonize
notice: Starting Puppet client version 2.7.1
info: Retrieving plugin
info: Loading facts in ssa
info: Loading facts in ssa
info: Caching catalog for mon1a.internal
info: Applying configuration version '1314659756'
info: FileBucket got a duplicate file {md5}
f9994d7bb9f7daf111bcb4c2b1363017
info: /File[/etc/yum.repos.d/nixval.repo]: Filebucketed /etc/
yum.repos.d/nixval.repo to puppet with sum
f9994d7bb9f7daf111bcb4c2b1363017
notice: /File[/etc/yum.repos.d/nixval.repo]/ensure: removed
info: FileBucket got a duplicate file {md5}
7dc04195bbd206d01b59068f8742a940
info: /File[/etc/yum.repos.d/rpmforge-extras.repo]: Filebucketed /etc/
yum.repos.d/rpmforge-extras.repo to puppet with sum
7dc04195bbd206d01b59068f8742a940
notice: /File[/etc/yum.repos.d/rpmforge-extras.repo]/ensure: removed
info: FileBucket got a duplicate file {md5}
75377ca8231483ff6e7ca8f5305f1bcc
info: /File[/etc/yum.repos.d/cloudera-cdh3.repo]: Filebucketed /etc/
yum.repos.d/cloudera-cdh3.repo to puppet with sum
75377ca8231483ff6e7ca8f5305f1bcc
notice: /File[/etc/yum.repos.d/cloudera-cdh3.repo]/ensure: removed
info: FileBucket got a duplicate file {md5}
e0c282cec3ec08ec8af395429a2f1cc0
info: /File[/etc/yum.repos.d/elff.repo]: Filebucketed /etc/yum.repos.d/
elff.repo to puppet with sum e0c282cec3ec08ec8af395429a2f1cc0
notice: /File[/etc/yum.repos.d/elff.repo]/ensure: removed
info: FileBucket got a duplicate file {md5}
e797e55a1390785b478c017130821404
info: /File[/etc/yum.repos.d/danny.repo]: Filebucketed /etc/
yum.repos.d/danny.repo to puppet with sum
e797e55a1390785b478c017130821404
notice: /File[/etc/yum.repos.d/danny.repo]/ensure: removed
info: FileBucket adding {md5}c0fcc5653e77aa5c74a47c41e2ed34bb
info: /File[/etc/yum.repos.d/rpmforge.repo]: Filebucketed /etc/
yum.repos.d/rpmforge.repo to puppet with sum
c0fcc5653e77aa5c74a47c41e2ed34bb
notice: /File[/etc/yum.repos.d/rpmforge.repo]/ensure: removed
info: FileBucket got a duplicate file {md5}
e318df8a6487d6cee2c9e2ed7b046225
info: /File[/etc/yum.repos.d/epel.repo]: Filebucketed /etc/yum.repos.d/
epel.repo to puppet with sum e318df8a6487d6cee2c9e2ed7b046225
notice: /File[/etc/yum.repos.d/epel.repo]/ensure: removed
info: create new repo rpmforge in file /etc/yum.repos.d/rpmforge.repo
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge]/
Yumrepo[rpmforge]/descr: descr changed '' to 'RHEL $releasever -
RPMforge.net - dag'
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge]/
Yumrepo[rpmforge]/mirrorlist: mirrorlist changed '' to 'http://
apt.sw.be/redhat/el5/en/mirrors-rpmforge'
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge]/
Yumrepo[rpmforge]/baseurl: baseurl changed '' to 'http://apt.sw.be/
redhat/el5/en/$basearch/rpmforge'
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge]/
Yumrepo[rpmforge]/enabled: enabled changed '' to '1'
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge]/
Yumrepo[rpmforge]/gpgcheck: gpgcheck changed '' to '1'
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge]/
Yumrepo[rpmforge]/gpgkey: gpgkey changed '' to 'file:///etc/pki/rpm-
gpg/RPM-GPG-KEY-rpmforge-dag'
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge]/
Yumrepo[rpmforge]/exclude: exclude changed '' to 'nagios-plugins*
rrdtool* perl-rrdtool'
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge]/
Yumrepo[rpmforge]/failovermethod: failovermethod changed '' to
'priority'
info: changing mode of /etc/yum.repos.d/rpmforge.repo from 600 to 644
info: create new repo rpmforge-extras in file /etc/yum.repos.d/
rpmforge-extras.repo
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge-extras]/
Yumrepo[rpmforge-extras]/descr: descr changed '' to 'RHEL $releasever
- RPMforge.net - extras'
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge-extras]/
Yumrepo[rpmforge-extras]/mirrorlist: mirrorlist changed '' to 'http://
apt.sw.be/redhat/el5/en/mirrors-rpmforge-extras'
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge-extras]/
Yumrepo[rpmforge-extras]/baseurl: baseurl changed '' to 'http://
apt.sw.be/redhat/el5/en/$basearch/extras'
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge-extras]/
Yumrepo[rpmforge-extras]/enabled: enabled changed '' to '1'
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge-extras]/
Yumrepo[rpmforge-extras]/gpgcheck: gpgcheck changed '' to '1'
notice: /Stage[main]/Yum::Rpmforge/Yum::Repo[rpmforge-extras]/
Yumrepo[rpmforge-extras]/failovermethod: failovermethod changed '' to
'priority'
info: changing mode of /etc/yum.repos.d/rpmforge-extras.repo from 600
to 644
info: create new repo cloudera-cdh3 in file /etc/yum.repos.d/cloudera-
cdh3.repo
notice: /Stage[main]/Yum::Cloudera/Yum::Repo[cloudera-cdh3]/
Yumrepo[cloudera-cdh3]/descr: descr changed '' to 'Cloudera's
Distribution for Hadoop, Version 3'
notice: /Stage[main]/Yum::Cloudera/Yum::Repo[cloudera-cdh3]/
Yumrepo[cloudera-cdh3]/mirrorlist: mirrorlist changed '' to 'http://
archive.cloudera.com/redhat/cdh/3/mirrors'
notice: /Stage[main]/Yum::Cloudera/Yum::Repo[cloudera-cdh3]/
Yumrepo[cloudera-cdh3]/enabled: enabled changed '' to '1'
notice: /Stage[main]/Yum::Cloudera/Yum::Repo[cloudera-cdh3]/
Yumrepo[cloudera-cdh3]/gpgcheck: gpgcheck changed '' to '1'
notice: /Stage[main]/Yum::Cloudera/Yum::Repo[cloudera-cdh3]/
Yumrepo[cloudera-cdh3]/gpgkey: gpgkey changed '' to 'file:///etc/pki/
rpm-gpg/RPM-GPG-KEY-cloudera'
notice: /Stage[main]/Yum::Cloudera/Yum::Repo[cloudera-cdh3]/
Yumrepo[cloudera-cdh3]/failovermethod: failovermethod changed '' to
'priority'
info: changing mode of /etc/yum.repos.d/cloudera-cdh3.repo from 600 to
644
info: create new repo nixval in file /etc/yum.repos.d/nixval.repo
notice: /Stage[main]/Yum::Nixval/Yum::Repo[nixval]/Yumrepo[nixval]/
descr: descr changed '' to 'Red Hat Enterprise 5 - NIXVAL - Updates'
notice: /Stage[main]/Yum::Nixval/Yum::Repo[nixval]/Yumrepo[nixval]/
baseurl: baseurl changed '' to 'http://repo.nixval.com/nixval-centos/5/
updates'
notice: /Stage[main]/Yum::Nixval/Yum::Repo[nixval]/Yumrepo[nixval]/
enabled: enabled changed '' to '1'
notice: /Stage[main]/Yum::Nixval/Yum::Repo[nixval]/Yumrepo[nixval]/
gpgcheck: gpgcheck changed '' to '0'
notice: /Stage[main]/Yum::Nixval/Yum::Repo[nixval]/Yumrepo[nixval]/
failovermethod: failovermethod changed '' to 'priority'
info: changing mode of /etc/yum.repos.d/nixval.repo from 600 to 644
info: create new repo epel in file /etc/yum.repos.d/epel.repo
notice: /Stage[main]/Yum::Epel/Yum::Repo[epel]/Yumrepo[epel]/descr:
descr changed '' to 'Extra Packages for Enterprise Linux 5 -
$basearch'
notice: /Stage[main]/Yum::Epel/Yum::Repo[epel]/Yumrepo[epel]/
mirrorlist: mirrorlist changed '' to 'http://mirrors.fedoraproject.org/
mirrorlist?repo=epel-5&arch=$basearch'
notice: /Stage[main]/Yum::Epel/Yum::Repo[epel]/Yumrepo[epel]/enabled:
enabled changed '' to '1'
notice: /Stage[main]/Yum::Epel/Yum::Repo[epel]/Yumrepo[epel]/gpgcheck:
gpgcheck changed '' to '1'
notice: /Stage[main]/Yum::Epel/Yum::Repo[epel]/Yumrepo[epel]/gpgkey:
gpgkey changed '' to 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL'
notice: /Stage[main]/Yum::Epel/Yum::Repo[epel]/Yumrepo[epel]/
failovermethod: failovermethod changed '' to 'priority'
info: changing mode of /etc/yum.repos.d/epel.repo from 600 to 644
info: create new repo danny in file /etc/yum.repos.d/danny.repo
notice: /Stage[main]/Yum::Zabbix/Yum::Repo[danny]/Yumrepo[danny]/
descr: descr changed '' to 'Dan Horak's repo'
notice: /Stage[main]/Yum::Zabbix/Yum::Repo[danny]/Yumrepo[danny]/
baseurl: baseurl changed '' to 'http://fedora.danny.cz/danny-el/
$releasever/$basearch'
notice: /Stage[main]/Yum::Zabbix/Yum::Repo[danny]/Yumrepo[danny]/
enabled: enabled changed '' to '1'
notice: /Stage[main]/Yum::Zabbix/Yum::Repo[danny]/Yumrepo[danny]/
gpgcheck: gpgcheck changed '' to '1'
notice: /Stage[main]/Yum::Zabbix/Yum::Repo[danny]/Yumrepo[danny]/
gpgkey: gpgkey changed '' to 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-
danny'
notice: /Stage[main]/Yum::Zabbix/Yum::Repo[danny]/Yumrepo[danny]/
failovermethod: failovermethod changed '' to 'priority'
info: changing mode of /etc/yum.repos.d/danny.repo from 600 to 644
info: create new repo elff in file /etc/yum.repos.d/elff.repo
notice: /Stage[main]/Yum::Elff/Yum::Repo[elff]/Yumrepo[elff]/descr:
descr changed '' to 'Enterprise Linux Fast Forward 5 - $basearch'
notice: /Stage[main]/Yum::Elff/Yum::Repo[elff]/Yumrepo[elff]/baseurl:
baseurl changed '' to 'http://download.elff.bravenet.com/5/$basearch'
notice: /Stage[main]/Yum::Elff/Yum::Repo[elff]/Yumrepo[elff]/enabled:
enabled changed '' to '0'
notice: /Stage[main]/Yum::Elff/Yum::Repo[elff]/Yumrepo[elff]/gpgcheck:
gpgcheck changed '' to '1'
notice: /Stage[main]/Yum::Elff/Yum::Repo[elff]/Yumrepo[elff]/gpgkey:
gpgkey changed '' to 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ELFF'
notice: /Stage[main]/Yum::Elff/Yum::Repo[elff]/Yumrepo[elff]/
failovermethod: failovermethod changed '' to 'priority'
info: changing mode of /etc/yum.repos.d/elff.repo from 600 to 644
notice: Finished catalog run in 8.10 seconds
notice: Caught INT; calling stop
[root@mon1a yum.repos.d]#
-----------------------------------------------

Do you need any further info?
Why does it happen in this case but not in simple case there only file
resources are involved?

Thanks
Alex


On Aug 29, 8:33 pm, Daniel Pittman <dan...@puppetlabs.com> wrote:

[jcbollinger]

You put your finger on it when you wrote "It will purge files managed
by other puppet resources in the same directory," but Daniel missed
your meaning, as did I on my first reading.

You have tripped over a subtlety in the meaning of File's 'purge'
parameter: it removes files that are not managed *via File
resources*. It is not aware of files that are managed via other
resources. You might consider filing a ticket about this, but do
check first whether there's an existing one (i.e. I didn't).

On the other hand, one reason you've run into trouble is that you're
mixing levels of resource abstraction and effectively managing the
same physical resources through two independent paths. To ensure that
only the repos managed via Puppet are in fact present at all, you
should be using the Resources metaresource:

resources { 'yumrepo': purge => true }

That says more directly what you seem really to want to say, without
assuming any particular back-end storage for repository declarations.
Moreover, it should do the right thing if you happen to end up with
managed and unmanaged repositories declared in the same file. (Note:
do not set the 'purge' parameter for File['/etc/yum.repos.d'], or set
it to 'false'.)


John

[Nan Liu]

Not sure if this covers all the files in the yum.repos.d directory.
You can try adding an additional file resource in the define resource
to avoid the file directory purge performing any action on files
created by yumrepo resource type:

file { "/etc/yum.repos.d/${name}.repo":
ensure => file,
}

Alternatively, I think yumrepo can be modeled as a file using a file
using ERB template instead of yumrepo resource:

define yum::repo {
...

file { "/etc/yum.repos.d/${name}.repo":
ensure => file,

owner => 'root',
group => 'root',

mode => '0644',
source => template('yum/yumrepo.erb')
}
}

yumrepo.erb:
[<%=name%>]
name=<%=desc%>
baseurl=<%=baseurl%>
...

Thanks,

Nan

[piavlo]

Hi Nan

Adding

file { "/etc/yum.repos.d/${name}.repo":
ensure => file
}

took care of the problem.

I still wonder if it should have worked without explicit file
resource?
Or "purge" is not supposed to be aware of other non file puppet
resources?

Thanks
Alex

On Aug 30, 5:11 pm, Nan Liu <n...@puppetlabs.com> wrote:

[Nan Liu]

On Tue, Aug 30, 2011 at 11:07 AM, piavlo <lolit...@gmail.com> wrote:
> Hi Nan
>
> Adding
> file { "/etc/yum.repos.d/${name}.repo":
>  ensure => file
> }
> took care of the problem.
>
> I still wonder if it should have worked without explicit file
> resource?
> Or "purge" is not supposed to be aware of other non file puppet
> resources?

The purge attribute for file is only aware of file resources and not
other puppet resources that creates a file. I suppose a feature
request to patch yumrepo type to autocreate a file will resolve this
issue, not sure if that has any undesirable side effects and that's a
decision for Puppet's product manager.

Thanks,

Nan

[Daniel Pittman]

Oh, dear. Yes, John has it. :/

This is definitely a bug, and something we should fix. Can you file a
ticket with the details, so that we don't lose track of this?

Daniel

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[piavlo]

Hi John

Thanks for pointing me to the resources purge option - i was not aware
of it.

However I get this error then trying it

err: Failed to apply catalog: Parameter purge failed: Purging is only
supported on types that accept 'ensure' at /etc/puppet/modules/yum/
manifests/init.pp:12

So I guess I'd stick with Nan solution for now.

Thanks

[piavlo]

http://projects.puppetlabs.com/issues/9277

> > For more options, visit this group athttp://groups.google.com/group/puppet-users?hl=en.

[jcbollinger]

On Aug 31, 6:41 am, piavlo <lolitus...@gmail.com> wrote:
>  Hi John
>
> Thanks for pointing me to the resources purge option - i was not aware
> of it.
>
> However I get this error then trying it
>
> err: Failed to apply catalog: Parameter purge failed: Purging is only
> supported on types that accept 'ensure' at /etc/puppet/modules/yum/
> manifests/init.pp:12

Oops. For what it's worth, you might consider voting for feature
request #949 and/or feature request #1023. Both have been accepted,
but neither yet implemented.


John