Puppet client removing yum / rpm cache?

[Christopher Opena]

Howdy folks,

I've got some weird behavior from one of my puppet clients (actually the client running on the puppet master), where running the agent somehow destroys either the yum cache or the rpm db.  Directly after running the agent I get the following message from yum (without modifying yum.conf or any of the .repo files in /etc/yum.repos.d/):

# yum list installed

YumRepo Error: All mirror URLs are not using ftp, http[s] or file.

 Eg. $releasever is not a valid release or hasnt been released yet/

removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt

Error: Cannot find a valid baseurl for repo: base

Prior to running the puppet agent, yum was working perfectly.  I was able to fix the above problem by reinstalling the centos-release package, but even after doing so a 'yum list installed' and 'rpm -qa' listed the centos-release package as *the only package installed* (clearly others were installed but the db probably got corrupted).

Restart the puppet agent, and the error with $releasever resumed, and centos-release was shown as not installed by both yum and rpm.

Looking through the puppet agent log (/var/log/puppet/puppet.log), I only found this in the startup procedure:

Thu Feb 20 14:26:50 -0800 2014 Puppet (notice): Caught TERM; calling stop

Thu Feb 20 14:26:51 -0800 2014 Puppet (notice): Reopening log files

Thu Feb 20 14:26:51 -0800 2014 Puppet (notice): Starting Puppet client version 3.4.2

Thu Feb 20 14:26:52 -0800 2014 /File[/var/lib/rpm]/ensure (notice): removed

Thu Feb 20 14:26:52 -0800 2014 /File[/var/lib/yum]/ensure (notice): removed

Thu Feb 20 14:26:55 -0800 2014 Puppet (err): Could not prefetch package provider 'yum': Execution of '/usr/bin/python /usr/lib/ruby/site_ruby/1.8/puppet/provider/package/yumhelper.py' returned 1: Loaded plugins: fastestmirror

So it looks like directly after the client starts, it removes(?) /var/lib/rpm and /var/lib/yum and then yum stops working again (with the same $releasever) error.  The odd thing is that we have several other nodes that are having the exact same modules applied and experiencing zero issues, so something odd is going on with this particular node (which is also the master).

Anyone experience anything like this or have any ideas what could be causing the puppet runs to basically destroy the yumdb?

TIA,

Chris

[jcbollinger]

On Thursday, February 20, 2014 5:03:53 PM UTC-6, Christopher Opena wrote:

Howdy folks,

I've got some weird behavior from one of my puppet clients (actually the client running on the puppet master), where running the agent somehow destroys either the yum cache or the rpm db.  Directly after running the agent I get the following message from yum (without modifying yum.conf or any of the .repo files in /etc/yum.repos.d/):

# yum list installed

YumRepo Error: All mirror URLs are not using ftp, http[s] or file.

 Eg. $releasever is not a valid release or hasnt been released yet/

removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt

Error: Cannot find a valid baseurl for repo: base

Prior to running the puppet agent, yum was working perfectly.  I was able to fix the above problem by reinstalling the centos-release package, but even after doing so a 'yum list installed' and 'rpm -qa' listed the centos-release package as *the only package installed* (clearly others were installed but the db probably got corrupted).

Restart the puppet agent, and the error with $releasever resumed, and centos-release was shown as not installed by both yum and rpm.

Looking through the puppet agent log (/var/log/puppet/puppet.log), I only found this in the startup procedure:

Thu Feb 20 14:26:50 -0800 2014 Puppet (notice): Caught TERM; calling stop

Thu Feb 20 14:26:51 -0800 2014 Puppet (notice): Reopening log files

Thu Feb 20 14:26:51 -0800 2014 Puppet (notice): Starting Puppet client version 3.4.2

Thu Feb 20 14:26:52 -0800 2014 /File[/var/lib/rpm]/ensure (notice): removed

Thu Feb 20 14:26:52 -0800 2014 /File[/var/lib/yum]/ensure (notice): removed

Thu Feb 20 14:26:55 -0800 2014 Puppet (err): Could not prefetch package provider 'yum': Execution of '/usr/bin/python /usr/lib/ruby/site_ruby/1.8/puppet/provider/package/yumhelper.py' returned 1: Loaded plugins: fastestmirror

So it looks like directly after the client starts, it removes(?) /var/lib/rpm and /var/lib/yum and then yum stops working again (with the same $releasever) error.

Wow.

 

 The odd thing is that we have several other nodes that are having the exact same modules applied and experiencing zero issues, so something odd is going on with this particular node (which is also the master).

Anyone experience anything like this or have any ideas what could be causing the puppet runs to basically destroy the yumdb?

I have never heard of such behavior.

The problem likely is not directly related to your manifests, because it looks like the agent is removing the rpm and yum databases before it starts applying the catalog.  I would start by looking for differences between the puppet configuration files (puppet.conf) on the master and on a node that does not experience the problem.  Focus on differences in the [main] and [agent] sections, if there are any.

It is also possible that your Puppet installation is corrupted or compromised, though the likelihood of the latter seems remote.  It is unfortunate that destruction of /var/lib/rpm will make those alternatives harder to evaluate.


John

[Christopher Opena]

John,

Just wanted to drop a quick note to say thanks for responding and for the great variety of ideas to try to troubleshoot.  At the end of the day, the environment was so hosed that I eventually just rebuilt the puppetmaster from scratch with a setup script to ensure lack of errors in either sysconfig, puppet.conf, or any other location driving puppet.  Worked nicely this time :)

Cheers,

C

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/ab0f3c82-cacb-458d-a76b-33943495b85a%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.