Random Yum errors during provisioning

[Glenn Poston]

Running Amazon Linux (which is essentially Centos5.5).

Anyone seen random yum errors like this one?  I don't think it's necessarily related to Puppet, but it randomly fails my puppet runs and I don't know how to fix it.

Jun 28 08:41:34 ip-10-159-65-145 run_puppet: [Notice: /Stage[main]/Zookeeper/Package[zookeeper]/ensure: created

Jun 28 08:41:34 ip-10-159-65-145 run_puppet: [Notice: /Stage[main]/Zookeeper/File[/var/lib/zookeeper/data]/ensure: created

Jun 28 08:41:34 ip-10-159-65-145 run_puppet: [Notice: /Stage[main]/Zookeeper/File[/var/lib/zookeeper/data/myid]/ensure: created

Jun 28 08:41:34 ip-10-159-65-145 run_puppet: [Notice: /Stage[main]/Yum_repo::Configs/File[/etc/yum.repos.d/inin-epel.repo]/ensure: defined content as '{md5}b94171f63e31f07b8bd75444073e301c'

Jun 28 08:41:35 ip-10-159-65-145 run_puppet: [Notice: /Stage[main]/Zookeeper/File[/etc/zookeeper/zookeeper-env.sh]/content: content changed '{md5}cd666c7520ce5279ddbc185512b0b177' to '{md5}5cb59b25f5e7567d94ba14b06f6e7081'

Jun 28 08:41:38 ip-10-159-65-145 run_puppet: [Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install daemonize' returned 1: Existing lock /var/run/yum.pid: another copy is running as pid 2502.

Jun 28 08:41:38 ip-10-159-65-145 run_puppet: Another app is currently holding the yum lock; waiting for it to exit...

Jun 28 08:41:38 ip-10-159-65-145 run_puppet:   The other application is: yum

Jun 28 08:41:38 ip-10-159-65-145 run_puppet:     Memory :  40 M RSS (235 MB VSZ)

Jun 28 08:41:38 ip-10-159-65-145 run_puppet:     Started: Fri Jun 28 08:41:33 2013 - 00:03 ago

Jun 28 08:41:38 ip-10-159-65-145 run_puppet:     State  : Running, pid: 2502

Jun 28 08:41:38 ip-10-159-65-145 run_puppet: Error: database disk image is malformed

Jun 28 08:41:38 ip-10-159-65-145 run_puppet: [Error: /Stage[main]/Mcollective/Package[daemonize]/ensure: change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install daemonize' returned 1: Existing lock /var/run/yum.pid: another copy is running as pid 2502.

Jun 28 08:41:38 ip-10-159-65-145 run_puppet: Another app is currently holding the yum lock; waiting for it to exit...

Jun 28 08:41:38 ip-10-159-65-145 run_puppet:   The other application is: yum

Jun 28 08:41:38 ip-10-159-65-145 run_puppet:     Memory :  40 M RSS (235 MB VSZ)

Jun 28 08:41:38 ip-10-159-65-145 run_puppet:     Started: Fri Jun 28 08:41:33 2013 - 00:03 ago

Jun 28 08:41:38 ip-10-159-65-145 run_puppet:     State  : Running, pid: 2502

Jun 28 08:41:38 ip-10-159-65-145 run_puppet: Error: database disk image is malformed

Jun 28 08:41:38 ip-10-159-65-145 run_puppet: [mNotice: /Stage[main]/Zookeeper/File[/etc/zookeeper/zoo.cfg]/content: content changed '{md5}5c543298c5572c3caf40a3d108309019' to '{md5}31db609f6601a8a02561d411e98db12b'

Jun 28 08:41:39 ip-10-159-65-145 run_puppet: [mNotice: /Stage[main]/Puppet/File[/usr/local/bin/run_puppet.sh]/content: content changed '{md5}4e9496313a0b4152c663defce5100af5' to '{md5}49d78e473fa2202dea13e9b195e63575'

Jun 28 08:41:39 ip-10-159-65-145 run_puppet: [mNotice: /Stage[main]/Yum_repo::Configs/Exec[puppet_repo]/returns: executed successfully

Jun 28 08:41:48 ip-10-159-65-145 run_puppet: [mNotice: /Stage[main]/Mcollective::Common/Package[mcollective-package-agent]/ensure: created

The problem does not persist.  Yum packages are installed by puppet before and after the errors.  A subsequent puppet run installs the previously skipped packages fine.

It's as if some background process creates a lock, while updating the yum DB, but when the lock is released, the yum DB is still in a bad state (momentarily).

[Glenn Poston]

Any ideas???

[jcbollinger]

The diagnostic message claims that it is yum itself holding the lock.  Do you have an automatic package update daemon running on the affected nodes?  Or even an update checker, such as yum-updatesd?  Those sorts of tools wrap yum, so they can interfere with other usage of yum.  If you don't want such programs to interfere with Puppet then you need to ensure that they do not run concurrently with the Puppet agent.


John

[Glenn Poston]

We've disabled yum-updatesd on our nodes.

Thanks John,

Further investigation has shown that after any package update/install, yum automatically runs the following command…

/usr/bin/python /usr/bin/yum --security check-update

I'm curious if this is the issue.

Either way, it's odd that that the error is 'database disk image is malformed' and not 'Existing lock on yum.pid'

It's as if the yum is releasing the lock before the database is ready for subsequent requests.

~Glenn

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/B5M5SaLp7rA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

[jcbollinger]

On Wednesday, September 4, 2013 9:41:46 AM UTC-5, Glenn Poston wrote:

We've disabled yum-updatesd on our nodes.

Thanks John,

Further investigation has shown that after any package update/install, yum automatically runs the following command…

/usr/bin/python /usr/bin/yum --security check-update

I'm curious if this is the issue.

That does look like a good candidate.  I'm pretty sure it's not a core yum behavior, but rather one associated with an installed yum plugin, probably yum-security.  You may be able to disable it via the Yum config file, or else you could just remove the plugin.

 

Either way, it's odd that that the error is 'database disk image is malformed' and not 'Existing lock on yum.pid'

It's as if the yum is releasing the lock before the database is ready for subsequent requests.

 
That could be, but it could also be that the message is just erroneous.


John

[Stuart Cracraft]

Did you do a ps -ef | grep yum? It tells you the pid. I wouldn't kill it. Let it finish its other work.

If the pid isn't present, remove the pid-file and retry. If that fails, you have some serious 

troubleshooting to do tracing the process, googling the web for the error message, and other

drill-down.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.