lvm_support.rb file fails to load correctly with an execution expired warning in the logs

[karl....@ammeon.com]

Hi all,

We are seeing the following "execution expired" warning in the logs intermittently during a puppet cycle:

Apr 25 19:06:53 Node1 puppet-agent[20513]: Could not load fact file /var/lib/puppet/lib/facter/lvm_support.rb: execution expired

On further investigation and with some logging in place, we were able to determine that each of the facts (from the lvm_support.rb file) were being loaded at 19:04:52 
(2 minutes before the warning appeared with not time delay), which would suggest that there was no hanging while running the code for any specific facts from the lvm_support.rb file.

The warning has tended to appear in early puppet runs just after puppet has been enabled. 
Future puppet cycles don't report any warnings and facter seems to be able to retrieve values for the facts in the lvm_support.rb file.

Has anyone seen this behaviour before?

thanks,

Karl

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.

[karl....@ammeon.com]

Just to add some context:

puppet --version ==> 3.3.2
facter --version ==> 1.6.18
ruby --version ==> ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux]
uname -a ==> Linux ms1 2.6.32-504.el6.x86_64 #1 SMP Tue Sep 16 01:56:35 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[jcbollinger]

On Tuesday, May 3, 2016 at 5:57:43 AM UTC-5, karl....@ammeon.com wrote:

Just to add some context:

puppet --version ==> 3.3.2
facter --version ==> 1.6.18
ruby --version ==> ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux]
uname -a ==> Linux ms1 2.6.32-504.el6.x86_64 #1 SMP Tue Sep 16 01:56:35 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

The facts to which you refer are not Facter built-ins: they are part of the external puppetlabs-lvm module, which has separate versioning from Puppet and Facter.  It might be helpful to know which version of that module you are using.  It might also be helpful to know which version of LVM you're using on the affected systems.  The most recent changes to the custom fact code are described as making the custom facts work with newer LVMs; I am uncertain whether they break it for any older LVMs, such those I might expect to find on an EL6 machine.

Are you certain, however, that Facter is loading all of the LVM facts?  Looking at the (current) fact code, it appears that that would be:

• lvm_support
• lvm_vgs
• for each volume group,
• lvm_vg_#i
• lvm_vg_#name_pvs (only in recent versions)
• lvm_pvs
• for each physical volume,
• lvm_pv_#i
  

To do that, a separate external command is run for each physical volume and each volume group, and if you have several VGs or several PVs then you might miss that one or more of those does not complete in a timely manner.  If that's the nature of the problem, then you should, in principle, be able to reproduce the observed behavior by running Facter directly from the command line.  For a reliable test, be sure to run Facter as the agent would do, especially with respect to user, security context, and environment.


John

[Karl Murphy]

Hi John, the logging would suggest that there is no hanging while loading the LVM facts. All the facts appear to load. Obviously, as we are not using the most recent puppetlabs-lvm module, we don't have the component relating to lvm_vg_#name_pvs as you highlight above.

Regarding lvm version and the module version we are using:

1) lvm version
    LVM version:     2.02.111(2)-RHEL6 (2014-09-01)
    Library version: 1.02.89-RHEL6 (2014-09-01)
    Driver version:  4.27.0

2) puppetlabs-lvm version ==> '0.2.0'

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/7WGA4UNhNvA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/6ba7bd93-5670-4cce-8280-96c34b5e36f7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[jcbollinger]

On Tuesday, May 3, 2016 at 9:20:39 AM UTC-5, Karl Murphy wrote:

Hi John, the logging would suggest that there is no hanging while loading the LVM facts. All the facts appear to load. Obviously, as we are not using the most recent puppetlabs-lvm module, we don't have the component relating to lvm_vg_#name_pvs as you highlight above.

Regarding lvm version and the module version we are using:

1) lvm version
    LVM version:     2.02.111(2)-RHEL6 (2014-09-01)
    Library version: 1.02.89-RHEL6 (2014-09-01)
    Driver version:  4.27.0

2) puppetlabs-lvm version ==> '0.2.0'

Googling around a little, I found this old thread: https://groups.google.com/forum/#!topic/puppet-users/8GzIjCxfntQ, which relates to a similar execution expired problem, and whose author traces it back to a purported problem with the Ruby gtk2 module interfering with reading from sysfs.  It doesn't sound so likely that you would be loading gtk2 specifically, but it is entirely plausible that evaluating the LVM facts might involve reading from sysfs, and that Ruby modules other than gtk2 might also cause such trouble.  It is also plausible that the problem might disappear after one Puppet run as a result of some key package being updated.

I'm afraid that's not very solid, but it is testable.  For example, provision a fresh machine as you normally do, up to, but not including the point where you perform the first Puppet run.  Run Puppet in --noop mode to try to observe the error without applying any changes.  Even in noop mode, the agent should still issue a CSR, perform pluginsync, and load facts.  If the error is inconsistent then you may need to make multiple tries, or otherwise try different initial provisioning -- I'm sure you get the idea.  If this problem is associated with the clean-from-provisioning state of your machines, then you should not only be able to reproduce it this way, but probably you should also be able to reproduce it multiple times with the same machine, as long as the agent runs in noop mode.  If you can in fact establish that, then from there it's probably a question of determining which package update or configuration change makes the problem go away.


John