Jira (PUP-2515) Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists)

[Den Mat (JIRA)]

Den Mat created an issue

Puppet / PUP-2515 Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists) Issue Type: Bug Affects Versions: 3.4.3 Assignee: Eric Sorenson Components: Community Created: 07/May/14 5:17 PM Environment: Oracle Linux 6.3 puppet-3.4.3-1.el6.noarch Labels: puppet-agent Priority: Normal Reporter: Den Mat Puppet stops responding on the client. We issue the following: $ sudo service puppet restart Stopping puppet agent: [ OK ] Starting puppet agent: [ OK ] $ sudo puppet agent --test Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists) $ date Thu May 8 10:14:41 EST 2014 $ ps aux |grep pupp root 12838 0.0 4.7 200780 96872 ? S May07 0:21 puppet agent: applying configuration $ sudo strace -p 12838 Process 12838 attached - interrupt to quit select(5, [4], NULL, NULL, NULL Wed May 07 06:25:30 +1000 2014 Puppet (notice): Finished catalog run in 207.42 seconds Wed May 07 06:55:12 +1000 2014 Puppet (notice): Finished catalog run in 204.99 seconds Thu May 08 03:29:01 +1000 2014 Puppet (notice): Caught HUP; calling restart This happens on for remote sites predominantly with 'flakey' but not troublesome connectivity. Happening enough to warrant an issue being raised. It may have been happening in prior releases but we have changed the way we monitor puppet agents now so it could be that we are just seeing it now - we updated from 2.6.16 to 3.4.3. I will try to get more detail but generally I can't reproduce and have to wait for it to happen. Add Comment

This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede)

[Den Mat (JIRA)]

Den Mat commented on an issue

Re: Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists) Here is another one: $ ps aux |grep pup root 29958 0.0 0.0 132156 45600 ? Ss May11 0:01 /usr/bin/ruby /usr/bin/puppet agent --server=puppet --logdest=/var/log/puppet/puppet.log --masterport=8140 root 30397 0.0 0.0 202160 98300 ? S May11 0:09 puppet agent: applying configuration $ sudo strace -p 30397 Process 30397 attached - interrupt to quit select(5, [4], NULL, NULL, NULL^C <unfinished ...> Process 30397 detached $ sudo strace -p 29958 Process 29958 attached - interrupt to quit wait4(30397, $ sudo netstat -p |grep 30397 tcp 0 0 <host>:56626 <ip>:8140 ESTABLISHED 30397/puppet agent $ sudo ls -l /proc/30397/fd total 0 lr-x------. 1 root root 64 May 12 10:03 0 -> /dev/null l-wx------. 1 root root 64 May 12 10:03 1 -> /dev/null l-wx------. 1 root root 64 May 12 10:03 2 -> /dev/null l-wx------. 1 root root 64 May 12 10:03 3 -> /var/log/puppet/puppet.log-20140512 (deleted) lrwx------. 1 root root 64 May 12 10:03 4 -> socket:[38371182] $ sudo cat /proc/30397/io rchar: 188095331 wchar: 1431794 syscr: 82612 syscw: 2423 read_bytes: 0 write_bytes: 2322432 cancelled_write_bytes: 274432 $ ruby --version ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] Sits there and does nothing. Have to stop the puppet service and delete the lock file to recover.

Add Comment

Puppet / PUP-2515 Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists)

Puppet stops responding on the client. We issue the following:

{code}

$ sudo service puppet restart Stopping puppet agent: [ OK ] Starting puppet agent: [ OK ]

{code} {code}

$ sudo puppet agent --test

Notice: Run of Puppet configuration client already in progress; skipping...

[Andy Parker (JIRA)]

Andy Parker updated an issue

Puppet / PUP-2515 Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists)

Change By: Andy Parker Component/s: Community Component/s: Catalog Application

Add Comment

[Joshua Cooper (JIRA)]

Joshua Cooper commented on an issue

Re: Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists) Den Mat When the problem occurs, can you check the contents of the agent's lock file: cat `puppet agent --configprint agent_catalog_run_lockfile`. I'm specifically wondering if it's empty (as described in PUP-1070), or contains the pid of the currently running, but blocked agent. You're seeing two processes, because the agent forks and applies the catalog in a child process, and the child is stuck in select trying to either connect or read from the master, while the parent process waits for the child process to exit. However, puppet sets both a TCP connect and read timeout based on the configtimeout setting, which defaults to 2 minutes. So I would not expect the agent to block indefinitely. The 2 minute read timeout is actually surprising to me... When this happens again, can you verify that the master has an established connection from the agent? If not, then it sounds like the agent's blocking connect or read system call was not interrupted. Also, can you confirm that you are not using a load-balancer to front your puppetmaster?

Add Comment

Puppet / PUP-2515 Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists)

Puppet stops responding on the client. We issue the following: {code} $ sudo service puppet restart Stopping puppet agent: [ OK ] Starting puppet agent: [ OK ] {code} {code} $ sudo puppet agent --test

Notice: Run of Puppet configuration client already in progress; skipping...

[Den Mat (JIRA)]

Den Mat commented on an issue

Re: Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists)

Sorry, I didn't get an email seeing your comment (or if I did I deleted it!). As luck would have it I have another stuck one today. $ ps aux |grep pupp root 3535 0.0 4.8 205648 99124 ? S Jun04 0:13 puppet agent: applying configuration puppet 4366 0.0 6.2 230604 128708 ? S Jun04 0:14 Rack: /var/www/html/puppet/rack puppet 4619 0.0 3.7 178604 76636 ? S Jun04 0:01 Rack: /var/www/html/puppet/rack puppet 6153 0.0 3.3 163400 68300 ? S Jun04 0:06 master 533 10471 0.0 0.0 10084 556 ? S Apr28 0:00 git --git-dir /var/cache/r10k/https---github.com-puppetlabs-puppetlabs-firewall.git fetch --prune 533 10472 0.0 0.1 90648 2444 ? S Apr28 0:00 git-remote-https origin https://github.com/puppetlabs/puppetlabs-firewall.git 533 10475 0.0 0.0 11608 588 ? S Apr28 0:00 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://github.com/puppetlabs/puppetlabs-firewall.git efs/pull/350/head efs/pull/350/merge root 19141 0.0 2.2 132528 45816 ? Ss Jun04 0:00 /usr/bin/ruby /usr/bin/puppet agent --server=puppet.drp.acx --logdest=/var/log/puppet/puppet.log --masterport=8140 root 20843 0.0 0.1 198568 2952 ? S Apr10 0:08 puppet agent: applying configuration puppet 24532 0.0 1.8 238644 38992 ? S May21 0:26 Rack: /var/www/html/puppet/rack puppet 24568 0.0 2.2 226748 45344 ? S May21 0:13 Rack: /var/www/html/puppet/rack puppet 24602 0.0 0.3 165428 6784 ? S May21 0:00 Rack: /var/www/html/puppet/rack root 24638 0.0 0.6 145468 12636 ? S May21 0:00 puppet agent: applying configuration puppet 31670 51.7 6.3 233016 129980 ? S 00:43 0:24 Rack: /var/www/html/puppet/rack user@<dedacted>:/home/user [PRODUCTION] $ sudo puppet agent --configprint agent_catalog_run_lockfile /var/lib/puppet/state/agent_catalog_run.lock user@<dedacted>:/home/user [PRODUCTION] $ cat /var/lib/puppet/state/agent_catalog_run.lock cat: /var/lib/puppet/state/agent_catalog_run.lock: Permission denied user@<dedacted>:/home/user [PRODUCTION] $ sudo cat /var/lib/puppet/state/agent_catalog_run.lock 3535 user@<dedacted>:/home/user [PRODUCTION] $ sudo puppet agent --test --debug Debug: Puppet::Type::User::ProviderLdap: true value when expecting false Debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not exist Debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dsimport does not exist Debug: Puppet::Type::User::ProviderPw: file pw does not exist Debug: Using settings: adding file resource 'hostpubkey': 'File[/var/lib/puppet/ssl/public_keys/<dedacted>.pem]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"644", :backup=>false, :path=>"/var/lib/puppet/ssl/public_keys/<dedacted>.pem", :ensure=>:file}' Debug: Using settings: adding file resource 'requestdir': 'File[/var/lib/puppet/ssl/certificate_requests]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :backup=>false, :path=>"/var/lib/puppet/ssl/certificate_requests", :ensure=>:directory}' Debug: Using settings: adding file resource 'client_datadir': 'File[/var/lib/puppet/client_data]{:loglevel=>:debug, :links=>:follow, :mode=>"750", :backup=>false, :path=>"/var/lib/puppet/client_data", :ensure=>:directory}' Debug: Using settings: adding file resource 'lastrunfile': 'File[/var/lib/puppet/state/last_run_summary.yaml]{:loglevel=>:debug, :links=>:follow, :mode=>"644", :backup=>false, :path=>"/var/lib/puppet/state/last_run_summary.yaml", :ensure=>:file}' Debug: Using settings: adding file resource 'privatedir': 'File[/var/lib/puppet/ssl/private]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"750", :backup=>false, :path=>"/var/lib/puppet/ssl/private", :ensure=>:directory}' Debug: Using settings: adding file resource 'resourcefile': 'File[/var/lib/puppet/state/resources.txt]{:loglevel=>:debug, :links=>:follow, :owner=>"root", :mode=>"640", :backup=>false, :path=>"/var/lib/puppet/state/resources.txt", :ensure=>:file}' Debug: Puppet::Type::Group::ProviderLdap: true value when expecting false Debug: Puppet::Type::Group::ProviderDirectoryservice: file /usr/bin/dscl does not exist Debug: Puppet::Type::Group::ProviderPw: file pw does not exist Debug: Using settings: adding file resource 'rundir': 'File[/var/run/puppet]{:group=>"puppet", :loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"755", :backup=>false, :path=>"/var/run/puppet", :ensure=>:directory}' Debug: Using settings: adding file resource 'hostcrl': 'File[/var/lib/puppet/ssl/crl.pem]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"644", :backup=>false, :path=>"/var/lib/puppet/ssl/crl.pem", :ensure=>:file}' Debug: Using settings: adding file resource 'certdir': 'File[/var/lib/puppet/ssl/certs]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :backup=>false, :path=>"/var/lib/puppet/ssl/certs", :ensure=>:directory}' Debug: Using settings: adding file resource 'logdir': 'File[/var/log/puppet]{:group=>"puppet", :loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"750", :backup=>false, :path=>"/var/log/puppet", :ensure=>:directory}' Debug: Using settings: adding file resource 'hostprivkey': 'File[/var/lib/puppet/ssl/private_keys/<dedacted>.pem]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"600", :backup=>false, :path=>"/var/lib/puppet/ssl/private_keys/<dedacted>.pem", :ensure=>:file}' Debug: Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :backup=>false, :path=>"/var/lib/puppet/ssl/public_keys", :ensure=>:directory}' Debug: Using settings: adding file resource 'clientyamldir': 'File[/var/lib/puppet/client_yaml]{:loglevel=>:debug, :links=>:follow, :mode=>"750", :backup=>false, :path=>"/var/lib/puppet/client_yaml", :ensure=>:directory}' Debug: Using settings: adding file resource 'libdir': 'File[/var/lib/puppet/lib]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/lib", :ensure=>:directory}' Debug: Using settings: adding file resource 'vardir': 'File[/var/lib/puppet]{:group=>"puppet", :loglevel=>:debug, :links=>:follow, :owner=>"puppet", :backup=>false, :path=>"/var/lib/puppet", :ensure=>:directory}' Debug: Using settings: adding file resource 'localcacert': 'File[/var/lib/puppet/ssl/certs/ca.pem]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"644", :backup=>false, :path=>"/var/lib/puppet/ssl/certs/ca.pem", :ensure=>:file}' Debug: Using settings: adding file resource 'privatekeydir': 'File[/var/lib/puppet/ssl/private_keys]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"750", :backup=>false, :path=>"/var/lib/puppet/ssl/private_keys", :ensure=>:directory}' Debug: Using settings: adding file resource 'hiera_config': 'File[/etc/puppet/hiera.yaml]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/etc/puppet/hiera.yaml", :ensure=>:file}' Debug: Using settings: adding file resource 'classfile': 'File[/var/lib/puppet/classes.txt]{:loglevel=>:debug, :links=>:follow, :owner=>"root", :mode=>"640", :backup=>false, :path=>"/var/lib/puppet/classes.txt", :ensure=>:file}' Debug: Using settings: adding file resource 'lastrunreport': 'File[/var/lib/puppet/state/last_run_report.yaml]{:loglevel=>:debug, :links=>:follow, :mode=>"640", :backup=>false, :path=>"/var/lib/puppet/state/last_run_report.yaml", :ensure=>:file}' Debug: Using settings: adding file resource 'clientbucketdir': 'File[/var/lib/puppet/clientbucket]{:loglevel=>:debug, :links=>:follow, :mode=>"750", :backup=>false, :path=>"/var/lib/puppet/clientbucket", :ensure=>:directory}' Debug: Using settings: adding file resource 'pluginfactdest': 'File[/var/lib/puppet/facts.d]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/facts.d", :ensure=>:directory}' Debug: Using settings: adding file resource 'graphdir': 'File[/var/lib/puppet/state/graphs]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/state/graphs", :ensure=>:directory}' Debug: Using settings: adding file resource 'hostcert': 'File[/var/lib/puppet/ssl/certs/<dedacted>.pem]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"644", :backup=>false, :path=>"/var/lib/puppet/ssl/certs/<dedacted>.pem", :ensure=>:file}' Debug: Using settings: adding file resource 'ssldir': 'File[/var/lib/puppet/ssl]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"771", :backup=>false, :path=>"/var/lib/puppet/ssl", :ensure=>:directory}' Debug: Using settings: adding file resource 'statefile': 'File[/var/lib/puppet/state/state.yaml]{:loglevel=>:debug, :links=>:follow, :mode=>"660", :backup=>false, :path=>"/var/lib/puppet/state/state.yaml", :ensure=>:file}' Debug: Using settings: adding file resource 'statedir': 'File[/var/lib/puppet/state]{:loglevel=>:debug, :links=>:follow, :mode=>"1755", :backup=>false, :path=>"/var/lib/puppet/state", :ensure=>:directory}' Debug: Using settings: adding file resource 'confdir': 'File[/etc/puppet]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/etc/puppet", :ensure=>:directory}' Debug: /File[/var/lib/puppet/client_yaml]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/client_data]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/state/state.yaml]: Autorequiring File[/var/lib/puppet/state] Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/public_keys/<dedacted>.pem]: Autorequiring File[/var/lib/puppet/ssl/public_keys] Debug: /File[/var/lib/puppet/facts.d]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet] Debug: /File[/etc/puppet/hiera.yaml]: Autorequiring File[/etc/puppet] Debug: /File[/var/lib/puppet/ssl/private_keys/<dedacted>.pem]: Autorequiring File[/var/lib/puppet/ssl/private_keys] Debug: /File[/var/lib/puppet/state/graphs]: Autorequiring File[/var/lib/puppet/state] Debug: /File[/var/lib/puppet/state/last_run_report.yaml]: Autorequiring File[/var/lib/puppet/state] Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring File[/var/lib/puppet/ssl/certs] Debug: /File[/var/lib/puppet/state/resources.txt]: Autorequiring File[/var/lib/puppet/state] Debug: /File[/var/lib/puppet/state/last_run_summary.yaml]: Autorequiring File[/var/lib/puppet/state] Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/certs/<dedacted>.pem]: Autorequiring File[/var/lib/puppet/ssl/certs] Debug: /File[/var/lib/puppet/clientbucket]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/classes.txt]: Autorequiring File[/var/lib/puppet] Debug: Finishing transaction 70113082448100 Debug: Using settings: adding file resource 'hostpubkey': 'File[/var/lib/puppet/ssl/public_keys/<dedacted>.pem]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"644", :backup=>false, :path=>"/var/lib/puppet/ssl/public_keys/<dedacted>.pem", :ensure=>:file}' Debug: Using settings: adding file resource 'requestdir': 'File[/var/lib/puppet/ssl/certificate_requests]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :backup=>false, :path=>"/var/lib/puppet/ssl/certificate_requests", :ensure=>:directory}' Debug: Using settings: adding file resource 'privatedir': 'File[/var/lib/puppet/ssl/private]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"750", :backup=>false, :path=>"/var/lib/puppet/ssl/private", :ensure=>:directory}' Debug: Using settings: adding file resource 'rundir': 'File[/var/run/puppet]{:group=>"puppet", :loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"755", :backup=>false, :path=>"/var/run/puppet", :ensure=>:directory}' Debug: Using settings: adding file resource 'hostcrl': 'File[/var/lib/puppet/ssl/crl.pem]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"644", :backup=>false, :path=>"/var/lib/puppet/ssl/crl.pem", :ensure=>:file}' Debug: Using settings: adding file resource 'certdir': 'File[/var/lib/puppet/ssl/certs]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :backup=>false, :path=>"/var/lib/puppet/ssl/certs", :ensure=>:directory}' Debug: Using settings: adding file resource 'logdir': 'File[/var/log/puppet]{:group=>"puppet", :loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"750", :backup=>false, :path=>"/var/log/puppet", :ensure=>:directory}' Debug: Using settings: adding file resource 'hostprivkey': 'File[/var/lib/puppet/ssl/private_keys/<dedacted>.pem]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"600", :backup=>false, :path=>"/var/lib/puppet/ssl/private_keys/<dedacted>.pem", :ensure=>:file}' Debug: Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :backup=>false, :path=>"/var/lib/puppet/ssl/public_keys", :ensure=>:directory}' Debug: Using settings: adding file resource 'libdir': 'File[/var/lib/puppet/lib]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/lib", :ensure=>:directory}' Debug: Using settings: adding file resource 'vardir': 'File[/var/lib/puppet]{:group=>"puppet", :loglevel=>:debug, :links=>:follow, :owner=>"puppet", :backup=>false, :path=>"/var/lib/puppet", :ensure=>:directory}' Debug: Using settings: adding file resource 'localcacert': 'File[/var/lib/puppet/ssl/certs/ca.pem]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"644", :backup=>false, :path=>"/var/lib/puppet/ssl/certs/ca.pem", :ensure=>:file}' Debug: Using settings: adding file resource 'privatekeydir': 'File[/var/lib/puppet/ssl/private_keys]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"750", :backup=>false, :path=>"/var/lib/puppet/ssl/private_keys", :ensure=>:directory}' Debug: Using settings: adding file resource 'hiera_config': 'File[/etc/puppet/hiera.yaml]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/etc/puppet/hiera.yaml", :ensure=>:file}' Debug: Using settings: adding file resource 'pluginfactdest': 'File[/var/lib/puppet/facts.d]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/facts.d", :ensure=>:directory}' Debug: Using settings: adding file resource 'hostcert': 'File[/var/lib/puppet/ssl/certs/<dedacted>.pem]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"644", :backup=>false, :path=>"/var/lib/puppet/ssl/certs/<dedacted>.pem", :ensure=>:file}' Debug: Using settings: adding file resource 'ssldir': 'File[/var/lib/puppet/ssl]{:loglevel=>:debug, :links=>:follow, :owner=>"puppet", :mode=>"771", :backup=>false, :path=>"/var/lib/puppet/ssl", :ensure=>:directory}' Debug: Using settings: adding file resource 'statedir': 'File[/var/lib/puppet/state]{:loglevel=>:debug, :links=>:follow, :mode=>"1755", :backup=>false, :path=>"/var/lib/puppet/state", :ensure=>:directory}' Debug: Using settings: adding file resource 'confdir': 'File[/etc/puppet]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/etc/puppet", :ensure=>:directory}' Debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring File[/var/lib/puppet/ssl/certs] Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/etc/puppet/hiera.yaml]: Autorequiring File[/etc/puppet] Debug: /File[/var/lib/puppet/facts.d]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/private_keys/<dedacted>.pem]: Autorequiring File[/var/lib/puppet/ssl/private_keys] Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/certs/<dedacted>.pem]: Autorequiring File[/var/lib/puppet/ssl/certs] Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/public_keys/<dedacted>.pem]: Autorequiring File[/var/lib/puppet/ssl/public_keys] Debug: Finishing transaction 70113092545920 Debug: Using cached certificate for ca Debug: Using cached certificate for <dedacted> Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists)

Add Comment

[Den Mat (JIRA)]

Den Mat commented on an issue

Re: Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists)

When this happens again, can you verify that the master has an established connection from the agent? If not, then it sounds like the agent's blocking connect or read system call was not interrupted. On the master: $ sudo netstat -lpta |grep 192.168.1xx.1xx <no result>

Also, can you confirm that you are not using a load-balancer to front your puppetmaster?

I will explain the set up a little. We don't have loadbalancers in front of our masters. We have a central CA and remote puppet masters. This host in the above comment is one of the remote puppet masters. The remote puppet masters call into the CA to when they do their puppet run. Having the remote masters has significantly improved this issue in our remote DCs and now it appears that only remote masters are having the problem (as they are the only ones making calls back to the CA). All are apache/passenger. Most connect over our VPN. This host uses our VPN and it admittedly has a stable but high latency connection.

Add Comment