Nodes Could not retrieve catalog from remote server: execution expired

[iamauser]

Almost 4-5% of the total number of nodes are not receiving catalog from the master on an hourly run. We have about 250 nodes. The nodes that are receiving this error are rather random. The hourly cron happens almost at the same time. Is there any configuration changes for Puppet that can be done to avoid this ?

Puppet (err): Could not retrieve catalog from remote server: execution expired

Puppet (notice): Using cached catalog

/File[secure_host] (err): Could not evaluate: Connection reset by peer - SSL_connect Could not retrieve file metadata for puppet:///modules/certs/<filename>: Connection reset by peer - SSL_connect

Thanks for any suggestion.

-----------

[jcbollinger]

It sounds like you are pushing your master's capacity limit.  If your clients are all trying to check in at about the same time then you should be able to make more efficient use of your available resources by spreading the client check-ins more evenly over time.

Supposing that your Puppet cron jobs are managed by Puppet, you can make use of Puppet's built-in fqdn_rand() function to help generate launch times for your cron jobs that will be consistent on each node, but randomly varying from node to node.  If you were running the agent in daemon mode then the --splay option/parameter would address the same problem.


John

[iamauser]

Hi John,

Thanks for your suggestion. In our setup, puppet cron is managed by system crontab via a script and with  no-daemonize option.For now, I will divide the nodes and run them bi-hourly.  In the meantime, I will follow the example bunch on the setting up cron for puppet from here,

http://projects.puppetlabs.com/projects/1/wiki/Cron_Patterns

-------

[Matthew Burgess]

On Wed, Mar 27, 2013 at 8:39 PM, iamauser <tapas....@gmail.com> wrote:
> Almost 4-5% of the total number of nodes are not receiving catalog from the
> master on an hourly run. We have about 250 nodes. The nodes that are
> receiving this error are rather random. The hourly cron happens almost at
> the same time. Is there any configuration changes for Puppet that can be
> done to avoid this ?

In addition to what John said, you didn't mention whether you are
already running the puppet master behind Apache or Nginx. If you're
not, then you are most probably being constrained by Puppet's built-in
webserver, webrick, which is only single-threaded.

Regards,

Matt.

[iamauser]

Hi Matt,

Thanks for your reply.

On Friday, March 29, 2013 2:32:10 PM UTC-5, Matthew Burgess wrote:

In addition to what John said, you didn't mention whether you are
already running the puppet master behind Apache or Nginx.  If you're
not, then you are most probably being constrained by Puppet's built-in
webserver, webrick, which is only single-threaded.

I am not running puppetmaster behind Apache or Nginx. So far with the default.

I looked at some of the documentation. It seems mongrel is not the way to go in Puppet-3.x. I get this message while trying to start and stop puppetmaster after the installation.

http://projects.puppetlabs.com/projects/puppet/wiki/Using_Mongrel_On_Enterprise_Linux

#

The mongrel servertype is no longer built-in to Puppet. It appears as though mongrel is being used, as the number of ports is greater than one. Starting the puppetmaster service will not behave as expected until this is resolved. Only the first port has been used in the service. These settings are defined at /etc/sysconfig/puppetmaster

#

Do you have a pointer to the docs for using puppetmaster behind Apache ?

Thanks

--------------

 

Regards,

Matt.

[Matthew Burgess]

On Mon, Apr 1, 2013 at 5:24 PM, iamauser <tapas....@gmail.com> wrote:

Do you have a pointer to the docs for using puppetmaster behind Apache ?

Sure.  The official docs are at  http://docs.puppetlabs.com/guides/passenger.html.

Regards,

Matt.

[iamauser]

Thanks. Just got it configured for our cluster. Now it should be able to manage 250 nodes without a bunch of nodes complaining about timeout.

-----