Net::ReadTimeout errors

2,278 views
Skip to first unread message

Tim Dunphy

unread,
Jul 15, 2015, 3:12:23 AM7/15/15
to puppet...@googlegroups.com
Hey all,

 I'm running puppet version 3.7.5 with foreman 1.7.4. Everything was going well with this setup for quite a long time. Many months at least it ran without even so much as a hiccup!

Until recently when I started adding more nodes I've found this issue I'm having occurring every couple of days. Then as I added even more nodes it started happening every couple of hours!!

The way I've always resolved it was to just reboot the puppet host. Which wasn't so bad at first. Not that I like frequent reboots as any kind of solution to a problem. But when I was having to do it once or twice in a weeks time it wasn't so bothersome.

But now that it's occurring so often it's time to find a more permanent solution.

Without further delay, here's the error I'm getting:

#puppet agent --test

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: Net::ReadTimeout

Info: Retrieving pluginfacts

Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': Net::ReadTimeout

Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://puppet.example.com/pluginfacts: Net::ReadTimeout

Info: Retrieving plugin

Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Error 503 on SERVER: <h1>This website is under heavy load</h1><p>We're sorry, too many people are accessing this website at the same time. We're working on this problem. Please try again later.</p>

Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://puppet.example.com/plugins: Error 503 on SERVER: <h1>This website is under heavy load</h1><p>We're sorry, too many people are accessing this website at the same time. We're working on this problem. Please try again later.</p>

Info: Loading facts

Error: Could not retrieve catalog from remote server: Error 503 on SERVER: <h1>This website is under heavy load</h1><p>We're sorry, too many people are accessing this website at the same time. We're working on this problem. Please try again later.</p>

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Error: Could not send report: Error 503 on SERVER: <h1>This website is under heavy load</h1><p>We're sorry, too many people are accessing this website at the same time. We're working on this problem. Please try again later.</p>

I'm running this host on a CentOS 7 host with 4GB of ram with a single core 2.4 GhZ processor. Right now I'm managing a collection of 25 hosts with this puppet server. And as I mentioned I never really had this problem when I was managing less servers with this.

So what I'm wondering is if adding more ram to the server would be the only way of solving this problem? Other than reducing the number of hosts that I'm managing with it.

Thanks!

Tim


--
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Felix Frank

unread,
Jul 17, 2015, 8:16:12 PM7/17/15
to puppet...@googlegroups.com
Hi,

I'm not very familiar with the puppet/foreman stack, so this might be a silly question, but how is the master set up? Is it running through Apache/Passenger?

In that case, can we see the Apache vhost configuration?

Thanks,
Felix

Tim Dunphy

unread,
Jul 17, 2015, 8:38:02 PM7/17/15
to puppet...@googlegroups.com

Hi Felix,

 Thanks for getting back to me! And yes, you are correct. Puppet/Foreman is running through apache/passenger. 

And here's the vhost configurations. There's two of them, one for ssl and one for non ssl. Non ssl is first:

                                                                                                                                                                                                   # ************************************

# Vhost template in module puppetlabs-apache

# Managed by Puppet

# ************************************

<VirtualHost *:80>

  ServerName puppet.example.com

  ## Vhost docroot

  DocumentRoot "/usr/share/foreman/public"

  ## Directories, there should at least be a declaration for /usr/share/foreman/publ

  <Directory "/usr/share/foreman/public">

    Options SymLinksIfOwnerMatch

    AllowOverride None

    Require all granted

  </Directory>

  ## Load additional static includes

  ## Logging

  ErrorLog "/var/log/httpd/foreman_error.log"

  ServerSignature Off

  CustomLog "/var/log/httpd/foreman_access.log" combined

  ## Server aliases

  ServerAlias foreman

  ## Custom fragment

PassengerAppRoot /usr/share/foreman

PassengerRuby /usr/bin/ruby193-ruby

PassengerMinInstances 1

PassengerStartTimeout 600

AddDefaultCharset UTF-8

# Static public dir serving

<Directory /usr/share/foreman/public>

  <IfVersion < 2.4>

    Allow from all

  </IfVersion>

  <IfVersion >= 2.4>

    Require all granted

  </IfVersion>

</Directory>

<Directory /usr/share/foreman/public/assets>

  # Use standard http expire header for assets instead of ETag

  <IfModule mod_expires.c>

    Header unset ETag

    FileETag None

    ExpiresActive On

    ExpiresDefault "access plus 1 year"

  </IfModule>

  # Return compressed assets if they are precompiled

  <IfModule mod_rewrite.c>

    RewriteEngine on

    # Make sure the browser supports gzip encoding and file with .gz added

    # does exist on disc before we rewrite with the extension

    RewriteCond %{HTTP:Accept-Encoding} \b(x-)?gzip\b

    RewriteCond %{REQUEST_FILENAME}.gz -s

    RewriteRule ^(.+) $1.gz [L]

    # Set headers for all possible assets which are compressed

    <FilesMatch \.css\.gz$>

      ForceType text/css

      Header set Content-Encoding gzip

      SetEnv no-gzip

    </FilesMatch>

    <FilesMatch \.js\.gz$>

      ForceType text/javascript

      Header set Content-Encoding gzip

      SetEnv no-gzip

    </FilesMatch>

  </IfModule>

</Directory>

<IfVersion < 2.4>

  Include /etc/httpd/conf.d/05-foreman.d/*.conf

</IfVersion>

<IfVersion >= 2.4>

  IncludeOptional /etc/httpd/conf.d/05-foreman.d/*.conf

</IfVersion>

PassengerPreStart http://puppet.example.com

</VirtualHost>

And here's the SSL vhost:

# ************************************

# Vhost template in module puppetlabs-apache

# Managed by Puppet

# ************************************

<VirtualHost *:443>

  ServerName puppet.example.com

  ## Vhost docroot

  DocumentRoot "/usr/share/foreman/public"

  ## Directories, there should at least be a declaration for /usr/share/foreman/public

  <Directory "/usr/share/foreman/public">

    Options SymLinksIfOwnerMatch

    AllowOverride None

    Require all granted

  </Directory>

  ## Load additional static includes

  ## Logging

  ErrorLog "/var/log/httpd/foreman-ssl_error_ssl.log"

  ServerSignature Off

  CustomLog "/var/log/httpd/foreman-ssl_access_ssl.log" combined

  ## Server aliases

  ServerAlias foreman

  ## SSL directives

  SSLEngine on

  SSLCertificateFile      "/var/lib/puppet/ssl/certs/puppet.example.com.pem"

  SSLCertificateKeyFile   "/var/lib/puppet/ssl/private_keys/puppet.example.com.pem"

  SSLCertificateChainFile "/var/lib/puppet/ssl/certs/ca.pem"

  SSLCACertificatePath    "/etc/pki/tls/certs"

  SSLCACertificateFile    "/var/lib/puppet/ssl/certs/ca.pem"

  SSLVerifyClient         optional

  SSLVerifyDepth          3

  SSLOptions +StdEnvVars

  ## Custom fragment

PassengerAppRoot /usr/share/foreman

PassengerRuby /usr/bin/ruby193-ruby

PassengerMinInstances 1

PassengerStartTimeout 600

AddDefaultCharset UTF-8

# Static public dir serving

<Directory /usr/share/foreman/public>

  <IfVersion < 2.4>

    Allow from all

  </IfVersion>

  <IfVersion >= 2.4>

    Require all granted

  </IfVersion>

</Directory>

<Directory /usr/share/foreman/public/assets>

  # Use standard http expire header for assets instead of ETag

  <IfModule mod_expires.c>

    Header unset ETag

    FileETag None

    ExpiresActive On

    ExpiresDefault "access plus 1 year"

  </IfModule>

  # Return compressed assets if they are precompiled

  <IfModule mod_rewrite.c>

    RewriteEngine on

    # Make sure the browser supports gzip encoding and file with .gz added

    # does exist on disc before we rewrite with the extension

    RewriteCond %{HTTP:Accept-Encoding} \b(x-)?gzip\b

    RewriteCond %{REQUEST_FILENAME}.gz -s

    RewriteRule ^(.+) $1.gz [L]

    # Set headers for all possible assets which are compressed

    <FilesMatch \.css\.gz$>

      ForceType text/css

      Header set Content-Encoding gzip

      SetEnv no-gzip

    </FilesMatch>

    <FilesMatch \.js\.gz$>

      ForceType text/javascript

      Header set Content-Encoding gzip

      SetEnv no-gzip

    </FilesMatch>

  </IfModule>


</Directory>


<IfVersion < 2.4>

  Include /etc/httpd/conf.d/05-foreman-ssl.d/*.conf

</IfVersion>

<IfVersion >= 2.4>

  IncludeOptional /etc/httpd/conf.d/05-foreman-ssl.d/*.conf

</IfVersion>

PassengerPreStart https://puppet.example.com



</VirtualHost>

-- 

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/55A96286.1070007%40Alumni.TU-Berlin.de.

For more options, visit https://groups.google.com/d/optout.

Felix Frank

unread,
Jul 17, 2015, 8:40:59 PM7/17/15
to puppet...@googlegroups.com
On 07/17/2015 10:37 PM, Tim Dunphy wrote:
>
> Hi Felix,
>
> Thanks for getting back to me! And yes, you are correct.
> Puppet/Foreman is running through apache/passenger.
>
> And here's the vhost configurations. There's two of them, one for ssl
> and one for non ssl. Non ssl is first:
>

Hi Tim,

thanks, but those aren't the important ones I'm afraid. There should be
a vhost to bind port 8140. If there is not, please check carefully which
process is actually listening on that port. (If it's Ruby, I have bad
news and good news :-)

Best,
Felix

Tim Dunphy

unread,
Jul 18, 2015, 1:20:41 AM7/18/15
to puppet...@googlegroups.com
Hey Felix,

thanks, but those aren't the important ones I'm afraid. There should be
a vhost to bind port 8140. If there is not, please check carefully which
process is actually listening on that port. (If it's Ruby, I have bad
news and good news :-)

Whoops! OK I get you now. Here's the vhost that binds to 8140:

# ************************************

# Vhost template in module puppetlabs-apache

# Managed by Puppet

# ************************************

<VirtualHost *:8140>

  ServerName puppet

  ## Vhost docroot

  DocumentRoot "/etc/puppet/rack/public/"

  ## Directories, there should at least be a declaration for /etc/puppet/rack/public/

  <Directory "/etc/puppet/rack/public/">

    AllowOverride None

    Require all granted

    PassengerEnabled On

  </Directory>

  ## Load additional static includes

  ## Logging

  ErrorLog "/var/log/httpd/puppet_error_ssl.log"

  ServerSignature Off

  CustomLog "/var/log/httpd/puppet_access_ssl.log" combined

  ## SSL directives

  SSLEngine on

  SSLCertificateFile      "/var/lib/puppet/ssl/certs/puppet.example.com.pem"

  SSLCertificateKeyFile   "/var/lib/puppet/ssl/private_keys/puppet.example.com.pem"

  SSLCertificateChainFile "/var/lib/puppet/ssl/ca/ca_crt.pem"

  SSLCACertificatePath    "/etc/pki/tls/certs"

  SSLCACertificateFile    "/var/lib/puppet/ssl/ca/ca_crt.pem"

  SSLCARevocationFile     "/var/lib/puppet/ssl/ca/ca_crl.pem"

  SSLProtocol             ALL -SSLv2 -SSLv3

  SSLCipherSuite          EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA

  SSLHonorCipherOrder     on

  SSLVerifyClient         optional

  SSLVerifyDepth          1

  SSLOptions +StdEnvVars +ExportCertData

  ## Request header rules

  ## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader

  RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e

  RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e

  RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

  RequestHeader unset X-Forwarded-For

  ## Custom fragment

</VirtualHost>


And it looks like only apache is listening on port 8140:


[root@puppet:/etc/httpd/conf.d] #lsof -i :8140

COMMAND   PID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME

httpd    1144   root    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd    1643 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd    1643 apache   17u  IPv4 3341375      0t0  TCP puppet.example.com:8140->nfs1.example.com:35940 (ESTABLISHED)

httpd    9548 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd    9587 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd    9934 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd    9997 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   10398 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   10398 apache   17u  IPv4 3341633      0t0  TCP puppet.example.com:8140->ec2-54-174-216-240.compute-1.amazonaws.com:53957 (ESTABLISHED)

httpd   10955 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   16186 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   16188 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   16189 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   16189 apache   17u  IPv4 3341683      0t0  TCP puppet.example.com:8140->ec2-52-5-117-61.compute-1.amazonaws.com:51054 (ESTABLISHED)

httpd   17444 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   18372 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   18372 apache   17u  IPv4 3341585      0t0  TCP puppet.example.com:8140->varnish2.example.com:41184 (ESTABLISHED)

httpd   19116 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   19116 apache   17u  IPv4 3341603      0t0  TCP puppet.example.com:8140->ec2-54-173-242-112.compute-1.amazonaws.com:38536 (ESTABLISHED)

httpd   20462 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   21680 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   21680 apache   17u  IPv4 3341090      0t0  TCP puppet.example.com:8140->web1.example.com:50509 (ESTABLISHED)

httpd   23953 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   23955 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   23956 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   23956 apache   17u  IPv4 3341270      0t0  TCP puppet.example.com:8140->ec2-52-4-204-96.compute-1.amazonaws.com:48068 (ESTABLISHED)

httpd   29450 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)

httpd   29450 apache   17u  IPv4 3341677      0t0  TCP puppet.example.com:8140->ec2-52-5-152-109.compute-1.amazonaws.com:56020 (ESTABLISHED)

httpd   31405 apache    5u  IPv4   17778      0t0  TCP *:8140 (LISTEN)


 I hope that helps! And let me know if I can provide anything else that could help track down the issue.

Thanks!
Tim

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/55A96854.60500%40Alumni.TU-Berlin.de.

For more options, visit https://groups.google.com/d/optout.

Felix Frank

unread,
Jul 18, 2015, 1:59:06 AM7/18/15
to puppet...@googlegroups.com
On 07/18/2015 03:20 AM, Tim Dunphy wrote:
Hey Felix,

thanks, but those aren't the important ones I'm afraid. There should be
a vhost to bind port 8140. If there is not, please check carefully which
process is actually listening on that port. (If it's Ruby, I have bad
news and good news :-)

Whoops! OK I get you now. Here's the vhost that binds to 8140:

Yep, that's the one.

OK, so you manage that via puppetlabs-apache. Guess you will have to ultimately fix this in your manifest/data then.

The default template for the passenger vhost includes settings

PassengerHighPerformance on
and
PassengerPoolSize X

Make sure that both are present in your Apache config, and that the latter is set to a sufficiently high value. Allow Apache to fork off enough Ruby workers to handle your load. I suppose your hardware can run a few dozen processes.

HTH,
Felix

Tim Dunphy

unread,
Jul 18, 2015, 7:58:34 PM7/18/15
to puppet...@googlegroups.com
OK, so you manage that via puppetlabs-apache. Guess you will have to ultimately fix this in your manifest/data then.
The default template for the passenger vhost includes settings
PassengerHighPerformance on
and
PassengerPoolSize X
Make sure that both are present in your Apache config, and that the latter is set to a sufficiently high value. Allow Apache to fork off enough Ruby workers to handle your load. I suppose your hardware can run a few dozen processes.

Awesome Felix! Thanks for the info. I'll give it a go and follow up once I have some results.

Thanks!
Tim 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/55A9B2DC.7070102%40Alumni.TU-Berlin.de.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages