Puppet Agent's webrick crashes every 30sec.

[Peter Bauer]

hi,

i get the following output every 30sec. in the /var/log/puppet/http.log

[2014-03-28 10:30:55] ERROR OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0 state=SSLv2/v3 read client hello A

        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `accept'

        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `listen'

        /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'

        /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'

        /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'

        /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'

        /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'

        /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'

        /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'

        /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'

        /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'

        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'

        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize'

        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'

        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'

        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `synchronize'

        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'

        /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'

        /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'

        /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'

        /usr/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:359:in `main'

        /usr/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:314:in `run_command'

        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'

        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'

        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'

        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'

        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'

        /usr/sbin/puppetd:4

Its obvious that there is a problem with SSL, but since the Agent runs and everything else works fine, i have no idea what the problem is.

It seems that the Puppet Agent internally uses Webrick to receive the "puppet kick" HTTP requests, but there is some problem with Webrick and SSL though the puppet kick functionality works fine from the command line and from the Foreman.

thx for any hints,

Peter

[jcbollinger]

On Friday, March 28, 2014 5:45:15 AM UTC-5, Peter Bauer wrote:

hi,

i get the following output every 30sec. in the /var/log/puppet/http.log

[2014-03-28 10:30:55] ERROR OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0 state=SSLv2/v3 read client hello A

        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `accept'

        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `listen'

[...]

        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'

        /usr/sbin/puppetd:4

Its obvious that there is a problem with SSL, but since the Agent runs and everything else works fine, i have no idea what the problem is.

It seems that the Puppet Agent internally uses Webrick to receive the "puppet kick" HTTP requests, but there is some problem with Webrick and SSL though the puppet kick functionality works fine from the command line and from the Foreman.

The docs for SSL_accept() do not describe a return code 5, and the Ruby OpenSSL module seems completely lacking in meaningful documentation.  Nevertheless, the stack trace appears to show Puppet rejecting an SSL connection.  If it does this every 30 seconds, then that suggests you have something trying every 30 seconds to connect (and failing).  That could be a port scanner (either authorized or not), or perhaps some misconfigured service elsewhere on your network.  Since puppet kick works for you, I am inclined to say that the offender is not your puppetmaster.

Monitor traffic through your network interface (e.g. via wireshark) to get more information about what machine is making the failed connection attempts (and even to debug the SSL handshake, if you have the time, expertise, and desire to do so).  Alternatively, it is possible that puppet itself would provide more information if you run it with debug logging enabled.

And for the record, I suspect it's a mischaracterization to say "webrick crashes".  It looks to me like a simple rejection of a connection attempt, following which the system continues normally.  Dumping a stack trace into the log is not a particularly good way to communicate that, though.


John

[Peter Bauer]

thank you very much John! I will investigate this further with Wireshark.

[Peter Bauer]

For the records: it was Monit checking the port every 30sec., totally forgot about that one. Need to find a different way to check it.

g,

Peter

[Peter Bauer]

Simple solution: use Monits TCPSSL port check typ instead of TCP.