Random Internal Server Error after upgrading from 2.7.19 to 3.3.1
376 views
Skip to first unread message
Louise Baker
Oct 24, 2013, 1:54:28 AM10/24/13
to puppet...@googlegroups.com
Hello,
I have a rhel 6 puppet master with the following packages installed:
facter.x86_64 1:1.7.3-1.el6
hiera.noarch 1.2.1-1.el6
puppet.noarch 3.3.1-1.el6
puppet-server.noarch 3.3.1-1.el6
ruby.x86_64 1.8.7.352-12.el6_4
I have recently upgraded the puppet master from 2.7.19 to 3.3.1, downloaded from the puppetlabs yum repo.
I am now randomly seeing the following errors:
1. On the node I get:
…
Debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; using pson
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
root@localhost and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.2.15 (Red Hat) Server at <pmaster> Port 8140</address>
</body></html>
/opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:185:in `is_http_200?'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in `find'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in `find'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:243:in `retrieve_new_catalog'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:351:in `thinmark'
/opt/csw/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:350:in `thinmark'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:242:in `retrieve_new_catalog'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:67:in `retrieve_catalog'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:107:in `prepare_and_retrieve_catalog'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:159:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:20:in `lock'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'
/opt/csw/lib/ruby/1.8/sync.rb:230:in `synchronize'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:119:in `with_client'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:42:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:84:in `run_in_fork'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `call'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `controlled_run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:353:in `onetime'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:327:in `run_command'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:456:in `plugin_hook'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:504:in `exit_on_fail'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:132:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:86:in `execute'
/opt/csw/bin/puppet:4
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
2. In /var/log/messages I see errors such as:
Oct 24 14:01:57 <pmaster> puppet-master[13114]: Compiled catalog for <node> in environment production in 33.30 seconds
Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests is deprecated and will be removed in a future version. See http://links.puppetlabs.com/deprecate_yaml_on_network
Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:252:in `response_formatter_for')
Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests is deprecated and will be removed in a future version. See http://links.puppetlabs.com/deprecate_yaml_on_network
Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:65:in `request_format')
3. And in the apache error log I get errors such as:
[Thu Oct 24 14:04:50 2013] [error] [client xx.xx.xx.xx] Premature end of script headers: <node1>
[ pid=8205 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 time=2013-10-24 14:04:50.263 ]: The backend application (process 13114) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing bugs in this application.
/usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] rb_gc_mark(): unknown data type 0x20(0x495a580) non object
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
[Thu Oct 24 14:05:58 2013] [error] [client xx.xx.xx.xx] Premature end of script headers: <node2>
[ pid=8269 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 time=2013-10-24 14:05:58.762 ]: The backend application (process 11392) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing bugs in this application.
/usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] Segmentation fault
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
---
The nodes are a mix of rhel5 and 6 and solaris 10 servers, mostly running 2.7.19. I have upgraded some rhel6 nodes to 3.3.1, and some solaris 10 servers to 3.2.4 (as that is the latest csw package available).
The vast majority of the errors occur with the solaris servers running 2.7.19, but I have had a 3.2.4 fail consistently with the same error. Annoyingly I have had the error on a rhel server once, so I can’t say for certain the Solaris servers are the problem :/
I have searched for similar errors but cannot find anything exact. Any help would be greatly appreciated.
Many thanks,I have a rhel 6 puppet master with the following packages installed:
facter.x86_64 1:1.7.3-1.el6
hiera.noarch 1.2.1-1.el6
puppet.noarch 3.3.1-1.el6
puppet-server.noarch 3.3.1-1.el6
ruby.x86_64 1.8.7.352-12.el6_4
I have recently upgraded the puppet master from 2.7.19 to 3.3.1, downloaded from the puppetlabs yum repo.
I am now randomly seeing the following errors:
1. On the node I get:
…
Debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; using pson
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
root@localhost and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.2.15 (Red Hat) Server at <pmaster> Port 8140</address>
</body></html>
/opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:185:in `is_http_200?'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in `find'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in `find'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:243:in `retrieve_new_catalog'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:351:in `thinmark'
/opt/csw/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:350:in `thinmark'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:242:in `retrieve_new_catalog'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:67:in `retrieve_catalog'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:107:in `prepare_and_retrieve_catalog'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:159:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:20:in `lock'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'
/opt/csw/lib/ruby/1.8/sync.rb:230:in `synchronize'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:119:in `with_client'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:42:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:84:in `run_in_fork'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `call'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `controlled_run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:353:in `onetime'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:327:in `run_command'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:456:in `plugin_hook'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:504:in `exit_on_fail'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:132:in `run'
/opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:86:in `execute'
/opt/csw/bin/puppet:4
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
2. In /var/log/messages I see errors such as:
Oct 24 14:01:57 <pmaster> puppet-master[13114]: Compiled catalog for <node> in environment production in 33.30 seconds
Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests is deprecated and will be removed in a future version. See http://links.puppetlabs.com/deprecate_yaml_on_network
Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:252:in `response_formatter_for')
Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests is deprecated and will be removed in a future version. See http://links.puppetlabs.com/deprecate_yaml_on_network
Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:65:in `request_format')
3. And in the apache error log I get errors such as:
[Thu Oct 24 14:04:50 2013] [error] [client xx.xx.xx.xx] Premature end of script headers: <node1>
[ pid=8205 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 time=2013-10-24 14:04:50.263 ]: The backend application (process 13114) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing bugs in this application.
/usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] rb_gc_mark(): unknown data type 0x20(0x495a580) non object
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
[Thu Oct 24 14:05:58 2013] [error] [client xx.xx.xx.xx] Premature end of script headers: <node2>
[ pid=8269 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 time=2013-10-24 14:05:58.762 ]: The backend application (process 11392) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing bugs in this application.
/usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] Segmentation fault
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
---
The nodes are a mix of rhel5 and 6 and solaris 10 servers, mostly running 2.7.19. I have upgraded some rhel6 nodes to 3.3.1, and some solaris 10 servers to 3.2.4 (as that is the latest csw package available).
The vast majority of the errors occur with the solaris servers running 2.7.19, but I have had a 3.2.4 fail consistently with the same error. Annoyingly I have had the error on a rhel server once, so I can’t say for certain the Solaris servers are the problem :/
I have searched for similar errors but cannot find anything exact. Any help would be greatly appreciated.
Laurent Domb
Nov 28, 2013, 11:27:20 PM11/28/13
to puppet...@googlegroups.com
I am running into the exact same issue with 3.3.1 Did you find a solution for it?
james.e...@fasthosts.com
Nov 29, 2013, 7:39:00 AM11/29/13
to puppet...@googlegroups.com
Hi,
For reference, I've just upgraded my puppet masters from 2.7.22 to 3.3.2 and haven't seen any errors of this kind.
I presume you are running with passenger? I am too. CentOS EL6 masters.
Maybe there is a change between 3.3.1 and 3.3.2 that will resolve this for you both.
I have seen one nasty other error though.
If you aren't setting the vardir explicitly in all clients puppet.conf, I'd suggest doing so before upgrading.
After my upgrade, I had to manually intervene on almost all client nodes because puppet was failing to run. Bug filed at http://projects.puppetlabs.com/issues/23311
J
Louise Baker
Dec 10, 2013, 5:29:43 PM12/10/13
to puppet...@googlegroups.com
Hi,
Apologies for the delay in replying. I completely missed the posts!
I am yet to find a solution. I ended up reverting to 2.7.23 until I had more time to investigate the issue. Laurent, have you had any success in the past week or so?
Thanks for the heads up James. I'm not explicitly setting vardir, but will certainly do so. The puppet master is running passenger on RHEL 6. I will try 3.3.2, however if I can't be assured the problem won't occur again when I roll it into production, I am hesitant. Because of the randomness it is really hard to replicate, and didn't show up in my test environment. Perhaps it relates to load on the puppet master???
Thanks for the heads up James. I'm not explicitly setting vardir, but will certainly do so. The puppet master is running passenger on RHEL 6. I will try 3.3.2, however if I can't be assured the problem won't occur again when I roll it into production, I am hesitant. Because of the randomness it is really hard to replicate, and didn't show up in my test environment. Perhaps it relates to load on the puppet master???
Lou
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1c43515d-a4a8-4594-a59d-bdd6b785a4ec%40googlegroups.com.--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
Laurent Domb
Dec 10, 2013, 5:47:52 PM12/10/13
to puppet...@googlegroups.com
Hi Louise,
In my case the error was triggered by a faulty manifest which contained a variable ::$fqdn instead of $::fqdn. So it might be worth looking at the manifests and check for faulty vars.
Laurent
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/yXXuVN3Bb0w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAPWqdEZVxtqSH2jmg1OkJBggcYmjCU7iL59Od0XRQFgoUAbfsA%40mail.gmail.com.
Louise Baker
Dec 10, 2013, 6:11:20 PM12/10/13
to puppet...@googlegroups.com
Thanks for the response ... will do. Just to clarify though, were you getting the internal server errors every time the agent ran for a particular node, or were they sometimes successful?
LouTo view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAL4%2BNOeEy1UgpOS2jA2%3DrO8y56o%3DsEMr1n%2BAe_89ur8J_qF3jQ%40mail.gmail.com.
Laurent Domb
Dec 10, 2013, 6:23:12 PM12/10/13
to puppet...@googlegroups.com
No the error occurred randomly. Once I found the module which seemed to cause the issue and excluded all other module from the node declaration I could reproduce the error on every single puppetrun.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAPWqdEZdBt_46NYvY5ViJAzq57PNS1JFzoEseozgqaCm_%3DG_Cg%40mail.gmail.com.
Felix Frank
Dec 10, 2013, 6:25:41 PM12/10/13
to puppet...@googlegroups.com
On 11/29/2013 01:39 PM, james.e...@fasthosts.com wrote:
>
> If you aren't setting the vardir explicitly in all clients puppet.conf,
> I'd suggest doing so before upgrading.
> After my upgrade, I had to manually intervene on almost all client nodes
> because puppet was failing to run. Bug filed
> at http://projects.puppetlabs.com/issues/23311
I had some pokes at it. Unfortunately, it got duplicated in issue 23349.
>
> If you aren't setting the vardir explicitly in all clients puppet.conf,
> I'd suggest doing so before upgrading.
> After my upgrade, I had to manually intervene on almost all client nodes
> because puppet was failing to run. Bug filed
> at http://projects.puppetlabs.com/issues/23311
Fortunately, that newer one has already been solved, so 3.4 should be good.
Thanks,
Felix
Louise Baker
Dec 10, 2013, 6:36:06 PM12/10/13
to puppet...@googlegroups.com
Thank you both for your responses :)
--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/52A7A2F5.7050106%40Alumni.TU-Berlin.de.
James Eckersall
Dec 10, 2013, 6:37:38 PM12/10/13
to puppet...@googlegroups.com
And thanks from me.
Felix, thank you for your investigation on this issue, it's really appreciated J
--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/yXXuVN3Bb0w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAPWqdEbVeEW%3D3DJVLreV4wLS0uoK1wae2rT8uJu7BSj-0SnY9w%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages