Error 500

103 views
Skip to first unread message

Jonathan Gazeley

unread,
Oct 2, 2014, 9:02:58 AM10/2/14
to puppet...@googlegroups.com
For quite a long time now, we've intermittently been receiving Error 500
when doing puppet runs. There's nothing bad in the puppetmaster log, but
the Apache log on the puppetmaster shows a segfault (see below). Any
ideas for debugging/fixing this?

Thanks,
Jonathan


[jg4461@radius03 ~]$ sudo puppet agent -t
Notice: Ignoring --listen on onetime run
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Loading facts
Warning: The package type's allow_virtual parameter will be changing its
default value from false to true in a future release. If you do not want
to allow virtual packages, please explicitly set allow_virtual to false.
(at /usr/lib/ruby/site_ruby/1.8/puppet/type/package.rb:430:in `default')
Error: Could not retrieve catalog from remote server: Error 500 on
SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
[no address given] and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.2.15 (CentOS) Server at puppet Port 8140</address>
</body></html>

Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run





[Thu Oct 02 12:42:04 2014] [error] [client 137.222.7.114] Premature end
of script headers: radius03.nomadic-core.bris.ac.uk
[ pid=7969 thr=140310963910624 file=ext/apache2/Hooks.cpp:841
time=2014-10-02 12:42:04.312 ]: The backend application (process 16160)
did not send a valid HTTP response; instead, it sent nothing at all. It
is possible that it has crashed; please check whether there are crashing
bugs in this application.
/usr/lib/ruby/1.8/pathname.rb:286: [BUG] Segmentation fault
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

Brian Morris

unread,
Oct 3, 2014, 8:43:19 AM10/3/14
to puppet...@googlegroups.com
This is bombing on your facters. I would try commenting out successively more facters, from the bottom of each facter file, until you locate the source of the error.

Jonathan Gazeley

unread,
Oct 3, 2014, 9:43:14 AM10/3/14
to puppet...@googlegroups.com
On 03/10/14 13:43, Brian Morris wrote:
> This is bombing on your facters. I would try commenting out successively more facters, from the bottom of each facter file, until you locate the source of the error.
>

OK. How do you tell this? I couldn't really make head or tail of the
error message.

Thanks,
Jonathan

Brian Morris

unread,
Oct 3, 2014, 10:06:09 AM10/3/14
to puppet...@googlegroups.com
I was wrong, Jonathan. That error is actually somewhere in a manifest. You can troubleshoot it the same way, though. Just start commenting out sections of code in the order that your manifests are invoked in, until you find the culprit.

jcbollinger

unread,
Oct 6, 2014, 9:18:46 AM10/6/14
to puppet...@googlegroups.com


On Friday, October 3, 2014 9:06:09 AM UTC-5, Brian Morris wrote:
I was wrong, Jonathan. That error is actually somewhere in a manifest. You can troubleshoot it the same way, though. Just start commenting out sections of code in the order that your manifests are invoked in, until you find the culprit.


The error appears to reflect a bug in your Ruby implementation.  No segmentation fault can be directly attributed to your manifests.  On the other hand, it is likely that the segfault arises from some form of unexpected input conveyed to Ruby by Puppet at the behest of one of your manifests, so you probably do have a contributory issue in one of your manifests and/or in a fact implementation.  (The error does not arise from loading or evaluating a fact, but it may be related to a bad fact value.)

Since the error is intermittent, I would start by trying to characterize the situations under which it occurs.  For example,
  • Is the error specific to a subset of you client machines (or even to just one)?
  • Is there a specific time of day window in which the error usually occurs?
  • If you are load balancing across multiple masters, does the error always occur at the same master?
It would be very much to your advantage in troubleshooting this if you can figure out conditions under which you can reliably produce the error.  For one thing, that in itself might make the nature of the problem clear.  Even if the problem remains mysterious at that point, being able to reproduce it will greatly facilitate finding the problem and verifying your fix.


John

Reply all
Reply to author
Forward
0 new messages