Difficulty debugging crashing PuppetDB

[Cody Robertson]

Hello! How is everyone this splendid evening?

I've recently migrated to the latest Puppet and PuppetDB (using the build in database) however I'm noticing PuppetDB keeps crashing without any errors that I can find in the logs. I've ran it in the foreground using the puppetdb-foreground command however it simply exits after awhile. The only thing I can consistently do is see it crash - I can't get any useful debug information beyond that. 

Can anyone shed some light on how I should go about this? Thank you!

puppet-3.0.2-1.el6.noarch

puppet-dashboard-1.2.17-1.el6.noarch

puppetdb-1.0.5-1.el6.noarch

puppetdb-terminus-1.0.5-1.el6.noarch

puppetlabs-release-6-6.noarch

puppet-server-3.0.2-1.el6.noarch

[Matthew Burgess]

On Wed, Jan 9, 2013 at 1:37 AM, Cody Robertson <codyha...@gmail.com> wrote:
> Hello! How is everyone this splendid evening?
>
> I've recently migrated to the latest Puppet and PuppetDB (using the build in
> database) however I'm noticing PuppetDB keeps crashing without any errors
> that I can find in the logs. I've ran it in the foreground using the
> puppetdb-foreground command however it simply exits after awhile. The only
> thing I can consistently do is see it crash - I can't get any useful debug
> information beyond that.
>
> Can anyone shed some light on how I should go about this? Thank you!

What does /var/log/messages say. Just a stab in the dark, but if your
server is short of memory, then the kernel's oom killer may be
targetting the puppetdb process; that would certainly be evident in
your /var/log/messages output.

If that's not the culprit, then attaching 'strace' to the puppetdb
process might be informative (strace -p <pid>).

Regards,

Matt.

[Chris Price]

You could also try `puppetdb-foreground --debug` for some more verbose logging.  And, is there anything at all in /var/log/puppetdb?

[Cody Robertson]

There is nothing in the logs as previously noted. It simply crashed quietly.

This is the same for when I'm running it in the foreground with --debug or when it's a daemon. It simply quietly crashes.

--

013-01-09 18:00:15,841 DEBUG [command-proc-89] [bonecp.PreparedStatementHandle] SELECT timestamp FROM certname_catalogs WHERE certname='typhoon.xxx.com' ORDER BY timestamp DESC LIMIT 1

2013-01-09 18:00:16,185 DEBUG [command-proc-89] [bonecp.PreparedStatementHandle] SELECT 1 FROM catalogs WHERE hash='b9915aef874b1a291e32f1b7cbe0efa9848fb923' LIMIT 1

2013-01-09 18:00:16,185 DEBUG [command-proc-89] [bonecp.StatementHandle] UPDATE catalogs SET api_version=1, catalog_version='1357754062' WHERE hash='b9915aef874b1a291e32f1b7cbe0efa9848fb923'

2013-01-09 18:00:16,185 DEBUG [command-proc-89] [bonecp.StatementHandle] DELETE FROM certname_catalogs WHERE certname='typhoon.xxx.com'

2013-01-09 18:00:16,186 DEBUG [command-proc-89] [bonecp.PreparedStatementHandle] INSERT INTO certname_catalogs (certname,catalog,timestamp) VALUES ('typhoon.xxx.com','b9915aef874b1a291e32f1b7cbe0efa9848fb923',2013-01-09 18:00:15.815)

2013-01-09 18:00:16,186 INFO  [command-proc-89] [puppetdb.command] [7779017b-5be2-415d-afd6-264d6d4d789e] [replace catalog] typhoon.xxx.com

sh-4.1# 

--

I'll attempt to attach a strace to it however it's so remarkably verbose it's always a treat to sift through it. Is there an easy way to increase the verbosity of puppetDB perhaps?

-Cody

[Ken Barber]

Do you get a core dump? Does it seriously just silently 'stop' with no
SEGV or anything - even in the forground?

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/vADEbpPDw7IJ.
>
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.

[Cody Robertson]

I have no core dumps however I need to make sure I have it set to allow them. It literally just goes kaput - very strange. I've yet to have time to strace it yet today however I did it briefly and it was merely doing a bunch of waits.

[Ken Barber]

> I have no core dumps however I need to make sure I have it set to allow
> them.

Yeah check ulimit -a for the puppetdb user, might need ulimit -c
unlimited or some such.

> It literally just goes kaput - very strange. I've yet to have time to
> strace it yet today however I did it briefly and it was merely doing a bunch
> of waits.

This was an strace while it was running yeah?

Just to be clear ... when you say 'crash' you mean - the java process
totally exits right? Some people use the term to mean different things
and I just want to be sure :-). You don't mean something like -
"process is still running, but stops processing stuff do you?"

I guess you really want the strace when it 'crashes' or stops running.
What doesn't make sense really is that a java process either dies and
throws an exception (which should get logged) or dies and throws a
core dump ... usually with a crash related message as well. I think
somehow we don't have all the nobs on to get the right information ...
alas we need more data :-).

For it to just stop silently would mean the program itself has called
a System.exit() or some such almost as if it was a graceful shutdown,
which although not impossible seems really weird :-).

ken.

[Chris Price]

Cody,

Can you provide some details on your OS and JVM flavors / versions?  Also, you mentioned there was "nothing in the logs"--does this include the syslog?  And are there any other *files* in the /var/log/puppetdb directory?

[Cody Robertson]

I'm not sure how I missed it but it appears the OOM was killing it (how embarassing). Once I raised the memory thresholds on the VM it was fine. Even though I somehow missed it in the log the catalyst for me finding it was looking at the PuppetDB command line - in particular the "-XX:OnOutOfMemoryError=kill" switch.

Sorry for the confusion / time wasting.

-Cody

[Ken Barber]

So the log had enough information and it was just an oversight? Thats
good to hear.

We probably chose to kill the process as apposed to continually
running as OOM's in Java often leave things in an unknown state. We
could consider tanuki wrapper/java service wrapper around the process.
At least then it would restart, but you might not see there is a
problem ;-). We do this currently for ActiveMQ in PE for example.

> https://groups.google.com/d/msg/puppet-users/-/uPYvHTKzgHAJ.