PuppetDB throwing tons of exceptions

[Guillaume Blairon]

Hi,

We're using puppetdb 2.1.0 and puppet 3.6.2 on CentOS. PuppetDB has been working fine for a few weeks.

Yesterday we added a new puppet master to handle our ~400 servers. When we added the new master in the SRV records, PuppetDB began to show an erratic behaviour :

- the number of iops on /var has raised from 50/sec to 500+/sec

- the /var/lib/puppetdb/mq/localhost/scheduler is being filled with db-*.log files until filesystem gets full

- the logs are filled with this type of messages (~1 per second) :

---

2014-09-03 15:48:51,006 ERROR [c.p.p.command] [24506b0f-6fcd-4009-9761-bfe3652ee1bd] [replace facts] Retrying after attempt 5, due to: java.lang.IllegalArgumentException: No method in multimethod 'process-command!' for dispatch value: ["replace facts" 3]

java.lang.IllegalArgumentException: No method in multimethod 'process-command!' for dispatch value: ["replace facts" 3]

at clojure.lang.MultiFn.getFn(MultiFn.java:160) ~[puppetdb.jar:na]

at clojure.lang.MultiFn.invoke(MultiFn.java:231) ~[puppetdb.jar:na]

at com.puppetlabs.puppetdb.command$produce_message_handler$fn__10782.invoke(command.clj:630) ~[na:na]

at com.puppetlabs.puppetdb.command$wrap_with_discard$fn__10731$fn__10735.invoke(command.clj:537) ~[na:na]

at com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na]

at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na]

at com.puppetlabs.puppetdb.command$wrap_with_discard$fn__10731.invoke(command.clj:536) ~[na:na]

at com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__10716$fn__10717.invoke(command.clj:490) ~[na:na]

at com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na]

at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na]

at com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__10716.invoke(command.clj:489) ~[na:na]

at com.puppetlabs.puppetdb.command$wrap_with_command_parser$fn__10726.invoke(command.clj:512) [na:na]

at com.puppetlabs.puppetdb.command$wrap_with_meter$fn__10706.invoke(command.clj:450) [na:na]

at com.puppetlabs.puppetdb.command$wrap_with_thread_name$fn__10740.invoke(command.clj:552) [na:na]

at com.puppetlabs.mq$create_message_listener$reify__9909.onMessage(mq.clj:270) [na:na]

at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560) [puppetdb.jar:na]

at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) [puppetdb.jar:na]

at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) [puppetdb.jar:na]

at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) [puppetdb.jar:na]

at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) [puppetdb.jar:na]

at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058) [puppetdb.jar:na]

at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) [puppetdb.jar:na]

at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) [puppetdb.jar:na]

at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]

---

Until now, the only solution we have found is to stop puppetdb, remove /var/lib/puppetdb/mq/localhost and restart puppetdb. But it's only buying us a few hours before the filesystem gets full again.

I'm not actually sure that the problem has a relation with the new master, but that's more or less when the problem started. Do you have an idea of what's happening here ? Is there something else I could check ?

Cheers,

--

Guillaume

[Wyatt Alt]

Hi Guillaume,

Can you post the output of

rpm -qa | grep puppet

?

It looks as if your puppetdb-terminus version could be out of sync with that of puppetdb.

Wyatt

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/85fb831e-4bf4-4d3d-acd3-a9d8892839e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Guillaume Blairon]

Yes, I was actually running puppetdb-terminus 2.2.0 against puppetdb 2.1.0. I just upgraded puppetdb and now it's working like a charm.

Thanks for the hint !