puppetdb 4.x filling up /opt/puppetlabs/server/data/puppetdb/mq

[Ryan Anderson]

I'm using puppetdb-4.0.0-1.el7 (open source) and over time the ActiveMQ part of it has been saving a bunch of logs under /opt/puppetlabs/server/data/puppetdb/mq, and there does not appear to be any built-in log rotation, at least in the version I am using. After using this puppetdb a couple months, this is the sizing I observe:

• 6.5GB = /opt/puppetlabs/server/data/puppetdb/mq/localhost/scheduler
  
• 2GB = /opt/puppetlabs/server/data/puppetdb/mq/discard
  

The only solution I found online is this: https://docs.puppet.com/puppetdb/4.2/trouble_kahadb_corruption.html. The workaround there is to rename the mq directory and restart puppetdb, but it is intended as a troubleshoot, and I am seeking to save disk space and avoid regular puppetdb restarts.

What files are safe to delete? Is this still a problem if I upgrade to puppetdb 4.2?

[Wyatt Alt]

Hey Ryan,

At first glance this sounds like
https://tickets.puppetlabs.com/browse/PDB-2390, but that was fixed in
4.0.0 so it shouldn't be the issue. The fact that you have 2 gigs under
discarded means you are sending a significant number of commands that
PuppetDB is unable to accept. There are numerous reasons this can happen
-- you'll probably find more info in your logs. Given that you have so
many discards, it could be that the bloat in your scheduler directory is
actually legitimate (malformed commands get retried, and while being
retried they go to scheduler.

My suggestion would be to look at your PuppetDB logs and/or the files
under discard (these will have errors in addition to the actual message)
to figure out the problem. Post back with the errors if need be and we
can trouble shoot from there.

Wyatt

On 9/21/16 5:28 AM, Ryan Anderson wrote:
> I'm using puppetdb-4.0.0-1.el7 (open source) and over time the
> ActiveMQ part of it has been saving a bunch of logs
> under /opt/puppetlabs/server/data/puppetdb/mq, and there does not
> appear to be any built-in log rotation, at least in the version I am
> using. After using this puppetdb a couple months, this is the sizing I
> observe:
>

> * 6.5GB = /opt/puppetlabs/server/data/puppetdb/mq/localhost/scheduler
> * 2GB = /opt/puppetlabs/server/data/puppetdb/mq/discard

>
>
> The only solution I found online is
> this: https://docs.puppet.com/puppetdb/4.2/trouble_kahadb_corruption.html.
> The workaround there is to rename the mq directory and restart
> puppetdb, but it is intended as a troubleshoot, and I am seeking to
> save disk space and avoid regular puppetdb restarts.
>
> What files are safe to delete? Is this still a problem if I upgrade to
> puppetdb 4.2?

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/ec2b2314-cd8f-41c4-a6d6-16936ec157c9%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/ec2b2314-cd8f-41c4-a6d6-16936ec157c9%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Ryan Anderson]

It turns out this was an ID10T error. One of my puppet masters inadvertently got upgraded, including the puppetserver and puppetdb-termini RPMs. The problem went away when I reverted the puppet RPMs back to their previous release. From what I could tell, the newer puppetserver/puppetdb-termini was sending data to the puppetdb, who did not appreciate the format. Classic error of using two versions of softwares not intended to be used with each other.