PuppetDB & KahaDB db.data leak

[Ken Barber]

Hi all,

I've been looking at a potential problem, as documented here:

http://projects.puppetlabs.com/issues/19241

To do with a leak within the KahaDB persistence layer of ActiveMQ.
Specifically, there are reports of the db.data file growing unbounded:

https://issues.apache.org/jira/browse/AMQ-3956

I'm hoping to find out information from other PuppetDB users to see if
this is happening in the wild. What I'm hoping is that users can
provide me with information around the size of the files in
/var/lib/puppetdb/localhost/KahaDB. For example, here is the size on
my test machine:

root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# pwd
/var/lib/puppetdb/mq/localhost/KahaDB
root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# du -sk *
5552 db-1.log
32 db.data
32 db.redo
0 lock
root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB#

And an indication of how long its been running:

$ ps auxw | grep java
puppetdb 52606 0.3 3.6 1536236 216136 ? Sl 13:34 0:55
/usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx1g
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar
/usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d

And the version of PuppetDB:

# dpkg -l puppetdb
...
ii puppetdb 1.1.1-1puppetlab all PuppetDB
Centralized Storage.

(rpm -qi puppetdb on Redhat based machines).

I've so far had 1 suspected case of this (the directory was reported
at 17 GB), but no data to back it up - if you use PuppetDB and have
the time and inclination, I'd appreciate some outputs from your
PuppetDB hosts like the ones shown above (scrubbing private
information of course) - even if it looks fine - as it will help us
decide on the priority of this bug.

Thanks in advance!

ken.

[llowder]

/var/lib/puppetdb/mq/localhost/KahaDB$ du -sk *

7800    db-1366.log

32      db.data

32      db.redo

0       lock

$ ps auxw | grep java

puppetdb 10913  1.5  9.1 1639876 370180 ?      Sl   Jan16 622:58 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d

$ dpkg -l puppetdb

ii  puppetdb                  1.0.1-1puppetlabs1        PuppetDB Centralized Storage.

 

ken.

[Stefan Goethals]

and on my test VM

root@puppetdb01 KahaDB]# du -sk *

704 db-63.log

32 db.data

32 db.redo

0 lock

[root@puppetdb01 KahaDB]# ps auxw | grep java

puppetdb 11110  0.8 29.5 1237920 301552 ?      Sl   03:54   3:58 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d

[root@puppetdb01 KahaDB]# rpm -qi puppetdb 

Name        : puppetdb                     Relocations: (not relocatable)

Version     : 1.0.5                             Vendor: (none)

Release     : 1.el6                         Build Date: Thu 20 Dec 2012 12:17:23 AM UTC

Install Date: Thu 03 Jan 2013 02:42:03 PM UTC      Build Host: rpm-builder.puppetlabs.lan

Group       : System Environment/Daemons    Source RPM: puppetdb-1.0.5-1.el6.src.rpm

Size        : 18326834                         License: ASL 2.0

Signature   : RSA/SHA1, Thu 20 Dec 2012 09:08:52 PM UTC, Key ID 1054b7a24bd6ec30

URL         : http://github.com/puppetlabs/puppetdb

Summary     : Puppet Centralized Storage Daemon

Description :

Puppet Centralized Storage.

On Thu, Feb 14, 2013 at 12:26 PM, Stefan Goethals <ste...@zipkid.eu> wrote:

[root@kangaroo ~]$cd /var/lib/puppetdb/mq/localhost/KahaDB

[root@kangaroo KahaDB]$ls -al

total 9508

drwxr-xr-x 2 puppetdb puppetdb     4096 Feb 14 11:08 .

drwxr-xr-x 4 puppetdb puppetdb     4096 Jan 14 09:38 ..

-rw-r--r-- 1 puppetdb puppetdb 33030144 Feb 14 12:23 db-168.log

-rw-r--r-- 1 puppetdb puppetdb    32768 Feb 14 12:23 db.data

-rw-r--r-- 1 puppetdb puppetdb    28720 Feb 14 12:23 db.redo

-rw-r--r-- 1 puppetdb puppetdb        0 Jan 22 10:56 lock

[root@kangaroo KahaDB]$du -sk *

9464 db-168.log

32 db.data

32 db.redo

0 lock

[root@kangaroo KahaDB]$ps auxw | grep java

puppetdb 30154  0.3  7.7 1643468 304732 ?      Sl   Jan22 108:09 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d

[root@kangaroo KahaDB]$rpm -qi puppetdb

Name        : puppetdb                     Relocations: (not relocatable)

Version     : 1.0.5                             Vendor: (none)

Release     : 1.el6                         Build Date: Thu 20 Dec 2012 01:17:23 AM CET

Install Date: Mon 14 Jan 2013 09:37:27 AM CET      Build Host: rpm-builder.puppetlabs.lan

Group       : System Environment/Daemons    Source RPM: puppetdb-1.0.5-1.el6.src.rpm

Size        : 18326834                         License: ASL 2.0

Signature   : RSA/SHA1, Thu 20 Dec 2012 10:08:52 PM CET, Key ID 1054b7a24bd6ec30

URL         : http://github.com/puppetlabs/puppetdb

Summary     : Puppet Centralized Storage Daemon

Description :

Puppet Centralized Storage.

 

ken.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Stefan Goethals]

[root@kangaroo ~]$cd /var/lib/puppetdb/mq/localhost/KahaDB

[root@kangaroo KahaDB]$ls -al

total 9508

drwxr-xr-x 2 puppetdb puppetdb     4096 Feb 14 11:08 .

drwxr-xr-x 4 puppetdb puppetdb     4096 Jan 14 09:38 ..

-rw-r--r-- 1 puppetdb puppetdb 33030144 Feb 14 12:23 db-168.log

-rw-r--r-- 1 puppetdb puppetdb    32768 Feb 14 12:23 db.data

-rw-r--r-- 1 puppetdb puppetdb    28720 Feb 14 12:23 db.redo

-rw-r--r-- 1 puppetdb puppetdb        0 Jan 22 10:56 lock

[root@kangaroo KahaDB]$du -sk *

9464 db-168.log

32 db.data

32 db.redo

0 lock

[root@kangaroo KahaDB]$ps auxw | grep java

puppetdb 30154  0.3  7.7 1643468 304732 ?      Sl   Jan22 108:09 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d

[root@kangaroo KahaDB]$rpm -qi puppetdb

Name        : puppetdb                     Relocations: (not relocatable)

Version     : 1.0.5                             Vendor: (none)

Release     : 1.el6                         Build Date: Thu 20 Dec 2012 01:17:23 AM CET

Install Date: Mon 14 Jan 2013 09:37:27 AM CET      Build Host: rpm-builder.puppetlabs.lan

Group       : System Environment/Daemons    Source RPM: puppetdb-1.0.5-1.el6.src.rpm

Size        : 18326834                         License: ASL 2.0

Signature   : RSA/SHA1, Thu 20 Dec 2012 10:08:52 PM CET, Key ID 1054b7a24bd6ec30

URL         : http://github.com/puppetlabs/puppetdb

Summary     : Puppet Centralized Storage Daemon

Description :

Puppet Centralized Storage.

On Wed, Feb 13, 2013 at 7:34 PM, llowder <llow...@gmail.com> wrote:

 

ken.

--

[Martin Willemsma]

[root@ ~]# cd  /var/lib/puppetdb/mq/localhost/KahaDB

[root@ KahaDB]# du -sk *
32828    db-9947.log
432    db-9948.log

32    db.data
32    db.redo
0    lock

[root@ KahaDB]# ps auxw | grep java
root      5695  0.0  0.0 103232   824 pts/0    R+   19:09   0:00 grep java
puppetdb 12088  6.8  7.7 4574712 309892 ?      Sl   Feb05 916:30 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx2048m -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d
activemq 31704  0.5 40.5 5655332 1611528 ?     Sl    2012 1279:27 java -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Djavax.net.ssl.keyStorePassword=password -Djavax.net.ssl.trustStorePassword=password -Djavax.net.ssl.keyStore=/opt/activemq/conf/broker.ks -Djavax.net.ssl.trustStore=/opt/activemq/conf/broker.ts -Dcom.sun.management.jmxremote -Dorg.apache.activemq.UseDedicatedTaskRunner=false -Djava.util.logging.config.file=logging.properties -Dactivemq.conf=/opt/activemq/conf -Dactivemq.data=/opt/activemq/data -XX:+UseCompressedOops -Xms3072m -Xmx3072m -Djava.library.path=/opt/activemq/bin/linux/ -classpath /opt/activemq/bin/wrapper.jar:/opt/activemq/bin/run.jar -Dwrapper.key=IxFTP8aU4zzVoBh_ -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=31701 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp org.apache.activemq.console.Main start

[root@ KahaDB]# rpm -qi puppetdb

Name        : puppetdb                     Relocations: (not relocatable)

Version     : 1.1.1                             Vendor: (none)
Release     : 1.el6                         Build Date: Fri 01 Feb 2013 11:21:27 AM CET
Install Date: Tue 05 Feb 2013 12:20:17 PM CET      Build Host: rpm-builder.delivery.puppetlabs.net
Group       : System Environment/Daemons    Source RPM: puppetdb-1.1.1-1.el6.src.rpm
Size        : 19736313                         License: ASL 2.0
Signature   : RSA/SHA1, Mon 04 Feb 2013 07:51:33 PM CET, Key ID 1054b7a24bd6ec30

URL         : http://github.com/puppetlabs/puppetdb
Summary     : Puppet Centralized Storage Daemon
Description :
Puppet Centralized Storage.

2013/2/14 Julien Pivotto <roidel...@gmail.com>

[root@mo-puppetmaster ~]# cd /var/lib/puppetdb/mq/localhost/KahaDB
[root@mo-puppetmaster KahaDB]# pwd
/var/lib/puppetdb/mq/localhost/KahaDB
[root@mo-puppetmaster KahaDB]# du -sk *
22036    db-95.log

32    db.data
32    db.redo
0    lock

[root@mo-puppetmaster KahaDB]# ps auxw | grep java
root      9969  0.0  0.0 103300   812 pts/0    S+   14:25   0:00 grep java
puppetdb 29242  0.8 16.0 1708184 330716 ?      Sl   Feb05 112:26 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d
[root@mo-puppetmaster KahaDB]# rpm -qi puppetdb

Name        : puppetdb                     Relocations: (not relocatable)

Version     : 1.0.1                             Vendor: (none)
Release     : 1.el6                         Build Date: Tue 09 Oct 2012 02:26:45 AM CEST
Install Date: Wed 17 Oct 2012 03:30:11 PM CEST      Build Host: rpm-builder.puppetlabs.lan
Group       : System Environment/Daemons    Source RPM: puppetdb-1.0.1-1.el6.src.rpm
Size        : 18533978                         License: ASL 2.0
Signature   : RSA/SHA1, Wed 10 Oct 2012 10:42:55 PM CEST, Key ID 1054b7a24bd6ec30

URL         : http://github.com/puppetlabs/puppetdb
Summary     : Puppet Centralized Storage Daemon
Description :
Puppet Centralized Storage.

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Met vriendelijke groet, Kind Regards,

Martin Willemsma

[Chuck]

My log file is cycling at 32 MB

$ date;du -sk *

Thu Feb 14 19:35:38 UTC 2013

24288 db-12257.log

360 db.data

124 db.redo

0 lock

$ date;du -sk *

Thu Feb 14 19:37:29 UTC 2013

32816 db-12257.log

1820 db-12258.log

360 db.data

124 db.redo

0 lock

$ date;du -sk *

Thu Feb 14 19:37:47 UTC 2013

3660 db-12258.log

360 db.data

124 db.redo

0 lock

$ date;du -sk *

Thu Feb 14 19:40:00 UTC 2013

15520 db-12258.log

360 db.data

124 db.redo

0 lock

$ date;du -sk *

Thu Feb 14 19:42:41 UTC 2013

33024 db-12258.log

364 db-12259.log

360 db.data

124 db.redo

0 lock

$ ps auxw | grep puppetdb |grep -v grep

puppetdb  1486 24.6 11.7 2406524 459636 ?      Sl   Feb09 2050:43 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx1024m -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d

$ rpm -qi puppetdb

Name        : puppetdb                     Relocations: (not relocatable)

Version     : 1.1.1                             Vendor: (none)

Release     : 1.el6                         Build Date: Fri 01 Feb 2013 10:21:27 AM UTC

Install Date: Sat 09 Feb 2013 12:08:35 AM UTC      Build Host: rpm-builder.delivery.puppetlabs.net

Group       : System Environment/Daemons    Source RPM: puppetdb-1.1.1-1.el6.src.rpm

Size        : 19736313                         License: ASL 2.0

Signature   : RSA/SHA1, Mon 04 Feb 2013 06:51:33 PM UTC, Key ID 1054b7a24bd6ec30

URL         : http://github.com/puppetlabs/puppetdb

Summary     : Puppet Centralized Storage Daemon

Description :

Puppet Centralized Storage.

On Wednesday, February 13, 2013 12:00:40 PM UTC-6, Ken Barber wrote:

[Martijn]

Ken, here's my data from our (small) production environment:

root@puppet:/var/lib/puppetdb/mq/localhost/KahaDB# du -sk *
23648   db-996.log
32      db.data
36      db.redo
0       lock

root@puppet:/var/lib/puppetdb/mq/localhost/KahaDB# ps auxw | grep java
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
puppetdb  5320  0.5 16.0 1144504 163876 ?      Sl   Feb11  30:05 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d

root@puppet:/var/lib/puppetdb/mq/localhost/KahaDB# dpkg -s puppetdb | grep Version
Version: 1.1.1-1puppetlabs1

Regards, Martijn

Op woensdag 13 februari 2013 19:00:40 UTC+1 schreef Ken Barber het volgende: