postgres plugin puppetdb

[Antidot SAS]

Hi everyone,

We are using collectd collecting system information, I am trying to setup the postgres plugin for puppetdb (http://collectd.org/wiki/index.php/Plugin:PostgreSQL), what could be the relevant value to get? I am trying to see what is going on because lately sometime I get some "Sun Aug 05 10:39:12 +0200 2012 Puppet (err): Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace catalog' command for XXXXXXXXXX to PuppetDB at puppetmaster.fqdn:8081: [500 Server Error] "

I am using puppet:

ii  puppet                            2.7.18-1puppetlabs1          Centralized configuration management - agent startup and compatibility scripts

ii  puppet-common               2.7.18-1puppetlabs1          Centralized configuration management

ii  puppet-dashboard            1.2.10-1puppetlabs2          Dashboard for Puppet

ii  puppetdb                         0.9.2-1puppetlabs2           PuppetDB Centralized Storage.

ii  puppetdb-terminus           0.9.2-1puppetlabs2           Connect Puppet to PuppetDB by setting up a terminus for PuppetDB.

ii  puppetmaster                  2.7.18-1puppetlabs1          Centralized configuration management - master startup and compatibility scripts

ii  puppetmaster-common     2.7.18-1puppetlabs1          Puppet master common scripts

On debian squeeze for handling 120 server and the puppetmaster is a 1950 with 4go of memory and it doesn't seems to be under heavy load.

Regards,

JM

[Chris Price]

Hello A_SAAS,

• Do you see any messages in the file "/var/log/puppetdb/puppetdb.log"?
  
• If not, can you try editing the file "/etc/puppetdb/log4j.properties" and replace "INFO" with "DEBUG", and then see if you get any messages in the aforementioned log file?
• What version of java are you using?  We've encountered some SSL-related problems between very early versions of the Oracle JDK 7 and puppetdb.

Thanks!

Chris

[Antidot SAS]

Here is the puppetdb.log:

2012-08-05 10:39:12,431 WARN  [qtp2101818118-104743] [server.AbstractHttpConnection] /commands

java.lang.OutOfMemoryError: Java heap space

        at java.util.Arrays.copyOf(Arrays.java:2894)

        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117)

        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:589)

        at java.lang.StringBuffer.append(StringBuffer.java:337)

        at java.net.URLDecoder.decode(URLDecoder.java:199)

        at ring.util.codec$form_decode_str.doInvoke(codec.clj:97)

        at clojure.lang.RestFn.invoke(RestFn.java:423)

        at ring.util.codec$form_decode$fn__5979.invoke(codec.clj:110)

        at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:60)

        at clojure.core.protocols$fn__5565.invoke(protocols.clj:30)

        at clojure.core.protocols$fn__5543$G__5538__5552.invoke(protocols.clj:11)

        at clojure.core$reduce.invoke(core.clj:5995)

        at ring.util.codec$form_decode.doInvoke(codec.clj:113)

        at clojure.lang.RestFn.invoke(RestFn.java:423)

        at ring.middleware.params$parse_params.invoke(params.clj:6)

        at ring.middleware.params$assoc_form_params.invoke(params.clj:29)

        at ring.middleware.params$wrap_params$fn__5994.invoke(params.clj:51)

        at com.puppetlabs.middleware$wrap_with_authorization$fn__7935.invoke(middleware.clj:20)

        at com.puppetlabs.middleware$wrap_with_certificate_cn$fn__7939.invoke(middleware.clj:35)

        at com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__7945$fn__7946.invoke(middleware.clj:60)

        at com.puppetlabs.middleware.proxy$java.lang.Object$Callable$f8c5758f.call(Unknown Source)

        at com.yammer.metrics.core.Timer.time(Timer.java:91)

        at com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__7945.invoke(middleware.clj:58)

        at com.puppetlabs.middleware$wrap_with_globals$fn__7942.invoke(middleware.clj:43)

        at ring.adapter.jetty$proxy_handler$fn__2028.invoke(jetty.clj:18)

        at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$0.handle(Unknown Source)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)

        at org.eclipse.jetty.server.Server.handle(Server.java:349)

        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)

        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894)

        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948)

        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)

Regarding the java version:

ii  openjdk-6-jre-headless                                 6b18-1.8.13-0+squeeze2       OpenJDK Java runtime, using Hotspot JIT (headless)

ii  openjdk-6-jre-lib                                      6b18-1.8.13-0+squeeze2       OpenJDK Java runtime (architecture independent libraries)

Does it help?

Regards,

JM

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/JHdLQ5zvobAJ.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Antidot SAS]

In fact I am starting to have more and more errors regarding java exception:

2012-08-07 16:39:23,864 ERROR [command-proc-77] [puppetdb.command] Fatal error parsing command java.lang.OutOfMemoryError: Java heap space

2012-08-07 16:39:26,628 WARN  [command-proc-77] [listener.DefaultMessageListenerContainer] Execution of JMS message listener failed, and no ErrorHandler has been set. java.lang.OutOfMemoryError: Java heap space

Regards,

JM

[Antidot SAS]

If you wanna see the memory of the puppet server here it is.

Regards,

JM

[Chris Price]

JM,

Hmm... so, those versions of Java seem fine--those are probably the ones we've done the most testing with.

So, the error message that you sent indicates that the JVM is running out of RAM.  This could possibly indicate that you are still using the embedded database instead of postgres--the embedded database uses a lot of RAM.  Can we see what your database.ini file looks like?  In fact, perhaps we can take a look at all of the files in your /etc/puppetdb/conf.d directory?

You could also connect directly to your postgres database using psql or pgadmin, and check to see if the puppetdb tables exist and contain data.

The only other possibility that I can think of would be if one or more of your nodes has an extremely large catalog, and that the puppetdb JVM instance doesn't have enough memory to process the catalog.  If that's the case, we can try increasing your JVM max heap space in /etc/default/puppetdb... but we would be really interested in collecting some data from you about what that catalog looks like (for our own testing and debugging purposes) if it turns out that that is the culprit. 

Thanks!

Chris

JM

Here is the puppetdb.log:

To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.

[Antidot SAS]

Hi,

The configuration files are attached.

Here is the postgres login:

[root@puppetmaster]:/data/local/postgresql/dumps #  psql -h localhost puppetdb puppetdb

psql (8.4.12)

SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)

Type "help" for help.

puppetdb=> \l

                              List of databases

   Name    |  Owner   | Encoding  | Collation | Ctype |   Access privileges

-----------+----------+-----------+-----------+-------+-----------------------

 postgres  | postgres | SQL_ASCII | C         | C     |

 puppetdb  | puppetdb | SQL_ASCII | C         | C     |

 template0 | postgres | SQL_ASCII | C         | C     | =c/postgres

                                                      : postgres=CTc/postgres

 template1 | postgres | SQL_ASCII | C         | C     | =c/postgres

                                                      : postgres=CTc/postgres

(4 rows)

puppetdb=> \d

                  List of relations

 Schema |          Name           | Type  |  Owner

--------+-------------------------+-------+----------

 public | catalog_resources       | table | puppetdb

 public | catalogs                | table | puppetdb

 public | certname_catalogs       | table | puppetdb

 public | certname_facts          | table | puppetdb

 public | certname_facts_metadata | table | puppetdb

 public | certnames               | table | puppetdb

 public | classes                 | table | puppetdb

 public | edges                   | table | puppetdb

 public | resource_params         | table | puppetdb

 public | schema_migrations       | table | puppetdb

 public | tags                    | table | puppetdb

And by the way I changed the JVM max heap at 256M and since then no issue. Regarding the postgres dump file, I'll be willing to give it to you so contact me in private to get the URL or let me know where I can put it as long as nobody could access it publicly.

Regards,

JM

[Chris Price]

Thanks JM.  Your config files look good and the existence of all of those tables in your puppetdb database certainly makes it look as though puppetdb is communicating with postgres properly.  Since increasing the heap size seems to have gotten you past the issues for now, my next guess is that it is some combination of catalog size and number of concurrent agent requests, and perhaps number of cores on your machine.  We would love to investigate further, so if you are willing to share a postgres dump with us that would be fantastic!

I'll e-mail you off-list to follow up on that.  I'd also be interested in seeing the output of 'cat /proc/cpuinfo'.

Thanks!

Chris