PUPPETDB and pgpool II

[Robert Bachmann]

I am trying to set up puppet in an HA environment using pgpool to connect to the postgres DB and I get this kind of thing in the pgpool log any ideas?

2013-07-12 11:56:45 LOG:   pid 12275: DB node id: 1 backend pid: 732 statement:  DISCARD ALL
2013-07-12 11:57:47 LOG:   pid 12222: connection received: host=xlabsvcdb2.hostname.com port=49722
2013-07-12 11:57:47 LOG:   pid 12214: connection received: host=xlabsvcdb2.hostname.com port=49723
2013-07-12 11:57:47 LOG:   pid 12275: connection received: host=xlabsvcdb2.hostname.com port=49726
2013-07-12 11:57:47 LOG:   pid 12213: connection received: host=xlabsvcdb2.hostname.com port=49727
2013-07-12 11:57:47 LOG:   pid 12216: connection received: host=xlabsvcdb2.hostname.com port=49730
2013-07-12 11:57:47 LOG:   pid 12212: connection received: host=xlabsvcdb2.hostname.com port=49731
2013-07-12 11:57:47 LOG:   pid 12209: connection received: host=xlabsvcdb2.hostname.com port=49734
2013-07-12 11:57:47 LOG:   pid 12224: connection received: host=xlabsvcdb2.hostname.com port=49737
2013-07-12 11:57:47 LOG:   pid 12207: connection received: host=xlabsvcdb2.hostname.com port=49738
2013-07-12 11:57:47 LOG:   pid 12223: connection received: host=xlabsvcdb2.hostname.com port=49741
2013-07-12 11:57:47 LOG:   pid 12205: connection received: host=xlabsvcdb2.hostname.com port=49742
2013-07-12 11:57:47 LOG:   pid 12218: connection received: host=xlabsvcdb2.hostname.com port=49745
2013-07-12 11:57:47 ERROR: pid 12193: Child process 12222 was terminated by segmentation fault
2013-07-12 11:57:47 LOG:   pid 12214: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited

and this in teh puppetdb log

2013-07-12 11:57:47,737 ERROR [main] [puppetlabs.utils] Uncaught exception
org.postgresql.util.PSQLException: This connection has been closed.
        at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:714)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.getAutoCommit(AbstractJdbc2Connection.java:680)
        at com.jolbox.bonecp.ConnectionHandle.getAutoCommit(ConnectionHandle.java:587)
        at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:200)
        at com.puppetlabs.puppetdb.scf.migrate$migrate_BANG_.invoke(migrate.clj:379)
        at com.puppetlabs.puppetdb.cli.services$_main$fn__10279.invoke(services.clj:396)
        at clojure.java.jdbc.internal$with_connection_STAR_.invoke(internal.clj:186)
        at com.puppetlabs.puppetdb.cli.services$_main.doInvoke(services.clj:394)
        at clojure.lang.RestFn.invoke(RestFn.java:421)
        at clojure.lang.Var.invoke(Var.java:419)
        at clojure.lang.AFn.applyToHelper(AFn.java:163)
        at clojure.lang.Var.applyTo(Var.java:532)
        at clojure.core$apply.invoke(core.clj:601)
        at com.puppetlabs.puppetdb.core$_main.doInvoke(core.clj:79)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at com.puppetlabs.puppetdb.core.main(Unknown Source)
2013-07-12 11:57:47,759 INFO  [Thread-4] [cli.services] Shutdown request received; puppetdb exiting.

[Ken Barber]

Looks like a fault with pgpool specifically. The PuppetDB error is
just an effect of a forced disconnection. Perhaps you need to reach
out to the pgpool/postgresql mailing lists for advice?

I can certainly help with PuppetDB related items, but if pgpool is
segfaulting I'm pretty sure its a bug on their end. There is a small
chance this has something to do with the JDBC driver sending a command
pgpool didn't 'grok' hence the fault, but even that we don't maintain
(although fwiw we're using version 9.0-801.jdbc4 if that helps).

Have you been able to get pgpool working fine using other PostgreSQL
clients yet?

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To post to this group, send email to puppet...@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Jared Curtis]

If you're using pgpool from the EPEL repository (or version 3.2.0) and you've enabled the query cache feature then that's the probably the problem you're seeing. Upgrade to a newer release or disable the cache.

[Robert Bachmann]

Well from the command line I can issue SQL against the puppetdb database (via pgpool) and it returns fine. I will try the  caching setting in the next comment first and see if the problem persists , thanks for the reply!!

[Robert Bachmann]

ok so I got rid of caching and now in the pgpool.log I see the snippet below. Note that I can use pgpool to quesry the puppetdb to get data

2013-07-17 13:01:28 ERROR: pid 1193: Bind: cannot get parse message "S_1"
2013-07-17 13:01:28 LOG:   pid 1193: do_child: exits with status 1 due to error
2013-07-17 13:01:28 LOG:   pid 1178: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited
2013-07-17 13:01:28 LOG:   pid 1192: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited
2013-07-17 13:01:28 LOG:   pid 1191: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited
2013-07-17 13:01:28 LOG:   pid 1189: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited
2013-07-17 13:01:28 LOG:   pid 1187: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited
2013-07-17 13:01:28 LOG:   pid 1185: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited
2013-07-17 13:01:28 LOG:   pid 1181: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited
2013-07-17 13:01:28 LOG:   pid 1179: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited
2013-07-17 13:01:28 LOG:   pid 1183: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited
2013-07-17 13:01:28 LOG:   pid 1190: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited
2013-07-17 13:01:28 LOG:   pid 1188: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited

[Jared Curtis]

This error means the client is disconnecting without sending an "I'm Disconnecting" packet.

If the client is the puppet master then you probably should file a bug report.