PuppetDB Replication

[tsuave]

I have puppetdb setup on our puppetmaster with a postgreSQL DB setup on two servers db1 and db2. I am trying to setup replication between db1 and db2, using rubyrep. Rubyrep can copy the data but not the schema. I tired to dump the schema of a DB after puppet has connected to use as a template to create both DBs, start the replication, and then connect the puppetdb. Unfortunately, when puppetdb connects to a db that already has a schema/tables created, I get the following:

2012-09-06 10:44:51,102 ERROR [main] [puppetlabs.utils] Uncaught exception

java.lang.NullPointerException

        at clojure.lang.Numbers.ops(Numbers.java:942)

        at clojure.lang.Numbers.gt(Numbers.java:227)

        at com.puppetlabs.puppetdb.scf.migrate$pending_migrations$fn__1296.invoke(migrate.clj:184)

        at clojure.core$filter$fn__3830.invoke(core.clj:2478)

        at clojure.lang.LazySeq.sval(LazySeq.java:42)

        at clojure.lang.LazySeq.seq(LazySeq.java:60)

        at clojure.lang.RT.seq(RT.java:466)

        at clojure.core$seq.invoke(core.clj:133)

        at clojure.core$reduce.invoke(core.clj:5994)

        at clojure.core$into.invoke(core.clj:6005)

        at com.puppetlabs.puppetdb.scf.migrate$pending_migrations.invoke(migrate.clj:184)

        at com.puppetlabs.puppetdb.scf.migrate$migrate_BANG_.invoke(migrate.clj:190)

        at com.puppetlabs.puppetdb.cli.services$_main$fn__8398.invoke(services.clj:250)

        at clojure.java.jdbc.internal$with_connection_STAR_.invoke(internal.clj:186)

        at com.puppetlabs.puppetdb.cli.services$_main.doInvoke(services.clj:249)

        at clojure.lang.RestFn.invoke(RestFn.java:421)

        at clojure.lang.Var.invoke(Var.java:405)

        at clojure.lang.AFn.applyToHelper(AFn.java:163)

        at clojure.lang.Var.applyTo(Var.java:518)

        at clojure.core$apply.invoke(core.clj:600)

        at com.puppetlabs.puppetdb.core$_main.doInvoke(core.clj:80)

        at clojure.lang.RestFn.applyTo(RestFn.java:137)

        at com.puppetlabs.puppetdb.core.main(Unknown Source)

2012-09-06 10:44:51,106 INFO  [Thread-4] [cli.services] Shutdown request received; puppetdb exiting.

Also, If we connect puppetdb to a blank db and it creates any data, the replication will not work because the data will be duplicated. There seems to be a way to do if if we connect to db1 and then db2 without any nodes checking in, it should work fine from there. We want to automate the builds and replication, so going through these motions would be hard to automate, and if a node checks in while either db is connect and data is created, it wont replicate. 

Does anyone know of a way I can create the puppetdb schema without connecting the puppetdb service to the empty database that will not cause the above exception? I would like to create puppetdb on db1 and db2, load the schema, start the replication and then connect puppetdb to a load balancer that can choose either db1 or db2 and work correctly because of replication. 

Anyone have a better idea than rubyrep?

Thanks!

[Florian Koch]

hi,

hm why do you not using the postgresql      

replication? (wal sync with versions prior to 9 and streaming replication with 9+)

regards flo

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/kr4Ogs6iskoJ.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Nick Lewis]

PuppetDB uses the schema_migrations table to determine the migrated state of the database. If the table doesn't exist, it assumes the database needs to be fully created from the beginning. But it's not properly handling the case where the table exists but has no data. However, I think the right thing for it to do in that case is explicitly fail, noting that the database is in an invalid state. So either way, this isn't going to work.

The best way to create the database would be to run PuppetDB against it. Failing that, you should probably replicate the data *before* starting PuppetDB against the new database.

 

Thanks!