Broken puppetdb

[jdehnert]

Greetings All,

I decided to try and implement the puppetlabs-puppetdb and puppetlabs-postgres modules after I had both puppetdb and postgres installed and working on my puppet server.  This was ill advised as it broke my working setup and no amount of re-configuring seems to be able to get things working again.   

I finally pulled the manifest that "describes" my configuration, but now I cannot seem to get puppetdb working at all.  My clients are giving me these kind of errors:

    Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to execute '/pdb/cmd/v1?checksum=2b3883d84fd2c7f9899bd6280247edbccc31cf15&version=4&certname=it-lnx-01.myhost.com&command=replace_facts' on at least 1 of the following 'server_urls': https://puppetmaster.myhost.com:8081

    Warning: Not using cache on failed catalog

    Error: Could not retrieve catalog; skipping run

I have crawled through  the puppet docs to try and fix my configurations, but I can't seem to get things operating properly. 

It seemed pretty clear that I did not need to configure the SSL portion of puppetdb, but these errors seem to be flying the the face of what I read.

Do I need to configure SSL for puppetdb to get it to work?  and  What's my best chance of getting things working again.  Start from scratch, or am I just overlooking something from my configs.?

Thanks,

  jdeh...@gmail.com

  

[Trevor Vaughan]

Hi James,

The puppetdb module should take care of your DB certs. However, if it doesn't, or something goes wrong, you will need to fix the DB certs manually.

Also, if you update your Puppet certificates, the PuppetDB module will *not* notice and you'll need to erase the existing certs and run the manifests again.

Of course, the puppet server is probably tied to your DB and you'll need to disconnect the two before Puppet will run cleanly again.

Remove any mention of 'puppetdb' from puppet.conf as well as deleting your routes.yaml file (if PuppetDB is the only thing in there). Restart the puppet server and things should work again.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1f1b527d-9637-4ec4-b214-adf51555927e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

[jdehnert]

Thanks for the tips Trevor.

I did make the change on the puppet server, but my clients are still complaining:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to execute '/pdb/cmd/v1?checksum=5c49765a6d1ef8e77fa85c53f70e9f572615463e&version=4&certname=it-lnx-01.myhost.com&command=replace_facts' on at least 1 of the following 'server_urls': https://puppetmaster.myhost.com:8081

any idea how I can clean that up, or why its happening?

[Trevor Vaughan]

That still looks like your server is trying to send things to PuppetDB.

Check the puppet.conf and grep for puppetdb in your puppet and puppet server configuration settings and make sure it's been removed.

Also, fully restart your puppetserver process. An API hit won't do the job.

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/ac8e8d16-7dbd-48ab-8696-dcc70ac9330f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Zeke Dehnert]

Hmm, I grepped for puppetdb in all of the puppet config files and all the files in puppeserver/conf.d and removed any reference to puppetdb from all of them.  O also made sure that puppetdb and postgres don’t start after a reboot, then i rebooted the puppet server systemm, and my clients are still getting the following message:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to execute '/pdb/cmd/v1?checksum=f6be4a58649c171db656ecbc791fc023ac1b0208&version=4&certname=it-lnx-01.myhost.com&command=replace_facts' on at least 1 of the following 'server_urls': https://puppetmaster.myhost.com:8081

Warning: Not using cache on failed catalog

I’m totally at a loss.  Should I uninstall puppetdb, puppet-termini, and postgres?  I was thinking about doing that to help get a clean install, then later I was going to make a copy of this server, and try installing puppetdb and postgres on the copy.  The bummer of this is it was all working before I tried to switch to the “Recommended” puppet labs modules. 

From: Trevor Vaughan <tvau...@onyxpoint.com>
Reply: puppet...@googlegroups.com <puppet...@googlegroups.com>
Date: June 11, 2016 at 3:10:26 PM
To: puppet...@googlegroups.com <puppet...@googlegroups.com>
Subject:  Re: [Puppet Users] Broken puppetdb

That still looks like your server is trying to send things to PuppetDB.

Check the puppet.conf and grep for puppetdb in your puppet and puppet server configuration settings and make sure it's been removed.

Also, fully restart your puppetserver process. An API hit won't do the job.

Trevor

--

Thanks,
   James “Zeke” Dehnert

Enphase Energy  |  Senior Linux Administrator

 

(707)763.4787 x7636  // Office

GPG Fingerprint: E899 A10A 960D 4B6C 22D2 8FC2 EA5E4E81

The information contained in this message may be privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete or destroy any copy of this message!

[Matthaus Owens]

It seems like you still have some puppetdb settings in puppet.conf, like fact_terminus for example. I'd look there before uninstalling the packages. 

HTH

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/etPan.575c954b.1f0068d6.1b4d%40enphaseenergy.com.

[Zeke Dehnert]

Here is my puppet.conf.

# This file can be used to override the default puppet settings.

# See the following links for more details on what settings are available:

# - https://docs.puppetlabs.com/puppet/latest/reference/config_important_settings.html

# - https://docs.puppetlabs.com/puppet/latest/reference/config_about_settings.html

# - https://docs.puppetlabs.com/puppet/latest/reference/config_file_main.html

# - https://docs.puppetlabs.com/references/latest/configuration.html

[main]

  certname = puppetmaster.enphaseenergy.com

  dns_alt_names = puppet,puppet.enphaseenergy.com,puppetmaster,puppetmaster.enphaseenergy.com

  server = puppetmaster

  environment = production

  runinterval = 1h

  strict_variables = true

  ordering = manifest

  reports = http,store

  module_repository = https://forgeapi.puppetlabs.com

[master]

  vardir = /opt/puppetlabs/server/data/puppetserver

  logdir = /var/log/puppetlabs/puppetserver

  rundir = /var/run/puppetlabs/puppetserver

  pidfile = /var/run/puppetlabs/puppetserver/puppetserver.pid

  codedir = /etc/puppetlabs/code

  reports = store

  storeconfigs = true

  environment_timeout = unlimited

  

[agent]

  pluginsync = true

  graph = true

  graphdir = /opt/puppetlabs/puppet/cache/state/graphs

I can’t find any more puppetdb, or fact_terminus references at all.  Unless its a but obscure.  I don’t know all of them so some things could be sneaking by me.

There was a puppetdb.conf file in the /etc/puppetlabs/puppet directory, but I moved it out, restarted puppet server and I’m still getting errors:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to execute '/pdb/cmd/v1?checksum=2e834ba5e4cdeae4ab27b04da26fa8c52cfbcfca&version=4&certname=it-lnx-01.enphaseenergy.com&command=replace_facts' on at least 1 of the following 'server_urls': https://puppetdb:8081

Aaarrggg!!!  

From: Matthaus Owens <matt...@puppet.com>
Reply: puppet...@googlegroups.com <puppet...@googlegroups.com>
Date: June 11, 2016 at 5:37:04 PM
To: puppet...@googlegroups.com <puppet...@googlegroups.com>
Subject:  Re: [Puppet Users] Broken puppetdb

It seems like you still have some puppetdb settings in puppet.conf, like fact_terminus for example. I'd look there before uninstalling the packages. 

HTH

--

Thanks,
   James “Zeke” Dehnert

Enphase Energy  |  Senior Linux Administrator

 

(707)763.4787 x7636  // Office

GPG Fingerprint: E899 A10A 960D 4B6C 22D2 8FC2 EA5E4E81

[Trevor Vaughan]

Get rid of this:   storeconfigs = true

Trevor

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/etPan.575cca55.752f2e1b.1b4d%40enphaseenergy.com.

For more options, visit https://groups.google.com/d/optout.

[Wyatt Alt]

On 06/11/2016 07:35 PM, Zeke Dehnert wrote:

There was a puppetdb.conf file in the /etc/puppetlabs/puppet directory, but I moved it out, restarted puppet server and I’m still getting errors:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to execute '/pdb/cmd/v1?checksum=2e834ba5e4cdeae4ab27b04da26fa8c52cfbcfca&version=4&certname=it-lnx-01.enphaseenergy.com&command=replace_facts' on at least 1 of the following 'server_urls': https://puppetdb:8081

This is the configuration file for the puppetdb terminus, and puppet server uses it to find puppetdb. Without the file, the hostname "puppetdb" is assumed, hence the error you see. You need to configure it like this:

https://docs.puppet.com/puppetdb/latest/connect_puppet_master.html#edit-puppetdbconf

Wyatt

[Wyatt Alt]

On 06/12/2016 11:29 AM, Wyatt Alt wrote:
>
> This is the configuration file for the puppetdb terminus, and puppet
> server uses it to find puppetdb. Without the file, the hostname
> "puppetdb" is assumed, hence the error you see. You need to configure
> it like this:
>
> https://docs.puppet.com/puppetdb/latest/connect_puppet_master.html#edit-puppetdbconf

Sorry, just realized you're trying to disable puppetdb. I'd recommend
removing the storeconfigs option as Trevor says.
>
> Wyatt

[jdehnert]

We have a BINGO!.  That did the trick.  Thank You Trevor, and thanks to everyone who offered assistance. I truly appreciate it.

[jdehnert]

Thanks Wyatt.   I am just trying to disconnect my puppetserver from puppetdb, but I do want to re-connect them in the future.  Your information is very helpful.  I find it easy to get confused about what part us causing what problem when I have puppetserver, puppetdb, puppetdb terminus, postgres, puppetlabs-puppetdb, and puppet labs-postgrs all involved in one way or another.

I'd love to go to training, but it's not really an option at the moment.