puppetdb module fails to install or work with fresh node

[Robert Hafner]

I'm creating a puppetdb machine using puppetdb module. My code is simple:

> class profiles::puppetdb {

>   class { '::puppetdb':

>

>   }

> }

This, unfortunately, fails with the following error:

> Debug: Executing '/usr/local/bin/validate_postgresql_connection.sh 2 10 '/usr/bin/psql --tuples-only --quiet  -h localhost  -U puppetdb  -p 5432  --dbname puppetdb ''

> Error: /Stage[main]/Puppetdb::Server::Validate_db/Postgresql::Validate_db_connection[validate puppetdb postgres connection]/Exec[validate postgres connection for puppetdb@localhost:5432/puppetdb]/unless: Check "/usr/local/bin/validate_postgresql_connection.sh 2 10 '/usr/bin/psql --tuples-only --quiet  -h localhost  -U puppetdb  -p 5432  --dbname puppetdb '" exceeded timeout

I'm a bit at a loss as to what to do here, as there's nothing in the logs that can tell me why I'm getting this error.

Robert

[Robert Hafner]

Using the documented "database_validation" option to disable database validation also fails, but in a different way:

class profiles::puppetdb {

  class { '::puppetdb':

    database_validate => false

  }

}

Results in:

> Error: Failed to apply catalog: Could not find dependent Class[Puppetdb::Server::Validate_db] for Class[Puppetdb::Database::Postgresql] at /etc/puppet/modules/puppetdb/manifests/init.pp:133

Looking at the code in the puppetdb module I'm not even sure if the 'database_validate = false ' option ever worked as advertised.

[Robert Hafner]

Even using the “embedded” database is apparently useless, as puppet is still not able to connect to puppetdb.

In addition, puppetdb is very obviously not creating it’s firewall rules even though I haven’t disabled that feature. 

Does anyone have an example of this module actually working? It’s hard to imagine a more simple setup than the one I have right here, but this module is not working or giving me any reasons why it’s not working. PuppetDB is just failing completely, and with all the default settings.

Rob

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/VVKZhUanAlk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/afa578b8-902b-4d62-93a4-79fc3081ed6d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ken Barber]

> Even using the “embedded” database is apparently useless, as puppet is still
> not able to connect to puppetdb.
>
> In addition, puppetdb is very obviously not creating it’s firewall rules
> even though I haven’t disabled that feature.

This is interesting/surprising, but it sounds like the main pressing
issue is postgresql validation, perhaps we can focus on this to start
with, and I'll get back to this particular issue.

> Does anyone have an example of this module actually working? It’s hard to
> imagine a more simple setup than the one I have right here, but this module
> is not working or giving me any reasons why it’s not working. PuppetDB is
> just failing completely, and with all the default settings.

We've got it working for our acceptance tests for every patch, so this
is surprising as I said. Lets see if we can work out what is going on
together.

> class profiles::puppetdb {
>
> class { '::puppetdb':
>
> database_validate => false
>
> }
>
> }
>
>
> Results in:
>
>> Error: Failed to apply catalog: Could not find dependent
>> Class[Puppetdb::Server::Validate_db] for
>> Class[Puppetdb::Database::Postgresql] at
>> /etc/puppet/modules/puppetdb/manifests/init.pp:133
>
> Looking at the code in the puppetdb module I'm not even sure if the
> 'database_validate = false ' option ever worked as advertised.

Yep, it looks like a bug, we're looking into it - thanks for pointing
it out, but still - it shouldn't be required.

>> I'm creating a puppetdb machine using puppetdb module. My code is simple:
>>
>> > class profiles::puppetdb {
>> > class { '::puppetdb':
>> >
>> > }
>> > }
>>
>>
>> This, unfortunately, fails with the following error:
>>
>> > Debug: Executing '/usr/local/bin/validate_postgresql_connection.sh 2 10
>> > '/usr/bin/psql --tuples-only --quiet -h localhost -U puppetdb -p 5432
>> > --dbname puppetdb ''
>>
>> > Error:
>> > /Stage[main]/Puppetdb::Server::Validate_db/Postgresql::Validate_db_connection[validate
>> > puppetdb postgres connection]/Exec[validate postgres connection for
>> > puppetdb@localhost:5432/puppetdb]/unless: Check
>> > "/usr/local/bin/validate_postgresql_connection.sh 2 10 '/usr/bin/psql
>> > --tuples-only --quiet -h localhost -U puppetdb -p 5432 --dbname puppetdb
>> > '" exceeded timeout
>>
>>
>> I'm a bit at a loss as to what to do here, as there's nothing in the logs
>> that can tell me why I'm getting this error.

Okay, so lets start with the basics: what is the distro, arch release
version and version of the PuppetDB module? Are you obtaining the
module via forge, git or some other mechanism? I'm asking for these
details so I can try and recreate this and see what is happening. It
shouldn't bomb this hard, and I'm wary we have a bug somewhere.

Can you also send me a full debug output run for this in a gist
(gist.github.com) from start to finish? This should be the full output
of puppet agent -t --debug for example.

You say this is on a clean machine, is this vagrant perchance? If so
is there a vagrantfile you can show me so I can reproduce? Any other
puppet manifests in the mix here that you can show us to help us
reproduce it?

ken.

[Ken Barber]

FWIW: We've got a PR for this now, my colleague AJ has just raised it:
https://github.com/puppetlabs/puppetlabs-puppetdb/pull/182 you can try
this out yourself if you like, otherwise we'll be merging it once its
reviewed.

[Garrett Honeycutt]

On 5/28/15 4:17 PM, Robert Hafner wrote:
>
> Even using the “embedded” database is apparently useless, as puppet is
> still not able to connect to puppetdb.
>
> In addition, puppetdb is very obviously not creating it’s firewall rules
> even though I haven’t disabled that feature.
>
> Does anyone have an example of this module actually working? It’s hard
> to imagine a more simple setup than the one I have right here, but this
> module is not working or giving me any reasons why it’s not working.
> PuppetDB is just failing completely, and with all the default settings.
>
> Rob
>
>
>
>
>> On May 28, 2015, at 12:43 PM, Robert Hafner <ted...@tedivm.com

Hi Robert,

I put this in Hiera to extend the timeout to 30 seconds. This is often
needed when using underpowered VM's as PostgreSQL can take awhile to start.

puppetdb::master::config::puppetdb_startup_timeout: 30


Best regards,
-g

--
Garrett Honeycutt
@learnpuppet
Puppet Training with LearnPuppet.com
Mobile: +1.206.414.8658

[Robert Hafner]

I figured out the issue with the embedded database. For some reason letting the system default to ::cert for the “ssl_listen_address” had it binding to localhost instead of the actual interface it should have. Specifying “0.0.0.0” for that option made puppetdb work just fine when using the embedded database. It still does not add the firewall rules in though- I’m doing that myself.

Going back to postgres still results in the same failure. Since it’s related to the validation script I thought I would run it manually to see what happens-

> root@puppetdb01:/var/log# /usr/local/bin/validate_postgresql_connection.sh 2 10 '/usr/bin/psql --tuples-only --quiet  -h localhost  -U puppetdb  -p 5432  --dbname puppetdb '

The result is that the script hangs while waiting for the user to put a password in. 

Both machines are running Ubuntu 14.04 LTS. I am using Puppet 3.7.5 from the PuppetLabs repository. This is on a VMWare ESXI box.

I’m using a PuppetFile with librarian-puppet. Although the puppetdb machine is new, and is a “plank role” so to speak, it does include other modules as there are other services which are up and running. This is the first time I’ve used the postgres module (it’s not being used by other nodes) so there shouldn’t be any issues with conflicts. That being said, I did have to push through a change to postgres to allow it to use the puppet labs apt-module, since the puppet labs team have not managed to keep all their module dependencies in sync. You can see the pull request I made for that here- https://github.com/puppetlabs/puppetlabs-postgresql/pull/632/files

Rob

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/VVKZhUanAlk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNTm1CcN6-CD_YOrwZODjm-Lee_gdJiKxGVrUnVgV%3DA_SHQ%40mail.gmail.com.

[Ken Barber]

Okay, I think this is enough to start with, thanks Rob. I'm going to
get something working now to try and reproduce, I'm in the UK though
so I may end up going to bed before I get it going and respond,
otherwise I'll pick it up again tomorrow. I suspect its because of
Ubuntu 14.04, which we don't have automated tests for yet ... and
we've got a bug we haven't noticed. At least this wouldn't surprise
me.

You definitely want to prefer PostgreSQL over embedded btw, so this is
the correct direction to follow.

Sorry about all the hassles again mate :-).

ken.

[Ken Barber]

So Rob I've managed to do a successful install on a clean Ubuntu 14.04
box, you can see the full transcript from here:
https://gist.github.com/kbarber/837ff7e55e8940a7d1c8

What variations during the installation process do you think are here?

In regards to your other points yesterday:

> I figured out the issue with the embedded database. For some reason letting the system default to ::cert for the “ssl_listen_address” had it binding to localhost instead of the actual interface it should have. Specifying “0.0.0.0” for that option made puppetdb work just fine when using the embedded database. It still does not add the firewall rules in though- I’m doing that myself.

Its a silly default, I've raised a PR against master for it, should be
able to change it on the next major release 5.0.0:
https://github.com/puppetlabs/puppetlabs-puppetdb/pull/183

> That being said, I did have to push through a change to postgres to allow it to use the puppet labs apt-module, since the puppet labs team have not managed to keep all their module dependencies in sync. You can see the pull request I made for that here- https://github.com/puppetlabs/puppetlabs-postgresql/pull/632/files

Are you using manage_repos or something like that in the postgresql
module to pull in the PGDG upstream repos? My setup just uses the
Ubuntu 14.04 packages, since they come with PG 9.3.

> > root@puppetdb01:/var/log# /usr/local/bin/validate_postgresql_connection.sh 2 10 '/usr/bin/psql --tuples-only --quiet -h localhost -U puppetdb -p 5432 --dbname puppetdb '
>
> The result is that the script hangs while waiting for the user to put a password in.

Yeah, you've got to pass through the password in the environment
variable like the manifest does:
https://github.com/puppetlabs/puppetlabs-postgresql/blob/master/manifests/validate_db_connection.pp#L40-L43

Like so:

root@localhost:~# PGPASSWORD=puppetdb

/usr/local/bin/validate_postgresql_connection.sh 2 10 '/usr/bin/psql
--tuples-only --quiet -h localhost -U puppetdb -p 5432 --dbname
puppetdb '

1
1

root@localhost:~# echo $?
0

>> I’m using a PuppetFile with librarian-puppet.

Are you retrieving modules from git, or using forge versions like I
have? What are the particular versions of all these modules you are
using?

Also ... going back to your original error:

> Error: /Stage[main]/Puppetdb::Server::Validate_db/Postgresql::Validate_db_connection[validate puppetdb postgres connection]/Exec[validate postgres connection for puppetdb@localhost:5432/puppetdb]/unless: Check "/usr/local/bin/validate_postgresql_connection.sh 2 10 '/usr/bin/psql --tuples-only --quiet -h localhost -U puppetdb -p 5432 --dbname puppetdb '" exceeded timeout

You mention firewalling, is the port properly open? It sounds like
with your previous psql test it is, but it's odd that it's slow, which
brings me to Garrett's comment:

> I put this in Hiera to extend the timeout to 30 seconds. This is often needed when using underpowered VM's as PostgreSQL can take awhile to start.

Seem unlikely that a postgresql database, even a large one would need
30 seconds, but if you are seeing delays in starting postgresql then
it's worth a try, just seems odd for a small database, PG usually
takes less than a second to start.

ken.