No resources and catalog information with puppetdb

[Louis Coilliot]

Hello,

with my puppetdb
- I can't query any resource or catalog
- exported resources are not working


This is working :

# echo '["=", ["fact", "rubyversion"], "1.8.7"]' > queryfile
# curl -X GET http://localhost:8080/v3/nodes --data-urlencode query@queryfile 2>/dev/null | tail -7
}, {
    "name" : "el6.labolinux.fr",
    "deactivated" : null,
    "catalog_timestamp" : null,
    "facts_timestamp" : "2014-02-25T08:18:07.529Z",
    "report_timestamp" : "2014-02-25T08:18:10.018Z"
 
I can also query with success the facts, metrics, reports (reports=store,puppetdb) endpoints.

But it is not working with the resources or catalog endpoint :

curl -X GET 'http://localhost:8080/v3/resources/User'
-> []

curl -X GET 'http://localhost:8080/v3/resources/Package'
-> []

curl -X GET 'http://localhost:8080/v3/resources/File'
-> []

curl -X GET http://localhost:8080/v3/catalogs/el6.labolinux.fr
-> { "error" : "Could not find catalog for el6.labolinux.fr" }

curl -X GET http://localhost:8080/v3/nodes 2>/dev/null | grep name | grep el6.labolinux.fr
->  "name" : "el6.labolinux.fr",

In addition, this is not working from the puppet master :

# puppet node status el6.labolinux.fr
Error: Could not retrieve status for el6.labolinux.fr: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A

# puppet cert list el6.labolinux.fr
  + "el6.labolinux.fr" (SHA256) 76:00:C9:B9:0C:31:61:9C:A5:D9:B4:49:D7:17:39:76:15:9D:18:2C:E0:07:41:6B:6C:3A:4D:68:E1:BF:65:0D
 
I think that a consequence is that my exported resources don't work.
 
 
Here is my configuration.
 
On the master :

# dpkg-query -W | egrep 'puppet(master|db)'
puppetdb-terminus   1.6.2-1puppetlabs1
puppetmaster    3.4.3-1puppetlabs1
puppetmaster-common 3.4.3-1puppetlabs1

In puppet.conf on the master :
 
[master]
storeconfigs=true
storeconfig_backend=puppetdb
reports=store,puppetdb
 
In routes.yaml on the master :

---
master:
  facts:
    terminus: puppetdb
    cache: yaml

In puppetdb.conf on the master :
 
[main]
server=el6.labolinux.fr
port=8081
soft_write_failure=true
 
On the puppetdb node :
 
[root@el6 ~]# rpm -qa | grep '^puppet'
puppet-3.3.2-1.el6.noarch
puppetdb-1.6.2-1.el6.noarch

Database backend configuration :

[database]
classname = org.postgresql.Driver
subprotocol = postgresql
subname = //127.0.0.1:5432/puppetdb
username = puppetdb

What am I missing ?

Louis Coilliot

[Ken Barber]

This is the smoking gun. It looks like an SSL error is being thrown.

Set the soft_write_failure to false, and you should be seeing far more
errors relating to SSL I bet. I think the problem stems from the
errors being masked and probably just being stored in the masters log.
This is the correct behaviour when this setting is true, so as to
allow the master to continue to run when PuppetDB is not operational.

Switching to false will make the real error surface most probably. Can
you try changing that setting, restarting the puppet master and
display the results for us? Or find the error in the location where
you puppet master outputs its logs (daemon.log on Debian usually I
think?).

> On the puppetdb node :
>
> [root@el6 ~]# rpm -qa | grep '^puppet'
> puppet-3.3.2-1.el6.noarch

Why is your Puppet agent on the puppetdb node running an older
revision then the master? I presume you are running Puppet on the
PuppetDB node as well to manage that host correct?

> puppetdb-1.6.2-1.el6.noarch
>
> Database backend configuration :
>
> [database]
> classname = org.postgresql.Driver
> subprotocol = postgresql
> subname = //127.0.0.1:5432/puppetdb
> username = puppetdb
>
> What am I missing ?

This looks like a basic SSL setup issue but I have limited data to
work on. Try running "puppetdb ssl-setup", storing the results then
using "puppetdb ssl-setup -f" to force a manual repair of the
certificates PuppetDB uses. Make sure you restart PuppetDB before
trying again.

ken.

[Louis Coilliot]

Hello

It is still broken.

I set soft_write_failure=false

I upgraded puppet on the nodes, so now the puppet master and nodes are in version 3.4.3

This is the result of the puppetdb ssl-setup :

[root@el6 lofic]# puppetdb ssl-setup
PEM files in /etc/puppetdb/ssl already exists, checking integrity.
Setting ssl-host in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-port in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-key in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-cert in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-ca-cert in /etc/puppetdb/conf.d/jetty.ini already correct.

[root@el6 lofic]# puppetdb ssl-setup -f
PEM files in /etc/puppetdb/ssl already exists, checking integrity.
Overwriting existing PEM files due to -f flag
Copying files: /var/lib/puppet/ssl/certs/ca.pem, /var/lib/puppet/ssl/private_keys/el6.labolinux.fr.pem and /var/lib/puppet/ssl/certs/el6.labolinux.fr.pem to /etc/puppetdb/ssl
Setting ssl-host in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-port in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-key in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-cert in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-ca-cert in /etc/puppetdb/conf.d/jetty.ini already correct.

I restarted the puppetdb

The catalogs are still absent.

When I launch the master in debug +trace mode, I see :

Debug: Failed to load library 'msgpack' for feature 'msgpack'
Debug: file_metadata supports formats: pson b64_zlib_yaml yaml raw
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': iconv will be deprecated in the future, use String#encode instead.
Warning: ActiveRecord-based storeconfigs and inventory are deprecated. See http://links.puppetlabs.com/activerecord-deprecation
   (at /usr/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:334:in `new')
Debug: Using settings: adding file resource 'dblocation': 'File[/var/lib/puppet/state/clientconfigs.sqlite3]{:path=>"/var/lib/puppet/state/clientconfigs.sqlite3", :mode=>"660", :owner=>"puppet", :group=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'railslog': 'File[/var/log/puppet/rails.log]{:path=>"/var/log/puppet/rails.log", :mode=>"600", :owner=>"puppet", :group=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Finishing transaction 23034320
Info: Connecting to sqlite3 database: /var/lib/puppet/state/clientconfigs.sqlite3
Debug: Configuring PuppetDB terminuses with config file /etc/puppet/puppetdb.conf

The name resolution seems fine for the master, the puppetd and the nodes

[root@el6 conf.d]# host beaker.labolinux.fr
beaker.labolinux.fr has address 192.168.0.10
[root@el6 conf.d]# host 192.168.0.10
10.0.168.192.in-addr.arpa domain name pointer beaker.labolinux.fr.
[root@el6 conf.d]# host el6.labolinux.fr
el6.labolinux.fr has address 192.168.0.16
[root@el6 conf.d]# host 192.168.0.16
16.0.168.192.in-addr.arpa domain name pointer el6.labolinux.fr.
[root@el6 conf.d]# host el6d.labolinux.fr
el6d.labolinux.fr has address 192.168.0.63
[root@el6 conf.d]# host 192.168.0.63
63.0.168.192.in-addr.arpa domain name pointer el6d.labolinux.fr.

I still have the SSL problem :

# puppet node status el6.labolinux.fr --verbose --debug --trace
Debug: Configuring PuppetDB terminuses with config file /etc/puppet/puppetdb.conf
Debug: Failed to load library 'selinux' for feature 'selinux'
Debug: Using settings: adding file resource 'confdir': 'File[/etc/puppet]{:path=>"/etc/puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Puppet::Type::User::ProviderPw: file pw does not exist
Debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/uuidgen does not exist
Debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not exist
Debug: Failed to load library 'ldap' for feature 'ldap'
Debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
Debug: /User[puppet]: Provider useradd does not support features libuser; not managing attribute forcelocal
Debug: Puppet::Type::Group::ProviderPw: file pw does not exist
Debug: Puppet::Type::Group::ProviderDirectoryservice: file /usr/bin/dscl does not exist
Debug: Failed to load library 'ldap' for feature 'ldap'
Debug: Puppet::Type::Group::ProviderLdap: feature ldap is missing
Debug: /Group[puppet]: Provider groupadd does not support features libuser; not managing attribute forcelocal
Debug: Using settings: adding file resource 'vardir': 'File[/var/lib/puppet]{:path=>"/var/lib/puppet", :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'logdir': 'File[/var/log/puppet]{:path=>"/var/log/puppet", :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'statedir': 'File[/var/lib/puppet/state]{:path=>"/var/lib/puppet/state", :mode=>"1755", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'rundir': 'File[/var/run/puppet]{:path=>"/var/run/puppet", :mode=>"755", :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'libdir': 'File[/var/lib/puppet/lib]{:path=>"/var/lib/puppet/lib", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'hiera_config': 'File[/etc/puppet/hiera.yaml]{:path=>"/etc/puppet/hiera.yaml", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'certdir': 'File[/var/lib/puppet/ssl/certs]{:path=>"/var/lib/puppet/ssl/certs", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'ssldir': 'File[/var/lib/puppet/ssl]{:path=>"/var/lib/puppet/ssl", :mode=>"771", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:path=>"/var/lib/puppet/ssl/public_keys", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'requestdir': 'File[/var/lib/puppet/ssl/certificate_requests]{:path=>"/var/lib/puppet/ssl/certificate_requests", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'privatekeydir': 'File[/var/lib/puppet/ssl/private_keys]{:path=>"/var/lib/puppet/ssl/private_keys", :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'privatedir': 'File[/var/lib/puppet/ssl/private]{:path=>"/var/lib/puppet/ssl/private", :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'hostprivkey': 'File[/var/lib/puppet/ssl/private_keys/beaker.labolinux.fr.pem]{:path=>"/var/lib/puppet/ssl/private_keys/beaker.labolinux.fr.pem", :mode=>"600", :owner=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'hostpubkey': 'File[/var/lib/puppet/ssl/public_keys/beaker.labolinux.fr.pem]{:path=>"/var/lib/puppet/ssl/public_keys/beaker.labolinux.fr.pem", :mode=>"644", :owner=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'localcacert': 'File[/var/lib/puppet/ssl/certs/ca.pem]{:path=>"/var/lib/puppet/ssl/certs/ca.pem", :mode=>"644", :owner=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'hostcrl': 'File[/var/lib/puppet/ssl/crl.pem]{:path=>"/var/lib/puppet/ssl/crl.pem", :mode=>"644", :owner=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'pluginfactdest': 'File[/var/lib/puppet/facts.d]{:path=>"/var/lib/puppet/facts.d", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
Debug: /File[/etc/puppet/hiera.yaml]: Autorequiring File[/etc/puppet]
Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl/private_keys/beaker.labolinux.fr.pem]: Autorequiring File[/var/lib/puppet/ssl/private_keys]
Debug: /File[/var/lib/puppet/ssl/public_keys/beaker.labolinux.fr.pem]: Autorequiring File[/var/lib/puppet/ssl/public_keys]
Debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring File[/var/lib/puppet/ssl/certs]
Debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/facts.d]: Autorequiring File[/var/lib/puppet]
Debug: Finishing transaction 19965520

Error: Could not retrieve status for el6.labolinux.fr: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A

In the puppetdb.log I see :

2014-02-28 14:13:11,984 INFO  [clojure-agent-send-off-pool-2] [server.AbstractConnector] Started SelectChannelConnector@localhost:8080
2014-02-28 14:13:12,229 INFO  [clojure-agent-send-off-pool-2] [ssl.SslContextFactory] Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
2014-02-28 14:13:12,241 INFO  [clojure-agent-send-off-pool-2] [server.AbstractConnector] Started SslSelectCha...@el6.labolinux.fr:8081

And something like :

2014-02-28 14:13:14,120 WARN  [qtp1396798521-46] [io.nio] javax.net.ssl.SSLHandshakeException: null cert chain

when I try the command = puppet node status

When I run the agent on a node I see for example :
2014-02-28 14:19:09,268 INFO  [command-proc-52] [puppetdb.command] [87024599-94b4-4b2c-a324-b8ea39d26bf0] [replace facts] el6d.labolinux.fr
2014-02-28 14:19:19,985 INFO  [command-proc-52] [puppetdb.command] [bd719c3f-aeb4-41a1-98ac-a3524acb1107] [store report] puppet v3.4.3 - el6d.labolinux.fr

and no errors, but still no catalogs in the db.

It seems that /var/lib/puppet/state/clientconfigs.sqlite3 is refreshed after each agent run.

Is this a normal store ?

I had to install the activerecord and sqlite3 gems, otherwise the run of puppet on the nodes was complaining with a lack of activerecord.

When I do a puppet run I see in the logs of the master :


Info: Caching node for el6f.labolinux.fr
Debug: Saved catalog to database in 1.19 seconds
Debug: Failed to load library 'msgpack' for feature 'msgpack'
Debug: catalog supports formats: pson b64_zlib_yaml yaml dot raw

(... many messages like the previous 2 lines...)

Debug: Failed to load library 'msgpack' for feature 'msgpack'
Debug: file_metadata supports formats: pson b64_zlib_yaml yaml raw
Debug: Received report to process from el6f.labolinux.fr
Debug: Processing report from el6f.labolinux.fr with processor Puppet::Reports::Store
Debug: Processing report from el6f.labolinux.fr with processor Puppet::Reports::Puppetdb
Info: 'store report' command for el6f.labolinux.fr submitted to PuppetDB with UUID 860538fe-4f58-4b35-888b-86b1678df601

I'm stuck there.

Louis

ken.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNTmPhj2xpBiitN2e3q3%2BDmt43w%2BDLidA3j8yX_oS9h3cuA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ken Barber]

Aah, the old null cert chain error. I haven't been able to fix this
yet and every time I try this over a mailing list I never get to a
conclusion ... I suggest jumping onto Freenode IRC and pinging me - my
IRC nick is ken_barber and I usually live in #puppet, I'll be happy to
help you there.

ken.

> https://groups.google.com/d/msgid/puppet-users/CAE9jN33eYAhA32i0zqdHi0a5bUimpQVb4oE2L8EtG3y6%2B5M%2B%3DA%40mail.gmail.com.