Troubleshooting Nagios_host <<||>>

[Mikael]

Hi all,
I've been struggling the last week trying to setup a simple Vagrant + Puppet + Nagios configuration. Everything has been working out quite well but now we've hit a roadblock trying to wrap our heads around how to use Nagios_host <<||>>.
The setup is basic: we use an Ubuntu Precise 64 Vagrant box, followed the instructions from Puppetlabs (http://docs.puppetlabs.com/puppet/latest/reference/lang_exported.html) on how to setup Puppetdb, connect the master etc. On top of this we installed Nagios3.

The goal is to make this first box a puppet master and Nagios server and to be able to add slaves to this configuration at a later stage. Things seem to work, running "puppet agent --test" generates some sane log entries in the Puppetdb log and so on. However, as soon as we add Nagios_host <<||>> to the mix we start to get the following warnings:

"Warning: Not collecting exported resources without storeconfigs" and "Warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line /...".

No configuration files are generated in /etc/nagios3/ either. We would at least expect a /etc/nagios3/nagios_host.conf file to created.

The googable information one can find regarding issues like these tend to point towards adding storeconfigs=true and storeconfigs_backend=puppetdb to /etc/puppet/puppet.conf. That was done automatically for us since we used the puppet::puppetdb::config and puppetdb classes in our manifests. Issuing "puppet config print all | grep store" and "puppet master --configprint all | grep store" differes depending on the logged in user. In our Vagrant box the puppet master service is run as the user "puppet". This user for some reason tries to pull the configuration from "/var/lib/puppet/.puppet". Is there a way to figure out which configuration file Puppet actually loaded? To me it seems "--configprint" and "config print" shows what Puppet would try to load starting the service as the logged in user. Have we misunderstood this completely? Or is it something else causing Puppet not to generate the expected Nagios configuration files?

We have tried a few things in order to understand this issue but still no go:
- Run the service/services as root "puppet master --user root" (would load the configuration from /etc/puppet).
- Set the confdir, vardir, stireconfigs etc from the command line "puppet master --storeconfigs --confdir /etc/puppet/ --vardir /var/lib/puppet"
- Tried CentOS
- Added tons of logging where we could "--debug --verbose --trace"

We also tried the following Vagrant+Puppet box from GitHub (removing the Puppet dashboard stuff, adding Nagios): https://github.com/grahamgilbert/vagrant-puppetmaster. Same result.

Clearly people have this working so it really has to be us not figuring this out. What would be a good next step in order to troubleshoot this issue? Can we add more logging somewhere?

Any input would be much appreciated!

Thanks,
Mikael

Attaching some information below for reference:
// /etc/puppet/puppet.conf
[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates
pluginsync=true
[agent]
report=true
[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
dns_alt_names = puppet, puppet.local, puppet.pebbleit.dev
reports = store, http, puppetdb
#reports = store, http
reporturl = http://localhost:3000/reports/upload
node_terminus = exec
external_nodes = /usr/bin/env PUPPET_DASHBOARD_URL=http://localhost:3000 /usr/share/puppet-dashboard/bin/external_node
storeconfigs = true
storeconfigs_backend = puppetdb

// /etc/puppet/puppetdb.conf
[main]
port = 8081
server = puppet.grahamgilbert.dev

// /etc/puppet/routes.yaml
# This file was automatically generated by the puppetdb module.

master:
      facts:
        terminus: puppetdb
        cache: yaml

// dpkg -l | grep puppet
vagrant@puppet:~$ dpkg -l | grep puppet
ii  facter                           2.0.1-1puppetlabs1          Ruby module for collecting simple facts about a host operating system
ii  hiera                            1.3.2-1puppetlabs1          A simple pluggable Hierarchical Database.
ii  puppet                           3.5.1-1puppetlabs1          Centralized configuration management - agent startup and compatibility scripts
ii  puppet-common                    3.5.1-1puppetlabs1          Centralized configuration management
ii  puppet-dashboard                 1.2.21-1puppetlabs1         Dashboard for Puppet
ii  puppetdb                         1.6.3-1puppetlabs1          PuppetDB Centralized Storage.
ii  puppetdb-terminus                1.6.3-1puppetlabs1          Connect Puppet to PuppetDB by setting up a terminus for PuppetDB.
ii  puppetlabs-release               1.0-7                       "Package to install Puppet Labs gpg key and apt repo"
ii  puppetmaster                     3.5.1-1puppetlabs1          Centralized configuration management - master startup and compatibility scripts
ii  puppetmaster-common              3.5.1-1puppetlabs1          Puppet master common scripts
ii  ruby-rgen                        0.6.5-1puppetlabs1          A framework supporting Model Driven Software Development (MDSD)

// uname -a
Linux puppet 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

// ps -Af | grep puppet
vagrant@puppet:~$ ps -Af | grep puppet
puppetdb  1492     1  1 09:04 ?        00:02:00 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d
postgres  1977   926  0 09:04 ?        00:00:00 postgres: puppetdb puppetdb 127.0.0.1(53676) idle                                                                          
postgres  2040   926  0 09:04 ?        00:00:00 postgres: puppetdb puppetdb 127.0.0.1(53687) idle                                                                          
postgres  4200   926  0 10:04 ?        00:00:00 postgres: puppetdb puppetdb 127.0.0.1(53721) idle                                                                          
root      4591     1  0 10:25 ?        00:00:00 /usr/bin/ruby /usr/bin/puppet agent --storeconfigs
puppet    7606     1  1 10:37 ?        00:00:41 /usr/bin/ruby /usr/bin/puppet master --masterport=8140
postgres  8118   926  0 11:04 ?        00:00:00 postgres: puppetdb puppetdb 127.0.0.1(53801) idle                                                                          
postgres  8119   926  0 11:04 ?        00:00:00 postgres: puppetdb puppetdb 127.0.0.1(53802) idle                                                                          
postgres  8120   926  0 11:04 ?        00:00:00 postgres: puppetdb puppetdb 127.0.0.1(53803) idle                                                                          
postgres  8121   926  0 11:04 ?        00:00:00 postgres: puppetdb puppetdb 127.0.0.1(53804) idle                                                                          
vagrant   8251  7685  0 11:14 pts/0    00:00:00 grep --color=auto puppet
vagrant@puppet:~$
 

[Mikael]

I ended up moving storeconfigs=true and storeconfigs_backend=puppetdb to the [main] section in /etc/puppet/puppet.conf. No idea why that works as it seems to contradict every piece of documentation and examples I've come across. If I find time I'll try to figure out why this is.

Thanks,
Mikael