empty catalog on Puppet 3.8.7

[Jeffrey Honig]

I'm using Puppet 3.8.7 on Ubuntu 14.04 64-bit.

After a system reboot on the server (also a client in this example), puppet master is returning an empty catalog:

{"document_type":"Catalog","data":{"tags":["settings"],"name":"mittens.honig.net","version":1467562258,"environment":"production","resources":[{"type":"Stage","title":"main","tags":["stage","main"],"exported":false},{"type":"Class","title":"Settings","tags":["class","settings"],"exported":false},{"type":"Class","title":"Main","tags":["class","main"],"exported":false,"parameters":{"name":"main"}}],"edges":[{"source":"Stage[main]","target":"Class[Settings]"},{"source":"Stage[main]","target":"Class[Main]"}],"classes":["settings"]},"metadata":{"api_version":1}}root@mittens:/var/lib/puppet/client_data/catalog

I can not see anything else in the syslog.

Comparing backups of /etc/puppet and /var/lib/puppet and my , besides differences in reports I see this:

Only in daily.1/mittens/var/lib/puppet/lib: facter

Only in daily.1/mittens/var/lib/puppet/lib: puppet

Only in daily.1/mittens/var/lib/puppet/lib: puppet_x

When I restore those directories the first puppet run shows:

Jul  3 17:45:15 mittens puppet-agent[25625]: (/File[/var/lib/puppet/lib/puppet]/ensure) removed

Jul  3 17:45:15 mittens puppet-agent[25625]: (/File[/var/lib/puppet/lib/puppet_x]/ensure) removed

Jul  3 17:45:15 mittens puppet-agent[25625]: (/File[/var/lib/puppet/lib/facter]/ensure) removed

Also different from the backups is that /var/lib/puppet/state/classes.txt now only as "settings" instead of a large list of states and resources.txt is now empty. 

I do not know of the above are the cause of the problem or a symptom.

This happened about a week ago and I thought it was related to updating all my forge modules with librarian.  I restored them from backups, but held back the update of staging from 1.0.7 to 2.0 and that appeared to fix it.  However I have not used librarian since and the modules.  Backups show no changes to those modules.

Any advice for continued debugging?

Thanks.

Jeff

[Jeffrey Honig]

I finally figured it out.  I had some bad hiera configs for puppet, where I was trying to update puppet::params: 

-puppet::params::server: true

-puppet:params::server_foreman: false

-puppet:params::server_reports: "store"

-puppet:params::server_external_nodes: ""

+puppet::server: true

+puppet::server_foreman: false

+puppet::server_passenger: false

+puppet::server_reports: "store"

+puppet::server_external_nodes: ""

The newer version of theforeman-puppet changed the way it built puppet.conf and environmentpath was being left out of puppet.conf due to my blunder.  So a puppet run would update the server config and remove some necessary config items, but this would not take effect until I rebooted (or manually restarted puppetmaster).