Hiera seems to be using stale data

[Jonathan Gazeley]

Hi chaps,

I'm using Hiera in a light way to store a few global values for my
servers, including the site-wide SMTP relay etc. Yesterday I changed one
of the values to reflect my organisation's new SMTP server, but the
change hasn't been applied to my nodes. I'm using camptocamp/postfix to
configure the Postfix MTA on all my nodes

common.yaml:
---
uob_mail_relay: new-smtp.bris.ac.uk


init.pp:
class site_postfix {
# Configure basic Postfix with minimal options
class { 'postfix':
myorigin => 'bris.ac.uk',
relayhost => hiera('uob_mail_relay'),
root_mail_recipient => hiera('admin_email'),
inet_interfaces => 'localhost',
}
...
...
...
}


After having changed the value in common.yaml, the Postfix config was
not changed on any of my nodes. I stopped and restarted the puppetmaster
and double-checked that the right manifests are being applied to my
nodes. Running hiera on the command line returns the new value for
'uob_mail_relay' so I have no idea why it hasn't picked up inside
puppet. Nothing bad is shown in the logs on the master or the node. Can
anyone make a suggestion?

Thanks,
Jonathan

[jcbollinger]

Strange.  The problem must fall into one of these categories:

• The master is not obtaining the updated value from hiera
• The master is not including the hiera value in the compiled catalog
• The agents are not running
  
• The agents are applying stale catalogs that do not contain the change
  
• Despite the catalogs specifying an updated relayhost, the agents are not applying the change
• You are mistaken about the change not being applied
• The change is being reverted, either by some other puppet resource or by an external actor.
  

Supposing that we can discount the last alternative, I suggest driving at this first from the agent side:

1. Run the agent from the command line with the --test and --debug options, and capture the output for analysis.
2. Verify that the agent reports successfully obtaining a catalog from the master.  If so,
   
3. find in the output the report of the specific File resources you expect to be modified -- confirm that the reports are in fact there, and that they are reported either already in sync or successfully synchronized.  Do similarly for any Service or other resources that you would expect to need to be touched.  If all that looks good then
   
4. check the output for any other resource that might be touching the same files.  Also,
   
5. check whether the files are in fact as expected.  If not, then
   
6. look at the catalog.  Find the relevant resource(s) in it and check whether their properties are as expected.
   

Having done that, you should have a pretty good handle on the nature of the problem -- whether it is bad (or no) agent behavior or bad catalogs, and if the latter, the nature of the catalog fault.  If you still need help at that point then come back with your findings.


John