Hiera and puppet apply/puppet agent

[kai]

I have the following hiera.yaml file:

---

:backends:

  - yaml

  - file

:hierarchy:

  - defaults

  - "%{clientcert}"

  - "%{::domain}/%{::environment}/%{::osfamily}/%{::lsbdistcodename}"

  - global

:yaml:

  :datadir: /etc/puppet/data

and the following in /etc/puppet/data:

loc.example.com/production/Debian/wheezy.yaml

The domain fact returns loc.example.com, the osfamily fact returns Debian and the lsbdistcodename returns wheezy.

When I run 

puppet apply --environment=production -e '$test = hiera('ssh_service_config') notify { $test: }'

I get the ssh_service_config variable from the  /etc/puppet/data/loc.example.com/production/Debian/wheezy.yaml file, which in my mind means that the correct file was chosen based on the 3 facts (environment. osfamily and lsbdistcodename), which is what I want.

However if I run:

$ puppet agent -tv --environment=production

Info: Retrieving plugin

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item ssh_package_name in any Hiera data file and no default supplied at /srv/puppet/environments/production/modules/ssh/manifests/install.pp:2 on node puppetmaster.iad3.example.com

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Or:

$ hiera -d ssh_service_config environment=production osfamily=Debian lsbdistcodename=wheezy

DEBUG: 2014-01-06 19:18:08 +0000: Hiera YAML backend starting

DEBUG: 2014-01-06 19:18:08 +0000: Looking up ssh_service_config in YAML backend

DEBUG: 2014-01-06 19:18:08 +0000: Looking for data source defaults

DEBUG: 2014-01-06 19:18:08 +0000: Looking for data source global

DEBUG: 2014-01-06 19:18:08 +0000: Hiera File backend starting

DEBUG: 2014-01-06 19:18:08 +0000: Looking up ssh_service_config in File backend

DEBUG: 2014-01-06 19:18:08 +0000: Hiera File_backend: looking for data source 'defaults'

DEBUG: 2014-01-06 19:18:08 +0000: Cannot find datafile /var/lib/hiera/defaults.d, skipping

DEBUG: 2014-01-06 19:18:08 +0000: Hiera File_backend: looking for data source 'global'

DEBUG: 2014-01-06 19:18:08 +0000: Cannot find datafile /var/lib/hiera/global.d, skipping

nil

Any idea what is going on? I would like to have a simple hierarchy like the one in the hiera.yaml file, based on the three facts.

[Andrey Kozichev]

I assume "ssh_service_config" is indeed defined in your yaml file?

I would suggest to define it in the top level hierarchy in "defaults" and then gradually move 1 level down until you find when it "disappears". 

Andrey

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f139fb4c-92f2-4616-a522-6bd69ecfd9fb%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[kai]

Yes it's defined in :

"%{::domain}/%{::environment}/%{::osfamily}/%{::lsbdistcodename}"

which is expanded to loc.example.com/production/Debian/wheezy.yaml

The problem is that when I run:

$ puppet apply --environment=production -e '$test = hiera('ssh_service_config') notify { $test: }'

Debug: hiera(): Looking for data source loc.example.com/production/Debian/wheezy

Debug: hiera(): Found ssh_service_config in loc.example.com/production/Debian/wheezy

it works just fine, but when I use:

$ puppet agent -tv

it fails to find the variable. If I remove the %{::domain} however then it works with puppet agent as well.

$ facter domain

loc.example.com

[Andrey Kozichev]

Anything in the puppet master logs?

Not sure, but i think puppet apply - hiera lookup is done by client,
puppet agent - lookup is done by puppet master.

Andrey

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/126d00bd-2254-49a0-938b-db55a828a702%40googlegroups.com.

[jcbollinger]

On Monday, January 6, 2014 4:02:54 PM UTC-6, Andrew wrote:

Anything in the puppet master logs?

Not sure, but i think puppet apply - hiera lookup is done by client,
puppet agent - lookup is done by puppet master.

That was the area I was about to prod, too.  The hiera lookup is performed as part of catalog compilation, therefore it is done by the master when you use "puppet agent".  On the other hand, the local machine performs the compilation, and thus the hiera lookup, when you use "puppet apply".

Even when running the test on the machine hosting the master, it is possible that the master sees a different puppet configuration than "puppet agent" or "puppet apply" does, depending on the user identity with which those are launched.  Furthermore, discrepancies could arise if access controls prevent the master from reading the data file, but do permit "puppet apply" to read it.


John

[kai]

Hm nothing in the logs. It's worth mentioning that I run this on the puppet master. Here's the config in case I missed something there:

[main]

pluginsync = true

factpath = /lib/facter

[agent]

environment = production

server = puppetmaster.loc.example.com

runinterval = 360

splay = true

configtimeout = 1200

noop = false

autoflush = true

graph = false

report = false

ca_server = puppetmaster.loc.example.com

report_server = puppetmaster.loc.example.com

pidfile = /var/run/puppet/agent.pid

[master]

autosign = true

autoflush = true

reports = store

ssl_client_header = HTTP_X_CLIENT_DN

ssl_client_verify_header = HTTP_X_CLIENT_VERIFY

[production]

manifestdir = /srv/puppet/environments/production/manifests

manifest    = /srv/puppet/environments/production/manifests/site.pp

modulepath  = /srv/puppet/environments/production/modules

[jcbollinger]

On Monday, January 6, 2014 4:42:54 PM UTC-6, kai wrote:

Hm nothing in the logs. It's worth mentioning that I run this on the puppet master. Here's the config in case I missed something there:

What version of Puppet are you running?

As what user is the master running?  (Typically an unprivileged user named 'puppet'.)

As what user are you running the agent in your tests?

As what user are you running "puppet apply" in your tests?

What are the ownership and permissions of /etc/puppet/data/loc.example.com/production/Debian/wheezy.yaml and all the directories in the path to it?  Do any of those have extended ACLs set on them?

Are you running SELinux in enforcing mode?  If so, then do you see relevant AVC messages in the system log?


John

[kai]

What version of Puppet are you running?

3.4.1 for both master and agent

As what user is the master running?  (Typically an unprivileged user named 'puppet'.)

The master is running as user puppet

As what user are you running the agent in your tests?

I am running the agent and the apply commands on the master as the root user

As what user are you running "puppet apply" in your tests?

root

What are the ownership and permissions of /etc/puppet/data/loc.example.com/production/Debian/wheezy.yaml and all the directories in the path to it?  Do any of those have extended ACLs set on them?

The permissions of the directory and the yaml file are root:root. No ALC's. I chown-ed then to puppet:puppet with the same result.

Are you running SELinux in enforcing mode?  If so, then do you see relevant AVC messages in the system log?

This is a Debian Wheezy, so no selinux or apparmor on it.

The bellow works fine with this:  

$ puppet apply --debug --environment=production -e '$test = hiera('ssh_service_config') notify { $::test: }'

Debug: hiera(): Hiera YAML backend starting

Debug: hiera(): Looking up ssh_service_config in YAML backend

Debug: hiera(): Looking for data source defaults

Debug: hiera(): Looking for data source puppetmaster.iad3.example.com

Debug: hiera(): Cannot find datafile /etc/puppet/data/puppetmaster.iad3.example.com.yaml, skipping

Debug: hiera(): Looking for data source production/Debian/wheezy

Debug: hiera(): Found ssh_service_config in production/Debian/wheezy

but not with 

$ pupept agent -tv

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item ssh_package_name in any Hiera data file and no default supplied at /srv/puppet/environments/production/modules/ssh/manifests/install.pp:2 on node puppetmaster.iad3.example.com

:hierarchy:

  - "%{::environment}/%{::osfamily}/%{::lsbdistcodename}"     <--- This works with  puppet apply, but not puppet agent -tv 

This is really strange, any help will be greatly appreciated. (Appending the domain fact produces the same result)

[Jose Luis Ledesma]

Mmm the error is about ssh_package_name, but you have tried the puppet apply with ssh_service_name. could be this the problem?

[Andrey Kozichev]

Content of the yaml file - any quotes there which might cause issues?
Try puppet master compile for the node - is it failing as well?

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/2df4b281-7a41-4f16-a59f-34c574ae985f%40googlegroups.com.

[kai]

One more thing to mention is that if I change the hierarchy from being a directory as mentioned above to just a file like so:

:hierarchy:

    - "%{::environment}_%{::osfamily}_%{::lsbdistcodename}"

both puppet apply and puppet agent are able to find the variable.

However if I prepend the "domain" fact like this:

- "%{::domain}_%{::environment}_%{::osfamily}_%{::lsbdistcodename}"

even though it's just a single file again, puppet apply works, but puppet agent -tv again is not able to find the variable.

This does not make any sense to me and I am starting to wonder if this is a bug.

[kai]

I actually have both variables.

[kai]

---

:backends:

  - yaml

  - file

:hierarchy:

  - defaults

  - "%{clientcert}"

  - "%{::domain}/%{::environment}/%{::osfamily}/%{::lsbdistcodename}"

  - global

:yaml:

  :datadir: /etc/puppet/data

[jcbollinger]

On Tuesday, January 7, 2014 11:39:14 AM UTC-6, kai wrote:

What version of Puppet are you running?
3.4.1 for both master and agent
As what user is the master running?  (Typically an unprivileged user named 'puppet'.)
The master is running as user puppet
As what user are you running the agent in your tests?
I am running the agent and the apply commands on the master as the root user
As what user are you running "puppet apply" in your tests?
root
What are the ownership and permissions of /etc/puppet/data/loc.example.com/production/Debian/wheezy.yaml and all the directories in the path to it?  Do any of those have extended ACLs set on them?
The permissions of the directory and the yaml file are root:root. No ALC's. I chown-ed then to puppet:puppet with the same result.
Are you running SELinux in enforcing mode?  If so, then do you see relevant AVC messages in the system log?
This is a Debian Wheezy, so no selinux or apparmor on it.

The point here was that the master need to be able to access and read those hiera data files that exist, and indeed to determine which do exist.  That does not necessarily require user 'puppet' to own the file and directories, but it does require 'puppet' to have read permission on the file itself and both read and traverse permission on ALL directories in the path to the file.

 

The bellow works fine with this:  

$ puppet apply --debug --environment=production -e '$test = hiera('ssh_service_config') notify { $::test: }'

Debug: hiera(): Hiera YAML backend starting

Debug: hiera(): Looking up ssh_service_config in YAML backend

Debug: hiera(): Looking for data source defaults

Debug: hiera(): Looking for data source puppetmaster.iad3.example.com

Debug: hiera(): Cannot find datafile /etc/puppet/data/puppetmaster.iad3.example.com.yaml, skipping

Debug: hiera(): Looking for data source production/Debian/wheezy

Debug: hiera(): Found ssh_service_config in production/Debian/wheezy

but not with 

$ pupept agent -tv

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item ssh_package_name in any Hiera data file and no default supplied at /srv/puppet/environments/production/modules/ssh/manifests/install.pp:2 on node puppetmaster.iad3.example.com

So far I don't think you have presented the master's logs for the failure case.  Those might be illuminating.  Do first restart the master with debug logging enabled to gather the most information.

You could also put Notify resources or notice() calls in your manifests on the master to verify that the fact values the agent presents to the master are the same as the ones used by "puppet apply".

Overall, I see only two reasonable possibilities:

1. the master is looking for a different data file than "puppet apply" reads, or no file at all, or
2. the master is prevented from reading the specified file, and perhaps from even testing for its existence.

The former would suggest a configuration issue, whereas the latter would be some kind of access control issue.  I am so far disinclined to think that there is a Puppet bug in play here.


John

[kai]

John,

thank you for pointing me in the right direction. The problem was that each time I made changes to hiera.yaml file I'll restart the master with service puppetmaster restart.

Well apparently the master was not really restarting even though it was saying it was. So it was going into this strange state where sometimes puppet agent -tv would find the file and sometimes it would not.

So after each change I had to go and kill the process because using service puppetmaster restart/stop did not really kill the process. I tried this on Centos instead of Debian Wheezy and did not have this problem.