Could not find class defined for specific environment when running puppet agent

[Jamals]

Hi all,

I have a curious problem that I'd appreciate help/guidance on.

We are running puppetserver 2.6.0 (puppetmaster.domain.ca) with cobbler 2.6.11 as our ENC. All our servers/nodes are centos 7.3

On the puppet client (node.domain.ca) when running the puppet agent we receive the following output/error:

[ro...@node.domain.ca ~]# 

Redirecting to /bin/systemctl stop  puppet.service

Info: Caching certificate for node.domain.ca

Info: Caching certificate_revocation_list for ca

Info: Caching certificate for node.domain.ca

Info: Using configured environment 'ops'

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Loading facts

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find class users::opsusers for node.domain.ca on node node.domain.ca

Info: Using cached catalog from environment 'ops'

Info: Applying configuration version '1487721491'

Notice: /Stage[main]/Puppet::Agent/Service[puppet]/ensure: ensure changed 'stopped' to 'running'

Info: /Stage[main]/Puppet::Agent/Service[puppet]: Unscheduling refresh on Service[puppet]

Error: /Stage[main]/Main/Selinux::Audit2allow[zabbix_bug]/File[/etc/selinux/local/zabbix_bug/messages]: Could not evaluate: Could not retrieve information from environment ops source(s) puppet:///modules/node2.domain.ca/messages.zabbix

Notice: /Stage[main]/Main/Selinux::Audit2allow[zabbix_bug]/Exec[semodule -r localzabbix_bug; rm -f localzabbix_bug.*; audit2allow -M localzabbix_bug -i messages && semodule -i localzabbix_bug.pp]: Dependency File[/etc/selinux/local/zabbix/messages] has failures: true

Warning: /Stage[main]/Main/Selinux::Audit2allow[zabbix_bug]/Exec[semodule -r localzabbix_bug; rm -f localzabbix_bug.*; audit2allow -M localzabbix_bug -i messages && semodule -i localzabbix_bug.pp]: Skipping because of failed dependencies

Notice: /Stage[main]/Legato/File[/nsr/tmp/sec/nsr_shutdown.pid]/seluser: seluser changed 'unconfined_u' to 'system_u'

Notice: /Stage[main]/Legato/File[/nsr/tmp/sec/services]/seluser: seluser changed 'unconfined_u' to 'system_u'

Notice: /Stage[main]/Tools/Exec[check_presence]/returns: executed successfully

Info: Stage[main]: Unscheduling all events on Stage[main]

Notice: Applied catalog in 48.03 seconds

The problem is that despite the agent claiming to retrieve the catalog for the `ops` environment it throws two errors where its not able to find class /retrieve information. 

The strange part is that the class `users::opsuser` is only defined under the ops/classes directory for that environment.

The second error is thrown by module `selinux::audit2allow` module and claims that the environment `ops` can not find source puppet:///modules/node2.domain.ca/messages.zabbix  BUT this path is ONLY defined in the production environment's site.pp.  

The site.pp for the environment 'ops' has the following source defined for this message file for the selinux::audit2allow module:

puppet:///modules/selinux/messages.zabbix

This says to me that the agent is relying on some default `production` environment values but it is not clear to me how/where this is happening. 

Here's a snippet of the tree output for the /etc/puppetlabs/code directory on puppetmaster with key files shown listed: 

[root@puppetmaster /etc/puppetlabs/]# tree -L 6 code

code

├── environments

│   ├── ops

│   │   ├── classes

│   │   │   └── node.domain.ca

│   │   │       ├── files

│   │   │       └── manifests

│   │   │           └── init.pp

│   │   ├── environment.conf

│   │   ├── hieradata

│   │   ├── manifests

│   │   │   └── site.pp

│   │   ├── modules

│   │   │   ├── selinux

│   │   │   │   ├── files

│   │   │   │   │   └── messages.zabbix

│   │   │   │   ├── manifests

│   │   │   │   │   ├── audit2allow.pp

│   │   │   │   │   └── init.pp

│   │   │   ├── users

│   │   │   │   ├── files

│   │   │   │   ├── manifests

│   │   │   │   │   ├── init.pp

│   │   │   │   │   └── opsusers.pp

│   │   │   └── zabbix

│   ├── node.rb

│   └── production

│       ├── classes

│       │   ├── node2.domain.ca

│       │   │   ├── files

│       │   │   │   └── messages.zabbix

│       │   │   └── manifests

│       │   │       └── init.pp

│       ├── environment.conf

│       ├── hieradata

│       ├── manifests

│       │   └── site.pp

│       ├── modules

│       │   ├── selinux

│       │   │   ├── files

│       │   │   │   └── messages.zabbix

│       │   │   ├── manifests

│       │   │   │   └── audit2allow.pp

│       │   ├── users

│       │   │   ├── files

│       │   │   └──  manifests

│       │   └── zabbix

├── hiera.yaml

└── modules

The 'ops' environment.conf file has these 2 lines defined:

modulepath = ./modules:./classes:$basemodulepath

manifest = ./manifests

The `production` environment.conf has the following:

modulepath = ./modules:./classes:$basemodulepath

# manifest = (default_manifest from puppet.conf, which defaults to ./manifests)

The `/etc/puppetlabs/puppet/puppet.conf` :

[main]

environmentpath = /etc/puppetlabs/code/environments

basemodulepath = /etc/puppetlabs/code/modules:/opt/puppetlabs/puppet/modules/

node_terminus = exec

external_nodes = /etc/puppetlabs/code/environments/node.rb

[master]

vardir = /opt/puppetlabs/server/data/puppetserver

logdir = /var/log/puppetlabs/puppetserver

rundir = /var/run/puppetlabs/puppetserver

pidfile = /var/run/puppetlabs/puppetserver/puppetserver.pid

codedir = /etc/puppetlabs/code

server = puppetmaster.domain.ca

storeconfigs = true

storeconfigs_backend = puppetdb

node_terminus = exec

external_nodes = /etc/puppetlabs/code/environments/node.rb

ON the client (node.domain.ca), we run the agent like so:

[ro...@node.domain.ca ~]# service puppet stop && puppet agent -o -v --server puppetmaster.domain.ca --environment ops --no-daemonize

AND on that node the /etc/puppetlabs/puppet/puppet.conf file has the following defined in it:

server = puppetmaster.domain.ca

report = false

runinterval = 7200

environment = ops

Maybe I'm missing something trivial, but if anyone has ideas as to what that might be or why the puppet agent suggests that files/sources from the `production` environment are being retrieved as opposed to what clearly should be the `ops` environment, I'd be very happy to have your input.

Thank you.

[Carthik Sharma]

Hi,

The first error (Error 500) seems to indicate that the file
permissions for one or more files/directories in the path to the
opsusers.pp file does not allow puppet to read the file.

The second error might be due to code that specifically refers to the
path "puppet:///modules/node2.domain.ca/messages.zabbix" in the puppet
code in your ops environment. A grep on the code in the ops
environment should ascertain this. Could it be that you copied the
module se2linux from production to ops and the path to the file was
left unchanged in the puppet code?

Best,
Carthik.

[John Gelnaw]

I would ignore everything past "Using cached catalog from environment 'ops'" in your debugging, as it's cached and therefore probably represents a previous state of the puppet master catalog.

Especially for testing, you can replace your puppet run with:

service puppet stop && puppet agent --test --server puppetmaster.domain.ca --environment ops

since --test means "--onetime --verbose --no-daemonize" and a couple of cache-disabling options as well, so you'll eliminated cache data, and simplify your troubleshooting.

You don't really need the server or environment either, since you've got them defined in your puppet.conf, so really, you should be able to use:

puppet agent --test

Having said that, error 500 is a pretty broad error, which usually means "something went wrong".  In this case, I suspect it's telling you the truth that it can't find the users::opsuser class-- which means either the puppet server can't read the file (selinux or file permissions, or what's happened to me too many times to ignore, a typo in the class definition.... :) ), or it's actually looking in the production environment.

The only reason I can think of why it would be looking in the production environment would be if node.rb told it to.

What happens when you manually run:

/etc/puppetlabs/code/environments/node.rb node.domain.ca

I suspect node.rb will spit out an "environment: production" yaml string, which is overriding the "environment ops" in your config / run statement.

[Jamals]

I'm very sorry for the late reply, and thank you so much for your interest and input.

The first error (Error 500) seems to indicate that the file
permissions for one or more files/directories in the path to the
opsusers.pp file does not allow puppet to read the file.

I verified permissions and this doesn't seem likely as everything under /etc/puppetlabs seems to be globally readable.

 

The second error might be due to code that specifically refers to the
path "puppet:///modules/node2.domain.ca/messages.zabbix" in the puppet
code in your ops environment. A grep on the code in the ops
environment should ascertain this. Could it be that you copied the
module se2linux from production to ops and the path to the file was
left unchanged in the puppet code?

 Yes, indeed the selinux was essentially a copy of the one in production.  I listed the key difference was the path reference to the messages.zabbix file.  I've gone over these files with a toothpick at this point and verified 'diff' output.

[Jamals]

Thank you so much for your response.

I would ignore everything past "Using cached catalog from environment 'ops'" in your debugging, as it's cached and therefore probably represents a previous state of the puppet master catalog.

I had gone ahead and registered the client cert with puppetmaster again to force a fresh catalog, just in case.

 

since --test means "--onetime --verbose --no-daemonize" and a couple of cache-disabling options as well, so you'll eliminated cache data, and simplify your troubleshooting.

Thank you, I've since then forced a fresh catalog a number of times.

 

You don't really need the server or environment either, since you've got them defined in your puppet.conf, so really, you should be able to use:

 

I've simplified the agent run parameters - I'm just paranoid I suppose =).

Having said that, error 500 is a pretty broad error, which usually means "something went wrong".  In this case, I suspect it's telling you the truth that it can't find the users::opsuser class-- which means either the puppet server can't read the file (selinux or file permissions, or what's happened to me too many times to ignore, a typo in the class definition.... :) ), or it's actually looking in the production environment.

Yes, that was my struggle with dis-ciphering what an error 500 code would detail.  I verified permissions as @Carthik above suggested as well, and then compared any similar class/module definitions (there's only so many in this new environment).  

 

The only reason I can think of why it would be looking in the production environment would be if node.rb told it to.

Yes, I've been verifying the node.rb output regularly (that was my initial struggle with ensuring the right environment data was being returned to puppetmaster - initially I was using --ksmeta tags on Cobbler to return the environment lablel but I found a way to modify Cobbler's system status tags to give us what we wanted). 

 

What happens when you manually run:

/etc/puppetlabs/code/environments/node.rb node.domain.ca

Here's a sample output:

[root@puppetmaster environments]# ./node.rb node.domain.ca

classes:

  fwrules: {}

  legato: {}

  postfix: {}

  users::groups: {}

  users::users: {}

  node: {}

  zabbix: {}

environment: ops

parameters:

  from_cobbler: 1

  refvm: '1'

  tree: http://@@http_server@@/cblr/links/centos_7_x64-x86_64  

 

I suspect node.rb will spit out an "environment: production" yaml string, which is overriding the "environment ops" in your config / run statement.

Unfortunately not, as you can see.  This is what was the worry part since it seems by all means/tests that the right environment was being passed to puppetmaster AND the agent. 

[Jamals]

@Carthik and @John, thank you both for your input.  I finally got this to work earlier in the week but did not have a chance to post.

Here's what I changed on puppetmaster's `/etc/puppetlabs/puppet/puppet.conf` :

[main]

environmentpath = /etc/puppetlabs/code/environments

node_terminus = exec

external_nodes = /etc/puppetlabs/code/environments/node.rb

[master]

vardir = /opt/puppetlabs/server/data/puppetserver

logdir = /var/log/puppetlabs/puppetserver

rundir = /var/run/puppetlabs/puppetserver

pidfile = /var/run/puppetlabs/puppetserver/puppetserver.pid

codedir = /etc/puppetlabs/code

server = puppetmaster.domain.ca

storeconfigs = true

storeconfigs_backend = puppetdb

node_terminus = exec

external_nodes = /etc/puppetlabs/code/environments/node.rb

I essentially, removed basemodulepath as this was what was causing my suspicion (obvioulsy the wrong `modules` path was being loaded and this had to do with order of file loading).
In the `ops` environment's environment.conf file, I did the same in the hopes that I was forcing the default environmental paths only:

modulepath = ./modules:./classes

manifest = ./manifests

So now, my puppet runs do exactly as I expect for both `production` and `ops` (pasting ops node output here) :

[ro...@node.domain.ca ~]# puppet agent -o -v --no-daemonize

Info: Using configured environment 'ops'

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Loading facts

Info: Caching catalog for node.domain.ca

Info: Applying configuration version '1488089754'

Notice: Applied catalog in 23.46 seconds

This is great, now. Thank you both!

Of course, if there are any recommendations on basemodulepath management or if anyone thinks this should be overridden in a different way, I'm open to suggestions.