troubleshoot classes to nodes assignments

[puppetrier]

I am trying to assign classes to node either through site.pp or a custom ENC and I am getting nowhere.
The nodes successfully receive empty catalogs from the master.
No signs of errors that I can see in any of the log files - the system just silently fails to apply my settings.

For ENC, I followed the instructions from here:
https://docs.puppetlabs.com/guides/external_nodes.html

I have made the following changes in : /etc/puppetlabs/puppet/puppet.conf
[master]
   node_terminus = exec
   external_nodes = /etc/puppetlabs/puppet/deployment-config/scripts/enc-classify.pl

however after restarting the puppet services the master restores node_terminus to classifier

[master]
   node_terminus = classifier

Adding nodes definitions in site.pp are completely ignored as well.

Is there a way to turn up the verbose level and to follow how the master processes catalog requests from individual nodes?

Thank you

[jcbollinger]

On Monday, January 12, 2015 at 4:49:50 PM UTC-6, puppetrier wrote:

I am trying to assign classes to node either through site.pp or a custom ENC and I am getting nowhere.
The nodes successfully receive empty catalogs from the master.

I suggest leaving ENCs out of the picture for the time being.  For the moment, focus on classifying nodes via your site manifest.  Fewer moving parts are involved.

 

No signs of errors that I can see in any of the log files - the system just silently fails to apply my settings.

[...]
 

Is there a way to turn up the verbose level and to follow how the master processes catalog requests from individual nodes?

You should be able to start the master with --debug logging enabled to get more output on that side.

Since your nodes receive empty catalogs instead of no catalogs at all, you are already over the first hump.  The master acknowledges your nodes as valid, known clients.  If it is not assigning any classes or resources to them then that's because it doesn't think any have been matched to those node.  The most likely possibilities are these:

• You are attempting to classify your nodes via a site manifest for a different environment than the one tow which your nodes are assigned (or via a manifest that is not actually a site manifest at all, name notwithstanding).
• Your master is using a cached copy of the nodes' environment
• You disagree with Puppet with respect to which node identifier to use to classify nodes.  Puppet the master uses the name specified on the node's SSL certificate (accessible also as $client_cert).  The agent chooses the node's fqdn as its certname by default, but that can be altered.

Things to try first off:

1. Create a "default" node block in your site manifest, and put a notify resource in it: notify{"I am ${client_cert}": }.  Delete or comment out all other node blocks, so that the default block will be matched to every node.  This can be used to verify that you are working with the right site manifest.
2. After changing your manifests, restart the puppetmaster service to ensure that the environment cache is cleared.

If your nodes do not receive the Notify resource in their catalogs after you do both of those, then you are working with the wrong site manifest.  If they do receive the resource, then the message it produces in the log provides an identifier you can use as a node name to classify that node.  (For FQDNs, Puppet also allows the short form of that name as a node name.)

Beyond that, it's a question of declaring the classes you want assigned.  In particular, be aware that installing Puppet modules in the master does not automatically cause them to be applied to your nodes (that would make it impossible to give different nodes different configuration).  Your node blocks (and/or ENC) must specify to Puppet which classes should be applied to each node -- that's the essence of "classification".  In its simplest form, you do that via 'include' statements in your node blocks.

Anyway, this is already starting to go speculative.  Let us know what happens when you try the above suggestions.


John

[jcbollinger]

Man, so many typos today :(.

But also this error:

1. Create a "default" node block in your site manifest, and put a notify resource in it: notify{"I am ${client_cert}": }.  Delete or comment out all other node blocks, so that the default block will be matched to every node.  This can be used to verify that you are working with the right site manifest.

That notify should be: notify{"I am ${clientcert}": }

John

[puppetrier]

Thank you. The --debug option looks very useful.

Indeed there seems to be a problem with the environment config folder location.
I assume the path in the "Did not match path" line bellow is not actually referring to an absolute path... 
Anyway, the class mapping still doesn't work. I made the change in all site.pp files for all environments then restarted the master and nothing happens on the node.

Here is the output on the master when I issue a "puppet agent --test" in the node.
 

Debug: Routes Registered:
Debug: Route /^\/v2\.0/
Debug: Route /.*/
Debug: Evaluating match for Route /^\/v2\.0/
Debug: Did not match path ("/production/catalog/dev-testbox-01.mydomain.com")
Debug: Evaluating match for Route /.*/
Debug: Creating new connection for https://puppet.internal.mydomain.com:8081
Info: 'replace facts' command for dev-testbox-01.mydomain.com submitted to PuppetDB with UUID 6f7eb279-45e4-42e0-9605-246950605d39
Debug: Using cached facts for dev-testbox-01.mydomain.com
Debug: Creating new connection for https://puppet.internal.mydomain.com:4433
Debug: Using cached facts for dev-testbox-01.mydomain.com
Info: Caching node for dev-testbox-01.mydomain.com
Notice: Compiled catalog for dev-testbox-01.mydomain.com in environment production in 0.05 seconds
Info: Caching catalog for dev-testbox-01.mydomain.com
Debug: Creating new connection for https://puppet.internal.mydomain.com:8081
Info: 'replace catalog' command for dev-testbox-01.mydomain.com submitted to PuppetDB with UUID 9c4c0794-8321-4ac3-8815-cbb43a1f086c
Debug: Puppet::Network::Format[msgpack]: feature msgpack is missing
Debug: catalog supports formats: pson b64_zlib_yaml yaml dot raw
Debug: Routes Registered:
Debug: Route /^\/v2\.0/
Debug: Route /.*/
Debug: Evaluating match for Route /^\/v2\.0/
Debug: Did not match path ("/production/report/dev-testbox-01.mydomain.com")
Debug: Evaluating match for Route /.*/
Debug: Received report to process from dev-testbox-01.mydomain.com
Debug: Processing report from dev-testbox-01.mydomain.com with processor Puppet::Reports::Console
Debug: Creating new connection for https://puppet.internal.mydomain.com:4435
Debug: Processing report from dev-testbox-01.mydomain.com with processor Puppet::Reports::Puppetdb
Debug: Creating new connection for https://puppet.internal.mydomain.com:8081
Info: 'store report' command for dev-testbox-01.mydomain.com submitted to PuppetDB with UUID e82937d8-fffa-48a8-9591-fad84b36da10


Thank you,

[puppetrier]

Some success: although I haven't solved the problem with site.pp (probably due to hiera configuring environment folders - that I still can't troubleshoot...) things start working with a custom ENC.
One of the reasons why ENC didn't work the first time was a class in default master modules that was always setting the node_terminus to classifier.
/opt/puppet/share/puppet/modules/puppet_enterprise/manifests/profile/master/classifier.pp

Changing node_terminus in puppet.conf to "exec" and restart puppetserver had no effect as node_terminus was always changed back to classifier. Here is the modified classifier.pp
 
  pe_ini_setting { 'node_terminus' :
    ensure  => present,
    path    => "${confdir}/puppet.conf",
    section => 'master',
    setting => 'node_terminus',
    value   => 'exec',  # changed from classifier
  }

Note: I also changed:
/opt/puppet/share/puppet/modules/puppet_enterprise/spec/classes/profile/master/classifier_spec.rb
from:
 it { should contain_pe_ini_setting('node_terminus').with_value('classifier') }
to:
 it { should contain_pe_ini_setting('node_terminus').with_value('exec') }

Anyway, it looks like I can move on for now, but I have to admit that after spending quite a few hours on these issues I am a bit disappointed with puppetlabs docs.
Why doesn't each chapter on "configuration" have a troubleshooting section with a minimum: "if things don't work out...", "relevant log files", "how to turn on debugging" ?

Thanks.

[jcbollinger]

On Wednesday, January 14, 2015 at 7:10:01 AM UTC-6, puppetrier wrote:

Some success: although I haven't solved the problem with site.pp (probably due to hiera configuring environment folders - that I still can't troubleshoot...) things start working with a custom ENC.
One of the reasons why ENC didn't work the first time was a class in default master modules that was always setting the node_terminus to classifier.
/opt/puppet/share/puppet/modules/puppet_enterprise/manifests/profile/master/classifier.pp

Changing node_terminus in puppet.conf to "exec" and restart puppetserver had no effect as node_terminus was always changed back to classifier. Here is the modified classifier.pp
 
  pe_ini_setting { 'node_terminus' :
    ensure  => present,
    path    => "${confdir}/puppet.conf",
    section => 'master',
    setting => 'node_terminus',
    value   => 'exec',  # changed from classifier
  }

Note: I also changed:
/opt/puppet/share/puppet/modules/puppet_enterprise/spec/classes/profile/master/classifier_spec.rb
from:
 it { should contain_pe_ini_setting('node_terminus').with_value('classifier') }
to:
 it { should contain_pe_ini_setting('node_terminus').with_value('exec') }

Anyway, it looks like I can move on for now, but I have to admit that after spending quite a few hours on these issues I am a bit disappointed with puppetlabs docs.
Why doesn't each chapter on "configuration" have a troubleshooting section with a minimum: "if things don't work out...", "relevant log files", "how to turn on debugging" ?

Thanks for the update, and I'm glad you have had some success.  I hope you indeed can move on, but if you cannot get your site manifest to work then I anticipate other problems arising.

If you are dissatisfied with the docs then I encourage you to file an issue against them in PL's issue tracker.  Overall I find their docs pretty good, but if they didn't work out for you, for this purpose then I'm sure they would appreciate hearing about it.


John