new puppet client not updating

[JGonza1]

Added new clients that are behind a firewall and none of them are updating with the configurations from the puppet master. I get the message below from the client when I run command puppet agent --server ct-eng-pup.caretools.ent --test. What ports need to be opened for client to talk to puppet master. The puppet client had no problem get aa certificate from the puppet client but it will not update.

 

info: Caching catalog for pdlnx-pntp02.kareoprod.ent
info: Applying configuration version '1351815499'
notice: Finished catalog run in 0.10 seconds
#

 

On the puppet master on the /var/lib/puppet/reports/pdlnx-pntp02.kareoprod.ent the transaction report I am getting a lot of failed, see log below.

--- !ruby/object:Puppet::Transaction::Report
  configuration_version: 1351815499
  environment: production
  host: pdlnx-pntp02.kareoprod.ent
  kind: apply
  logs:
    - !ruby/object:Puppet::Util::Log
      level: !ruby/sym info
      message: Caching catalog for pdlnx-pntp02.kareoprod.ent
      source: Puppet
      tags:
        - info
      time: 2012-11-13 12:23:01.849547 -08:00
    - !ruby/object:Puppet::Util::Log
      level: !ruby/sym info
      message: Applying configuration version '1351815499'
      source: Puppet
      tags:
        - info
      time: 2012-11-13 12:23:01.886192 -08:00
    - !ruby/object:Puppet::Util::Log
      level: !ruby/sym notice
      message: Finished catalog run in 0.10 seconds
      source: Puppet
      tags:
        - notice
      time: 2012-11-13 12:23:01.987541 -08:00
  metrics:
    events: !ruby/object:Puppet::Util::Metric
      label: Events
      name: events
      values:
        - - success
          - Success
          - 0
        - - total
          - Total
          - 0
        - - failure
          - Failure
          - 0
    time: !ruby/object:Puppet::Util::Metric
      label: Time
      name: time
      values:
        - - total
          - Total
          - 0.221700026885986
        - - config_retrieval
          - Config retrieval
          - 0.221333026885986
        - - filebucket
          - Filebucket
          - 0.000367
    resources: !ruby/object:Puppet::Util::Metric
      label: Resources
      name: resources
      values:
        - - changed
          - Changed
          - 0
        - - failed_to_restart
          - Failed to restart
          - 0
        - - restarted
          - Restarted
          - 0
        - - total
          - Total
          - 7
        - - out_of_sync
          - Out of sync
          - 0
        - - failed
          - Failed
          - 0
        - - skipped
          - Skipped
          - 6
        - - scheduled
          - Scheduled
          - 0
    changes: !ruby/object:Puppet::Util::Metric
      label: Changes
      name: changes
      values:
        - - total
          - Total
          - 0
  puppet_version: 2.7.19
  report_format: 2
  resource_statuses:
    "Filebucket[puppet]": !ruby/object:Puppet::Resource::Status
      change_count: 0
      changed: false
      evaluation_time: 0.000367
      events: []
      failed: false
      file:
      line:
      out_of_sync: false
      out_of_sync_count: 0
      resource: "Filebucket[puppet]"
      resource_type: Filebucket
      skipped: false
      tags:
        - filebucket
        - puppet
      time: 2012-11-13 12:23:01.972617 -08:00
      title: puppet
    "Schedule[weekly]": !ruby/object:Puppet::Resource::Status
      change_count: 0
      changed: false
      events: []
      failed: false

[jcbollinger]

On Tuesday, November 13, 2012 2:46:09 PM UTC-6, JGonza1 wrote:

Added new clients that are behind a firewall and none of them are updating with the configurations from the puppet master. I get the message below from the client when I run command puppet agent --server ct-eng-pup.caretools.ent --test. What ports need to be opened for client to talk to puppet master. The puppet client had no problem get aa certificate from the puppet client but it will not update.

 

info: Caching catalog for pdlnx-pntp02.kareoprod.ent
info: Applying configuration version '1351815499'
notice: Finished catalog run in 0.10 seconds
#

 

Those messages do not reflect any kind of problem with contacting the master (as would be expected since the client was able to have its certificate signed).  Instead, they suggest that the master is delivering an empty catalog.  That would arise from a problem with your Puppet manifests, not with your firewall.

 

I think you are misreading that.  As far as I can tell, it explicitly reports that there were zero failures in each of several categories it covers.  Furthermore, if you are getting reports from the agent then that is additional evidence that communication between the agent and master is working fine.  You do not have a firewall problem here.

Most likely the certnames (== hostnames by default) presented by the affected nodes do not match any node blocks in your top-level manifests.  The master's log should show more clearly and succinctly which nodes are requesting catalogs, including, I think, their certnames.  Compare those to your node blocks, and / or create a 'default' node to support otherwise-unmatched nodes.


John

[JGonza1]

The new server do belong to a different dns domain than the the orginal servers. The orginal servers belong to .caretools.ent and the new ones belong to .kareoprod.ent. I do have manifests for the pdlnx-pntp02.kareprod.ent, see below. What master log should I look at to see mismatches.Sorry new to puppet.

node "pdlnx-pntp02.kareoprod.ent"
     inherits default {
     include hosts
     include users
     include resolv_conf
     include tomcat-prod
     include ptnrjava
     include ntp
     include ptnrlocalhost
     include setjava
     include ptnrj2ee
     include webmanager
     include ptnrlinks
     include proxybase
     include webapps
     include proxyfiles
     include proxylib
     include ptnrcache
     include setclassfile
     include ptnrpartner
     include ptnriptables
     include sshkey
     include sendmail

[jcbollinger]

On Wednesday, November 14, 2012 9:38:05 PM UTC-6, JGonza1 wrote:

The new server do belong to a different dns domain than the the orginal servers. The orginal servers belong to .caretools.ent and the new ones belong to .kareoprod.ent. I do have manifests for the pdlnx-pntp02.kareprod.ent, see below. What master log should I look at to see mismatches.

Where the log messages would appear is configurable, but you should look first in the main system log, probably /var/log/messages.  It should show messages similar to

  Nov 11 04:37:02 <mastername> puppetmasterd[<pid>]: Compiled catalog for <certname> in 0.10 seconds

It will display more information if the master runs with the --debug switch enabled, so you might want to turn that on while you troubleshoot.

 

Sorry new to puppet.

node "pdlnx-pntp02.kareoprod.ent"
     inherits default {
     include hosts

[...]
}

And does that node block appear in the same file as node blocks for machines that get non-trivial catalogs from the master?

Or along a different line, are you sure your clients are contacting the right master?  If you have not put the correct master's name in the nodes' config files, and/or if your DNS does not point the default (domain-dependent) master name to the correct master, then you could be communicating with some other master, such as one in test environment somewhere.  That could happen accidentally if the master you are talking to has certificate auto-signing turned on.


John

[JGonza1]

On Thursday, November 15, 2012 7:36:46 AM UTC-8, jcbollinger wrote:

There is no other puppet Master in my company and I have defined the puppet master in the clients /etc/hosts files,see below.
#more /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
# Puppet server
10.23.40.78 ct-eng-pup.caretools.ent
#

[JGonza1]

I do see the that the client server pdlnx-pntp02.kareoprod.ent does have a certificate on the master under the directory /etc/puppet/ssl/ca/signed. Also the the puppet master I do not have auto-signing turned on. I have to run the puppet cert --sign <server name> command to create the certificate.

ll pdlnx-pntp02.kareoprod.ent.pem
-rw-r-----. 1 puppet puppet 1939 Nov 11 23:52 pdlnx-pntp02.kareoprod.ent.pem
#

On Tuesday, November 13, 2012 12:46:09 PM UTC-8, JGonza1 wrote:

[jcbollinger]

On Thursday, November 15, 2012 7:55:33 PM UTC-6, JGonza1 wrote:

I do see the that the client server pdlnx-pntp02.kareoprod.ent does have a certificate on the master under the directory /etc/puppet/ssl/ca/signed. Also the the puppet master I do not have auto-signing turned on. I have to run the puppet cert --sign <server name> command to create the certificate.

The question then remains why the master is not associating the new nodes with the node block(s) you created for them.  That was the thrust of the other question and suggestion in my previous response (about what file the node block appears in, and about running the master with the --debug switch enabled).  So?

Perhaps it will save some time if I point out that there is only one manifest that Puppet reads automatically.  Its name and location are configurable, but by default it is manifests/site.pp under your Puppet installation directory (often /etc/puppet).  Your node blocks should be either in that manifest or in a manifest 'import'ed directly or indirectly by it (and that is the only good use I know for the 'import' function).

Furthermore, the puppet master process must be able to read the file containing the node block.  That process normally runs without privilege, so you must ensure that ownership and permissions (and any other access controls) on that file and every directory in the path to it allow access to the master process.

If the master cannot or does not load the node block, then the declarations in it will not be applied.  If the master does load the node block but does not match it to your node, then either the node's certname is not what you think it is, or some other node block is matching at higher priority (see http://docs.puppetlabs.com/puppet/3/reference/lang_node_definitions.html#matching).


John

[jcbollinger]

On Friday, November 16, 2012 9:05:38 AM UTC-6, jcbollinger wrote:

[...] If the master does load the node block but does not match it to your node, then either the node's certname is not what you think it is, or [...].

One notable possibility along those lines is if your new nodes' ideas of their own hostnames are unqualified (e.g. "pdlnx-pntp02" instead of "pdlnx-pntp02.kareoprod.ent").  You should be able to check that via the 'hostname' command.  I think in that in choosing the hostname as certname by default, Puppet will in such cases use the unqualified name as certname.  That will not match the fully-qualified node name in your node declaration.

The reverse is ok, however: if your node's certname is a fully-qualified name and Puppet does not match the whole thing, then it will attempt to match the just the local part.


John