Puppet uses catalog of node B for node A

[Helmut Rickel]

Hello,

I installed puppet 2.6.17, 3.4.3 and finally 3.6.2 on SLES 11 SP1 or SLES 11 SP3 and always ran in this problem: Puppet is using the wrong catalog on some nodes.

I can run „puppet agent –t“ many times on node A without problems. When I run „puppet agent –t“ on node B and afterwards on node A, puppet uses the catalog of node B for both, node B and node A!

My final tests where done with these packages on the only master with SLES 11 SP3:

facter-2.0.0-11.1.x86_64.rpm
puppet-3.6.2-2.2.x86_64.rpm
puppet-server-3.6.2-2.2.x86_64.rpm
ruby-1.8.7.p357-3.7.x86_64.rpm
ruby-devel-1.8.7.p357-4.1.x86_64.rpm
rubygem-hiera-1.2.1-21.3.x86_64.rpm
rubygem-json_pure-1_6-1.6.7-1.19.x86_64.rpm
rubygem-ruby-shadow-2.2.0-11.4.x86_64.rpm
rubygems-1.8.15-27.9.x86_64.rpm

Actually, the first 10 characters of the fqdn's are identical. Can this cause the problem?

The clients (SLES 11 SP1) use the same packages. I used „node“ definitions as well as hiera to assign my classes to the nodes. No configurations concerning environments are done.

Thanks in advance for any help on this!

Kind regards

Helmut

[jcbollinger]

How are you determining that node A is getting node B's catalog?  There's a subtle but important difference between getting a complete catalog compiled for a different node and getting a catalog compiled for the requesting node, but unexpectedly influenced by prior compilation of a different node's catalog (e.g. including a resource intended only for a different node).  You can distinguish by making the agent report on the details presented to the master, for instance by putting this at top scope in your site.pp:

notify { 'Node ID':
  message => "My certificate identifies me as '${trusted['certname']}'; I identify myself as '${::clientcert}', or alternatively as '${::hostname}'"
}


Considering the range of Puppet versions affected, and the fact that this is not an issue I recognize, I am inclined to suspect a problem with your manifests.

In particular, I am inclined to guess that one or more of your manifests, other than site.pp or manifests 'imported' into it, has some top-scope declarations that are neither class nor type definitions.  Files do not serve as a scope boundary in Puppet, so such declarations apply in principle to all nodes.  They are only seen if the master has reason to parse the files containing them, however, which it may not have for some nodes.  Once such declarations are seen, however, the master may cache them, and apply them to other nodes (as is appropriate).

Example:

modules/site/manifests/alice.pp:
----
class site::alice {
  @user { 'alice': ensure => 'present' }
}

include 'restaurant'
----

The 'include' statement appearing at top scope there is a major problem.  It may be the author's intent that only nodes that have class 'site::alice' get class 'restaurant' as well, but the declaration is at top scope, so what it actually says that every node should get 'restaurant'.  Those that declare 'site::alice' certainly will get it, but others may or may not get it, depending on timing details, cache settings, and the phase of the moon.


John

[Helmut Rickel]

Hello John,

I saw that a class was installed on node A that only node B was supposed to use.

My holidays started right today so I will check this in detail right afterwards and give you a reply.

Thanks for your help!
Helmut

[Helmut Rickel]

Hello John,
I made the situation as simple as possible and - unfortunatly - I still can reproduce the problem.

I have two nodes called BGHW-DS-T-AA1 and BGHW-DS-T-SA1 and two modules called java and tomcat. The only task of the modules is to copy one file to the node. The module java is used on both nodes while tomcat is needed on BGHW-DS-T-SA1 only. This is defined via hiera. I append the files at the end of this posting.

After restarting the puppet master I call "puppet agent -t" on BGHW-DS-T-AA1 in a while-loop and the module "java" is applied. After some time I call "puppet agent -t" on BGHW-DS-T-SA1 and afterwards the module "tomcat" is applied on BGHW-DS-T-AA1 as well!

This is the shell on BGHW-P-AA1 (the important line is the red one):
BGHW-DS-T-AA1:/data/software # while true; do date ; puppet agent -t; sleep 1; echo ; done
Fr 8. Aug 14:22:11 CEST 2014
Info: Retrieving plugin
Info: Caching catalog for bghw-ds-t-aa1.bghw.de
Info: Applying configuration version '1407500588'
Notice: My certificate identifies me as 'bghw-ds-t-aa1.bghw.de'; I identify myself as 'bghw-ds-t-aa1.bghw.de', or alternatively as 'BGHW-DS-T-AA1'
Notice: /Stage[main]/Main/Notify[Node ID]/message: defined 'message' as 'My certificate identifies me as 'bghw-ds-t-aa1.bghw.de'; I identify myself as 'bghw-ds-t-aa1.bghw.de', or alternatively as 'BGHW-DS-T-AA1''
Notice: Finished catalog run in 2.20 seconds

Fr 8. Aug 14:22:17 CEST 2014
Info: Retrieving plugin
Info: Caching catalog for bghw-ds-t-aa1.bghw.de
Info: Applying configuration version '1407500588'
Notice: My certificate identifies me as 'bghw-ds-t-aa1.bghw.de'; I identify myself as 'bghw-ds-t-aa1.bghw.de', or alternatively as 'BGHW-DS-T-AA1'
Notice: /Stage[main]/Main/Notify[Node ID]/message: defined 'message' as 'My certificate identifies me as 'bghw-ds-t-aa1.bghw.de'; I identify myself as 'bghw-ds-t-aa1.bghw.de', or alternatively as 'BGHW-DS-T-AA1''
Notice: Finished catalog run in 2.13 seconds

Fr 8. Aug 14:22:25 CEST 2014
Info: Retrieving plugin
Info: Caching catalog for bghw-ds-t-aa1.bghw.de
Info: Applying configuration version '1407500588'
Notice: My certificate identifies me as 'bghw-ds-t-aa1.bghw.de'; I identify myself as 'bghw-ds-t-aa1.bghw.de', or alternatively as 'BGHW-DS-T-AA1'
Notice: /Stage[main]/Main/Notify[Node ID]/message: defined 'message' as 'My certificate identifies me as 'bghw-ds-t-aa1.bghw.de'; I identify myself as 'bghw-ds-t-aa1.bghw.de', or alternatively as 'BGHW-DS-T-AA1''
Notice: Finished catalog run in 2.24 seconds

Fr 8. Aug 14:22:32 CEST 2014
Info: Retrieving plugin
Info: Caching catalog for bghw-ds-t-aa1.bghw.de
Info: Applying configuration version '1407500588'
Notice: My certificate identifies me as 'bghw-ds-t-aa1.bghw.de'; I identify myself as 'bghw-ds-t-aa1.bghw.de', or alternatively as 'BGHW-DS-T-AA1'
Notice: /Stage[main]/Main/Notify[Node ID]/message: defined 'message' as 'My certificate identifies me as 'bghw-ds-t-aa1.bghw.de'; I identify myself as 'bghw-ds-t-aa1.bghw.de', or alternatively as 'BGHW-DS-T-AA1''
Notice: /Stage[main]/Tomcat/File[/data/software/apache-tomcat-7.0.37.tar.gz]/ensure: defined content as '{md5}a3cfe7ea8e8d2155cb2a0016e10b615f'
Notice: Finished catalog run in 4.97 seconds

Fr 8. Aug 14:22:43 CEST 2014
Info: Retrieving plugin
Info: Caching catalog for bghw-ds-t-aa1.bghw.de
Info: Applying configuration version '1407500588'
Notice: My certificate identifies me as 'bghw-ds-t-aa1.bghw.de'; I identify myself as 'bghw-ds-t-aa1.bghw.de', or alternatively as 'BGHW-DS-T-AA1'
Notice: /Stage[main]/Main/Notify[Node ID]/message: defined 'message' as 'My certificate identifies me as 'bghw-ds-t-aa1.bghw.de'; I identify myself as 'bghw-ds-t-aa1.bghw.de', or alternatively as 'BGHW-DS-T-AA1''
Notice: Finished catalog run in 2.56 seconds

On BGHW-DS-T-SA1 "puppet agent -t" was startet at 14:22:29
BGHW-DS-T-SA1:/data/software # date ; puppet agent -t
Fr 8. Aug 14:22:29 CEST 2014
Info: Retrieving plugin
Info: Caching catalog for bghw-ds-t-sa1.bghw.de
Info: Applying configuration version '1407500588'
Notice: My certificate identifies me as 'bghw-ds-t-sa1.bghw.de'; I identify myself as 'bghw-ds-t-sa1.bghw.de', or alternatively as 'BGHW-DS-T-SA1'
Notice: /Stage[main]/Main/Notify[Node ID]/message: defined 'message' as 'My certificate identifies me as 'bghw-ds-t-sa1.bghw.de'; I identify myself as 'bghw-ds-t-sa1.bghw.de', or alternatively as 'BGHW-DS-T-SA1''
Notice: Finished catalog run in 2.86 seconds

I reduced the number of files in /etc/puppet on the master as far as possible:
puppet@BGHW-P-SV1:/etc/puppet> find . -type f
./manifests/site.pp
./auth.conf
./hiera/node/BGHW-DS-T-SA1.bghw.de.yaml
./hiera/node/BGHW-DS-T-AA1.bghw.de.yaml
./modules/tomcat/manifests/init.pp
./modules/tomcat/files/apache-tomcat-7.0.37.tar.gz
./modules/java/manifests/init.pp
./modules/java/files/jdk-7u11-linux-x64.tar.gz
./puppet.conf
./hiera.yaml

Here you find the important files:
puppet@BGHW-P-SV1:/etc/puppet> cat hiera.yaml
:backends:
 - yaml
:yaml:
  :datadir: /etc/puppet/hiera
:hierarchy:
 - "node/%{::fqdn}"
puppet@BGHW-P-SV1:/etc/puppet> cat hiera/node/BGHW-DS-T-AA1.bghw.de.yaml
---
classes:
  - java


puppet@BGHW-P-SV1:/etc/puppet> cat hiera/node/BGHW-DS-T-SA1.bghw.de.yaml
---
classes:
  - java
  - tomcat

puppet@BGHW-P-SV1:/etc/puppet> cat modules/java/manifests/init.pp
class java () {
        file { "/data/software/jdk-7u11-linux-x64.tar.gz":
                ensure => present,
                owner  => uvdsadmin,
                group  => uvds-group,
                mode   => 644,
                source => "puppet:///modules/java/jdk-7u11-linux-x64.tar.gz",
        }
}

include java

puppet@BGHW-P-SV1:/etc/puppet> cat modules/tomcat/manifests/init.pp
class tomcat () {
        file { "/data/software/apache-tomcat-7.0.37.tar.gz":
                ensure => present,
                owner  => uvdsadmin,
                group  => uvds-group,
                mode   => 644,
                source => "puppet:///modules/tomcat/apache-tomcat-7.0.37.tar.gz",
        }
}

include tomcat

puppet@BGHW-P-SV1:/etc/puppet> cat manifests/site.pp
# File site.pp

notify { 'Node ID':
  message => "My certificate identifies me as '${trusted['certname']}'; I identify myself as '${::clientcert}', or alternatively as '${::hostname}'"
}

node default {
        hiera_include('classes')
}
puppet@BGHW-P-SV1:/etc/puppet>

As far as I can see, the identifications of my nodes are ok and the manifests of the modules look fine, too. I can stop the master giving "tomcat" to BGHW-DS-T-AA1 by restarting the puppet master service on the puppet master. The behaviour somehow corresponds to your information as the master has to read tomcat/manifests/init.pp when building the catalog for BGHW-DS-T-SA1. But this should have no effect on BGHW-DS-T-AA1, should it?

Thanks for any help on this!
This problem is severe for us as we cannot use puppet before it is solved.

[José Luis Ledesma]

Hi,

If I understand correctly you have an include java|tomcat inside the init.pp of each class definition?

I don't really know what is happening but I bet that if you remove these lines it will start to work correctly.

Regards,

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/9bfe7e99-50a5-4d24-8f62-2d2e787b2668%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Helmut Rickel]

Hello Jose,

thanks a lot for this hint. It solves my problem!

I am highly astonished as I got the idea for placing the include-statements in init.pp from https://docs.puppetlabs.com/learning/modules1.html .

Right now, I do not understand why the include-Statements are not necessary here and why they even mess things up. I will add a comment here as soon as I know the answers.

Kind regards
Helmut

[Helmut Rickel]

Hello Jose,

I have to correct myself: In https://docs.puppetlabs.com/learning/modules1.html it is not suggested to use "include" in init.pp, but for an example "include" is used in the same file as the class definition.

However, your hint is correct: include should not be used in init.pp. Otherwise the "included" class may be used for nodes do not need it according to "hiera" oder "node"-statements. This looks a bit like a bug for me, but I do not have enough experience with puppet to judge this.

Thanks again!
Helmut

[jcbollinger]

Hello Helmut,

I am glad you have resolved your issue.

I have recently been travelling, and hence not responsive to this list, but I feel compelled to observe that the problem turned out to be exactly what I said in my initial response it might be: "In particular, I am inclined to guess that one or more of your manifests [...] has some top-scope declarations that are neither class nor type definitions."  That response goes on to explain why having such declarations is a problem.

I can also be more specific now, since it seems you may be tripping over the semantics of the 'include' statement.  It is essential to understand that 'include' directs Puppet that the designated class must be included in the catalog being constructed -- NOT to interpolate any additional DSL code into the manifest file in which it appears.


John

[Helmut Rickel]

Hello John,
from a puppet-beginners point of view it seems astonishing that running "puppet agent -t" on node A can have influence on the catalog prepared for node B.

I agree that your inital response perfectly fitted to the solution of the problem. Thanks for that explanation!

Kind regards
Helmut