Small subset of agents not copying/running module's custom fact

[Samir Cury]

I'm having a pretty weird problem with a custom fact inside a module I wrote.

I would understand this problem better if it affected all clients (of same version) equally but it doesn't.

A long time after changes in the master (24h+), to make sure there are no caches, after several "puppet agent -t" runs, in short this is what I get :

[root@host1 ~]# find /var/lib/puppet/ -name discover-hdfs-partitions.* 

/var/lib/puppet/facts.d/discover-hdfs-partitions.sh 

[root@host2 ~]# find /var/lib/puppet/ -name discover-hdfs-partitions.* 

[root@host2 ~]#


As it helps debugging, here's the first puppet run on host2 (after completely wiping the agent, reinstalling RPM, revoking certificate and starting from scratch) :

http://pastebin.com/Eu46A5RK

As you see the fact file is pretty much ignored and not copied.

Both agents have the same exact version :

puppet-3.7.3-1.el6.noarch

And the master is the same. (very close, if not the same 3.7 release)

Any clues on how to debug this further? I have close to 300 nodes and most of them are fine. Few are presenting this but is enough to create the mess.

I also tried to workaround this by copying the file by hand to the right place, but even after that the fact won't appear. If I run the script by hand, it works : 

[root@host3 ~]# /var/lib/puppet/facts.d/discover-hdfs-partitions.sh
hdfs_partitions=/data1/hadoop/data,/data2/hadoop/data,/data3/hadoop/data 

So for some reason this fact is ignored. It looks like that facter -p won't show facts that are defined by modules, so any suggestion on how to debug this is welcome.

Thanks,

Samir

====================================================

For completeness, this is where the custom fact sits on the module :

modules/hdfs/facts.d/discover-hdfs-partitions.sh

And healthy nodes will use the fact for a notify, among other things :

Notice: /Stage[main]/Hdfs::Datanode/Notify[Found the following HDFS data mountpoints: /data1/hadoop/data,/data2/hadoop/data]/message: defined 'message' as 'Found the following HDFS data mountpoints: /data1/hadoop/data,/data2/hadoop/data'

[jcbollinger]

On Friday, January 23, 2015 at 2:23:34 PM UTC-6, Samir Cury wrote:

I'm having a pretty weird problem with a custom fact inside a module I wrote.

I would understand this problem better if it affected all clients (of same version) equally but it doesn't.

A long time after changes in the master (24h+), to make sure there are no caches, after several "puppet agent -t" runs, in short this is what I get :

[root@host1 ~]# find /var/lib/puppet/ -name discover-hdfs-partitions.* 

/var/lib/puppet/facts.d/discover-hdfs-partitions.sh 

[root@host2 ~]# find /var/lib/puppet/ -name discover-hdfs-partitions.* 

[root@host2 ~]#


As it helps debugging, here's the first puppet run on host2 (after completely wiping the agent, reinstalling RPM, revoking certificate and starting from scratch) :

http://pastebin.com/Eu46A5RK

As you see the fact file is pretty much ignored and not copied.

If two agents being served by the same master are working normally but do not receive the same auto-synced plugins (including custom facts), then the only likely explanation is that they are being assigned to different environments, one of which does not provide the fact in question.  This is especially likely if other plugins are synced as expected.  Plugin sync is not governed by node identity, except inasmuch as node identity affects environment assignment.


John