Nodes getting catalog with incorrect resource.

[Gino Lisignoli]

I'm having the strangest puppet problem I have ever seen:

I'm trying to get a nrpe module working on some nodes (Ubuntu 12.04 and 14.04), but both modules I have tried, seem to think the operating system is Redhat.

To explain this a little bit better:

- Facter on all these nodes reports that the osfamily is Debian, and the operatingsystem is Ubuntu, which is correct.

- When I check the reported facts for the nodes on the puppetserver they are Debian and Ubuntu

- When I look at the catalog that gets compiled I can see that all my other resources seem to be correct, we have some osfamily/system code that checks the os and sets up apt/yum repositories. So I know that's working correct.

- When I r10k my entire environment to the nodes and run it locally with 'puppet apply' there are no problems at all.

- Sometimes when I clear the environment cache, the problem will go away for the first puppet run, but all future puppet runs have the same problem.

- I've checked hiera, the only paramater for nrpe is to specify our ntp server

I'm running puppet 4.5.1, puppetsever 2.4.0 and puppetdb 4.1.0, we use Foreman to classify our nodes with no smart paramaters etc. The only classification it does is assign a role to a node.

My only ideas are:

- There's some some sort of catalog/resource cache that is getting generated with the wrong nrpe resource. AFAIK this doesn't exist? As only whole catalogs are cached on the puppetserver

- Somehow puppetdb might be invloved? I know puppetdb has resources/catalogs on it. Does puppetserver use puppetdb as a cache for resources/catalogs when generating a catalog for nodes? If so is there a way to debug why it would be generating a catalog with an incorrect resource for nodes?

Anyone have any ideas how to debug this?

[Luke Bigum]

Can you explain the symptoms of your problem a bit more, and link to the NRPE module you are using if it's open source?

You've described everything you've checked, but I still don't know exactly what's wrong / failing and what you are expecting to see :-)

Does the NRPE module collect exported resources? Could you be collecting Redhat resources onto your ubuntu machine?

There may be some level of catalog caching happening here, but to get just the operating system Facts confused and nothing else seems unlikely. You could try turn unlplug PuppetDB from your Master temporarily (storeconfigs = false?) and see what catalog gets compiled then. The Master should also have the latest Facts for every node on disk in YAML format here:

ls -ld /var/lib/puppet/yaml/facts/$(hostname).yaml

Assuming that's still the right path in Puppet 4.

--
Luke Bigum
Senior Systems Engineer

Information Systems

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/889f44ad-1ece-4dff-a769-b91c916d4bee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
---

LMAX Exchange, Yellow Building, 1A Nicholas Road, London W11 4AN
http://www.LMAX.com/

Recognised by the most prestigious business and technology awards

2016 Best Trading & Execution, HFM US Technology Awards
2016, 2015, 2014, 2013 Best FX Trading Venue - ECN/MTF, WSL Institutional Trading Awards

2015 Winner, Deloitte UK Technology Fast 50
2015, 2014, 2013, One of the UK's fastest growing technology firms, The Sunday Times Tech Track 100
2015 Winner, Deloitte EMEA Technology Fast 500
2015, 2014, 2013 Best Margin Sector Platform, Profit & Loss Readers' Choice Awards

---

FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved.

This message and its attachments are confidential, may not be disclosed or used by any person other than the addressee and are intended only for the named recipient(s). This message is not intended for any recipient(s) who based on their nationality, place of business, domicile or for any other reason, is/are subject to local laws or regulations which prohibit the provision of such products and services. This message is subject to the following terms (http://lmax.com/pdf/general-disclaimers.pdf), if you cannot access these, please notify us by replying to this email and we will send you the terms. If you are not the intended recipient, please notify the sender immediately and delete any copies of this message.

LMAX Exchange is the trading name of LMAX Limited. LMAX Limited operates a multilateral trading facility. LMAX Limited is authorised and regulated by the Financial Conduct Authority (firm registration number 509778) and is a company registered in England and Wales (number 6505809).

LMAX Hong Kong Limited is a wholly-owned subsidiary of LMAX Limited. LMAX Hong Kong is licensed by the Securities and Futures Commission in Hong Kong to conduct Type 3 (leveraged foreign exchange trading) regulated activity with CE Number BDV088.

[Gino Lisignoli]

NRPE Module: https://github.com/example42/puppet-nrpe

If you have a look at the params.pp file, lines 67-74 you will see a block of code that sets the value of a variable depending on the operating system version. In my environment my operating system is ubuntu, and I would expect the value of the $package variable to end up as: nagios-nrpe-server. But what's happening is the $package variable ends up as 'nrpe' which doesn't exist.

The NRPE Module doesn't do anything with exported resources

Anyway, running that code locally works fine. From a puppet master I get the problem described. Which is very strange. I also tried disabling our puppetdb server but still had the same problem.

So yesterday I managed to resolve the problem, the resolution is to disable puppet environment cache. 

I know the situation to make the problem happen (turn environment caching on), but I don't know why it is happening. When the environment cache is turned on I can clear the the cache using the puppet admin api, and then for one puppet run it will work fine, but (I'm assuming) after it is cached it is getting confused with other nodes generating/using the same cache but with different facts.

[jcbollinger]

On Monday, July 4, 2016 at 4:20:12 PM UTC-5, Gino Lisignoli wrote:

NRPE Module: https://github.com/example42/puppet-nrpe

If you have a look at the params.pp file, lines 67-74 you will see a block of code that sets the value of a variable depending on the operating system version. In my environment my operating system is ubuntu, and I would expect the value of the $package variable to end up as: nagios-nrpe-server. But what's happening is the $package variable ends up as 'nrpe' which doesn't exist.

The NRPE Module doesn't do anything with exported resources

Anyway, running that code locally works fine. From a puppet master I get the problem described. Which is very strange. I also tried disabling our puppetdb server but still had the same problem.

So yesterday I managed to resolve the problem, the resolution is to disable puppet environment cache. 

I know the situation to make the problem happen (turn environment caching on), but I don't know why it is happening. When the environment cache is turned on I can clear the the cache using the puppet admin api, and then for one puppet run it will work fine, but (I'm assuming) after it is cached it is getting confused with other nodes generating/using the same cache but with different facts.

This problem has the hallmarks of a top-scope resource declaration -- i.e. one outside any class, node block, or defined type.  These are allowed, and they are sometimes reasonable in a site manifest, but they should never appear in manifests that are not guaranteed always to be evaluated, such as those in modules.  The problem is that top-scope declarations anywhere in the manifest set apply in principle to all nodes, so if they appear in manifests that Puppet evaluates only conditionally, such as via the module autoloader, then the catalog builder gets an inconsistent view of the global declarations.

The usual symptom of inappropriate top-level declaration is that resources seem to leak from some catalogs to others.  It is also possible for resource parameters to leak under certain circumstances.  The fact that clearing the cache resolves the problem temporarily is a strong indication that your problem lies in this area.

If you do have a problem with top-level declarations, then it is probably in one of your own modules, not in the Example42 module.  Probably it's in a module that uses the example42 module, but it could be one that uses it indirectly.


John

[Gino Lisignoli]

Hi John

Thanks for the piece of information! I'm pretty sure (at least on the environment where the problem is happening) that everything is inside classes. I'll try and look through our custom modules to see if there are any resources outside of a class block.