Puppet calls the ENC twice for some nodes.

[pierra mathieu]

Hi everyone, 

I have an issue setting up Puppet with an ENC.

For some nodes, puppet calls my ENC twice with a 2 sec interval.

All my agents share the same configuration file. 

Considering this very basic ENC : 

#!/bin/bash
nodeName=$1
echo `date` $nodeName >> /tmp/encCalls
echo "---
classes:
  testClass:"

Here is the content of /tmp/encCalls after a few run on two nodes.

Mon Sep  9 11:36:15 CEST 2013 : host1
Mon Sep  9 11:36:17 CEST 2013 : host1
Mon Sep  9 11:41:04 CEST 2013 : host2
Mon Sep  9 11:42:04 CEST 2013 : host1
Mon Sep  9 11:42:06 CEST 2013 : host1
Mon Sep  9 11:45:13 CEST 2013 : host2

Anybody knows why two calls are made ?

Thanks.

[Steven VanDevender]

pierra mathieu writes:
> Hi everyone,
>
> I have an issue setting up Puppet with an ENC.
> For some nodes, puppet calls my ENC twice with a 2 sec interval.

Why is that a problem? Your ENC should always return the same data for
the same node, and it should be efficient enough that it can be called
for every node in your system (even twice).

[jcbollinger]

My first guess would be that you have two copies of the agent running on some nodes.  If your manifest set is as simple as your ENC, then the Puppet agent could perform a full cycle fast enough that two separate catalog requests from the same node are received within 2 seconds.


John

[pierra mathieu]

Thanks for the answers and sorry for my late one, 

I've only one agent process running :

ps -aef | grep puppet
root      4080  2811  0 14:14 pts/0    00:00:00 grep puppet
root     18328     1     0 Sep12 ?          00:00:02 /usr/bin/ruby /usr/bin/puppet agent

And the agent cycle isn't really fast, something like 5-10 sec.

I was thinking about adding an acknowledgment the first time a class is requested by a node, so i'd like to choose which class to return for each request.

Moreover, I'm curious and this comportment from puppet seems a little bit weird/useless.

For information, here is my agent puppet.conf:

[main]
    logdir = /var/log/puppet
    rundir = /var/run/puppet
    ssldir = $vardir/ssl
    pluginsync = true
[agent]
    classfile = $vardir/classes.txt
    localconfig = $vardir/localconfig
    server=********
    report = true
    usecacheonfailure = false

Commenting pluginsync or report doesn't change anything.

Any other ideas ? 

Thanks.

[Greg Sutcliffe]

Is this puppet3? As I recall, in puppet3, the master makes a separate call to the enc to determine the environment the should authoritatively be in. Once that's established, it makes a second call to get the classes and parameters.

Hth,
Greg

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

[Greg Sutcliffe]

On 20 Sep 2013 18:05, "Greg Sutcliffe" <greg.su...@gmail.com> wrote:
> the should authoritatively be in.

Gah, phone keyboard. "The *client* should authoritatively..."

Sorry
Greg

[jcbollinger]

On Friday, September 20, 2013 12:05:17 PM UTC-5, Greg Sutcliffe wrote:

Is this puppet3? As I recall, in puppet3, the master makes a separate call to the enc to determine the environment the should authoritatively be in. Once that's established, it makes a second call to get the classes and parameters.

Not exactly, but that may well be the right track.  It would be pointless for the master to run the ENC more than once for catalog compilation, for it would have no reason to expect that the ENC's output would change.  HOWEVER, the master's file server may need to run the ENC again to determine the environment from which to serve 'source'd files.


John

[James Ellis]

Hi, chanced across this discussion when I noticed an ENC was being called twice. I understand I may not be using the ENC terminus exactly as it's been designed, but it's unexpected that it was called twice. Also worth noting that I can't see a note about the ENC being called twice here: http://docs.puppetlabs.com/guides/external_nodes.html

In my case, I'm using an ENC to push virtual host changes to an agent running a web server, the YAML returned by the ENC uses create_resources to dynamically add resources to the catalogue.

I observed via logging in the ENC script that on the first run, the ENC was excecuted but the catalogue was not applied, on the second run the catalogue was applied on the agent.

This causes problems where we use an API to dynamically apply resources to a catalogue (1st run gets the catalogue resources, returns 'OK' to the API, 2nd run then tries to get resources but gets nothing as the 'OK'  sent to the API has effectively modified the resources to be applied).

I've worked around this, for now, by using a lock file, so that the 'OK' API call is only run once but this still applies two calls to the API to dynamically get resources for the catalogue, where only one is required. I double checked the master and the agent configs, and the master only shows the ENC being referenced once and there is one  agent being run, only.

Based on this, is there any way the agent can be set to call the ENC once only ? The only argument to the script is the agent hostname and there is no apparent difference in the environment of the first and second ENC calls.

Using 3.4 O/S on ubuntu with the following agent command (run as root manually to debug) :

puppet agent --no-usecacheonfailure --onetime --no-daemonize --server valid.server --verbose

Thanks

James

[Jason Antman]

James,

I vaguely remember seeing this 'node_terminus called twice' thing in the past. The simple answer, though I know it's not what people want to hear, is that the ENC should always return the right information. If you want to modify the content of the catalog based on something that happens between runs (which, by the way, I would highly suggest against, and suggest that if you're doing that, something in your puppet configuration is amiss), you should be checking somewhere like PuppetDB, not changing it based on how many times the ENC script is run. Aside from the problem you're having now, what happens if there's a timeout, or some other failure after the ENC script is called but before the catalog is applied?

-Jason

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/373455ff-43f4-4f27-8667-ab896c240ea6%40googlegroups.com.

[jcbollinger]

On Tuesday, January 14, 2014 5:20:55 PM UTC-6, James Ellis wrote:

Hi, chanced across this discussion when I noticed an ENC was being called twice. I understand I may not be using the ENC terminus exactly as it's been designed, but it's unexpected that it was called twice. Also worth noting that I can't see a note about the ENC being called twice here: http://docs.puppetlabs.com/guides/external_nodes.html

True, but the docs also don't say it's called only once.  If you read between the lines, you might infer that it can be called multiple times.  In particular, consider this excerpt from the doc: "In Puppet 3 and later, ENCs can set an environment for a node, overriding whatever environment the node requested. However, previous versions of Puppet use ENC-set and node-set environments inconsistently, with the ENC’s used during catalog compilation and the node’s used when downloading files."  The reason for the inconsistency in Puppet 2 is that file downloads occur in separate request contexts, and that version of Puppet did not re-run the ENC for each of those as it would have needed to do to compute the appropriate environment correctly.

To be sure, the docs could be clearer on the point.  Feel free to file an RFE.

 

In my case, I'm using an ENC to push virtual host changes to an agent running a web server, the YAML returned by the ENC uses create_resources to dynamically add resources to the catalogue.

I observed via logging in the ENC script that on the first run, the ENC was excecuted but the catalogue was not applied, on the second run the catalogue was applied on the agent.

By default, the agent issues at least two web requests to the master on every run because it first synchronizes plugins, then requests a catalog.  It may make other requests as well, to retrieve plugin contents when it determines those are needed, or to retrieve 'source'd files.  All of those requests require the master to determine the node's environment to provide correct service, and determining the environment requires running the ENC, if there is one.

 

This causes problems where we use an API to dynamically apply resources to a catalogue (1st run gets the catalogue resources, returns 'OK' to the API, 2nd run then tries to get resources but gets nothing as the 'OK'  sent to the API has effectively modified the resources to be applied).

I've worked around this, for now, by using a lock file, so that the 'OK' API call is only run once but this still applies two calls to the API to dynamically get resources for the catalogue, where only one is required. I double checked the master and the agent configs, and the master only shows the ENC being referenced once and there is one  agent being run, only.

Based on this, is there any way the agent can be set to call the ENC once only ? The only argument to the script is the agent hostname and there is no apparent difference in the environment of the first and second ENC calls.

If you disable pluginsync at the agent and avoid declaring any 'source'd File resources pointing back to the master, then I think the agent will make only one request to the master per catalog run.  In that case, the master should run the ENC only once for that catalog run.

Disabling pluginsync may make it difficult to insert certain new modules into your Puppet infrastructure or to update such modules, and avoiding 'source'd Files will require great discipline and vigilance.  I would think twice before going that route.  Instead, if you want to stick with your current ENC concept then you should consider how your ENC might recognize multiple calls associated with the same catalog run.

I think node facts are updated only once per catalog run, after pluginsync and just prior to running the ENC to kick off catalog compilation.  Perhaps, then, you could watch each node's 'uptime_seconds' fact, and perform your once-per-catalog-run work only if the value of that fact is different from what it was during the previous run of the ENC for the given node.  The doc you referred to has a bit of information about how the ENC can access node facts.

John

[James Ellis]

Thanks Jason for the info.

Bear with me, but I'm still unsure why the ENC would run twice, particularly given the dynamic nature of an ENC terminus where it would be connecting to other systems to get information for a catalogue -- those systems are then hit twice, regardless of whether or not the same information is returned, which seems a waste of resources (e.g maybe the external system has a quota limit ?)

I see in our logs running twice when the agent, in verbose mode, outputs "Info: retrieving plugin" and then another run when it outputs " Info: Caching catalog for ..."

In relation to applying changes to a virtual hosting setup, I'll look at other options. Given a quick example of say a domain  'enc.example.com' being created, then the customer fails to pay for their domain or hosts elsewhere, such that it should be deleted, the ENC should then provide this account for deletion in the catalogue, so that it is removed on the relevant server. I understand that this is changing the catalogue information across ENC runs but how else would this change be provided to the agent ?

Regards

James