Export variables from a manifest calling another module

[Brendon Standing]

Hi

I am busy writing puppet modules for logstash, and want to export tokens I get from the hiera application or %{calling_module} role.

So each of my applications/modules that will make use of the shipper module and have the inputs.conf file installed and populated will have the following:

class name::deployment inherits name::params {
include logstash::shipper
}

The token I want to export for each app/module is:
$logstash_logs = hiera('logstash_logs')

And this is taken from the %{calling_module} role like:
logstash_logs:
javabridge: 'javabridge-log.log'
actions_javabridge: 'ual/actions-javabridge.log'


The template that is deployed via logstash::shipper has:
<% @logstash_logs.each_pair do |key, value| -%>
  file {
    type => "<%= key %>"
    path => [ "<%= logdir %>/<%= value %>" ]
    tags => [ "<%= key %>" ]
  }
<% end -%>

Unfortunately the name:deployment class that is calling include logstash::shipper is not passing the tokens.

The error I get is:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template logstash/input.conf.erb:
  Filepath: /usr/lib/ruby/site_ruby/1.8/puppet/parser/templatewrapper.rb
  Line: 81
  Detail: Could not find value for 'logstash_logs' at /etc/puppet/environments/bstanding/modules/logstash/templates/input.conf.erb:1
 at /etc/puppet/environments/bstanding/modules/logstash/manifests/shipper.pp:15 on node free-4574-u37.dev.marinsw.net
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Any idea how best to achieve this, keeping in mind that I will need to export different options for each module / app I am deploying with the shipper?

Many thanks.

[jcbollinger]

On Monday, March 10, 2014 9:27:31 AM UTC-5, Brendon Standing wrote:

Hi

I am busy writing puppet modules for logstash, and want to export tokens I get from the hiera application or %{calling_module} role.

There is no need for "exporting" variables per se, since all Puppet variables (and resources) have global visibility once declared.  You do need to use appropriate (i.e. qualified) names to access variables from a different scope, however, and to access them from inside a template you need to explain to Puppet-unaware ERB what those names mean.

 

So each of my applications/modules that will make use of the shipper module and have the inputs.conf file installed and populated will have the following:

class name::deployment inherits name::params {
include logstash::shipper
}

The token I want to export for each app/module is:
$logstash_logs = hiera('logstash_logs')

If I understand what you are proposing, it will not work as you envision.  That is, if you put that variable declaration in class logstash::shipper, then the calling module that will apply is "logstash".  Moreover, you thereby declare only one variable, $logstash::shipper::logstash_logs, whose value will never change once it is set.  Puppet classes are not macros (or functions), and the 'include' function does not perform textual interpolation ala the C preprocessor.

 

And this is taken from the %{calling_module} role like:
logstash_logs:
javabridge: 'javabridge-log.log'
actions_javabridge: 'ual/actions-javabridge.log'

What you can do is put that variable declaration into some class of each module that needs it, then refer to the appropriate $*::logstash_logs variable via its full-qualified name.

Alternatively, you might be able to build a solution by converting logstash::shipper into a defined type, but in that case you will have an issue with %{calling_module} referring to to 'logstash' instead of the module you want.  There are ways you could work around that, I think, though I don't have details in mind at the moment.

 

The template that is deployed via logstash::shipper has:
<% @logstash_logs.each_pair do |key, value| -%>
  file {
    type => "<%= key %>"
    path => [ "<%= logdir %>/<%= value %>" ]
    tags => [ "<%= key %>" ]
  }
<% end -%>

Again, 'include' doesn't do what you seem to suppose.  There is only one logstash::shipper class, and Puppet classes are idempotent: declaring a class multiple times (e.g. via the 'include' function) has exactly the same effect as declaring it exactly once.  Thus, you will only ever get one instance of that template if you use it only from one declaration in class logstash::shipper.

Also, since I alluded to it before, if a template must access a Puppet variable from a different scope than the one in which the template() or inline_template() function was called, then it can do so with the fully-qualified name and the lookupvar() function of the 'scope' object.  For example,

<%  logs = scope.lookupvar('other_module::logstash_logs') %>

 

Unfortunately the name:deployment class that is calling include logstash::shipper is not passing the tokens.

The error I get is:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template logstash/input.conf.erb:
  Filepath: /usr/lib/ruby/site_ruby/1.8/puppet/parser/templatewrapper.rb
  Line: 81
  Detail: Could not find value for 'logstash_logs' at /etc/puppet/environments/bstanding/modules/logstash/templates/input.conf.erb:1
 at /etc/puppet/environments/bstanding/modules/logstash/manifests/shipper.pp:15 on node free-4574-u37.dev.marinsw.net
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Any idea how best to achieve this, keeping in mind that I will need to export different options for each module / app I am deploying with the shipper?

If you are determined to stick to your plan of using %{calling_module} in your hiera configs to direct lookups to the desired logstash data, then you are probably going to need to write a custom function instead of a class or defined type.  Even then I am only about 90% certain that you can do it.

It would probably be easier, and would involve discarding less work, to convert your logstash::shipper class into a defined type.  You will not then be able to use %{calling_module} in your hiera data, but you can still write module-specific keys.  To work with that, logstash::shipper would use the automatic $calling_module_name variable (http://docs.puppetlabs.com/puppet/latest/reference/lang_variables.html#parser-set-variables) to build the hiera keys with which it looks up data.


John