Using Hiera to Populate Defined Type in Puppet 3.6

[Brian Wilkins]

I am trying to use hiera to populate a defined type to feed the puppet-logstash module. So far, I have been unable to send the data from my hiera file to my defined type. I have tested my defined type and it is working, I just can't seem to populate the variables. It tells me that $content and $order are not set.

/etc/puppet/modules/profiles/manifests/logstash/shipper.pp:

class profiles::logstash::shipper() {

  $shipper_config = hiera('profiles::logstash::config')
  create_resources('config', $shipper_config)

  notice("${shipper_config[name]}")
  class { 'logstash':
    ensure  => 'present',
    version => '1.4.1-1_bd507eb',
    status  => 'enabled',
  }

  profiles::logstash::config { $name:
       content => $content,
       order   => $order,
  }

  include logstash
}

/etc/puppet/modules/profiles/manifests/logstash/config.pp:

define profiles::logstash::config(
  $content = undef,
  $order = undef,
) {
  logstash::configfile { $name:
    content => $content,
    order   => $order
  }
}

/etc/puppet/data/node/els4172.els.dev.yaml:

classes:
  - os::repo
  - profiles::logstash::shipper

profiles::logstash::config:
   input_file:
     content: 'this is a test'
     order: '10'


My notice is not called, it does not display a thing. Did I use create_resources correctly?

[Doug Forster]

profiles::logstash::config:
   input_file:
     content: 'this is a test'
     order: '10'

Should probably be
profiles::logstash::config:

   content: 'this is a test'
   order: '10'

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/c162ac6e-257c-4c42-a856-0b2f99dbd7f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Wilkins]

Oh ok, makes sense. I did that and now I get "can't convert String into Hash at /etc/puppet/modules/profiles/manifests/logstash/shipper.pp:15"

My shipper.pp

class profiles::logstash::shipper() {

  $shipper_config = hiera('profiles::logstash::config', {})
  create_resources('profiles::logstash::config', $shipper_config)

  notice("${shipper_config[name]}")
  class { 'logstash':
    ensure  => 'present',
    version => '1.4.1-1_bd507eb',
    status  => 'enabled',
  }

  profiles::logstash::config { 'shipper':

       content => $content,
       order   => $order,
  }

  include logstash
}

Line 15 is the create_resources line.

[Doug Forster]

Try setting your notice("${shipper_config}") before create resources and see what it prints out.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/305d7707-68ca-4814-940e-ef984fcb2c04%40googlegroups.com.

[Brian Wilkins]

It prints out:

order10contentthis is a test

It concatenated it all together.

[Doug Forster]

Just a thought try changing
  $shipper_config = hiera('profiles::logstash::config', {}) => $shipper_config = hiera_hash('profiles::logstash::config', {})

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/09cd7ee2-7231-499f-a05e-e4430389956a%40googlegroups.com.

[Brian Wilkins]

Same problem. I see it concatenated in the notice.

[Doug Forster]

FYI this is how I ended up doing my logstash config module.  It works but the content is not in Hiera we have flat files we manage.

 $logstash_configs = [
      'app1',

      'app2',

      'app3',
    ]
   
    logstash::configfile { "Banner":
        source => "puppet:///modules/company/logstash/banner",
        order  => 1
      }
   
    $logstash_configs.each |$my_conf| {
      logstash::configfile { "${my_conf}_input":
        source => "puppet:///modules/company/logstash/decc/${my_conf}_input_file",
        order  => 10
      }
      logstash::configfile { "${my_conf}__unfiltered":
        source => "puppet:///modules/company/logstash/decc/${my_conf}_unfiltered_filter",
        order  => 20
      }
      logstash::configfile { "${my_conf}_filter":
        source => "puppet:///modules/company/logstash/decc/${my_conf}_filter",
        order  => 30
      }
    }
    logstash::configfile { 'output':
      source => 'puppet:///modules/company/logstash/output_elasticsearch_http',
      order  => 100
    }

[Doug Forster]

If you use hiera on the server does it show up as a hash?

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/b519938d-da64-4674-8e6e-b98db1dcb9d4%40googlegroups.com.

[Brian Wilkins]

On puppetmaster:

$ sudo hiera --debug profiles::logstash::config ::fqdn=hostname

DEBUG: Thu May 29 13:12:56 -0400 2014: Hiera YAML backend starting
DEBUG: Thu May 29 13:12:56 -0400 2014: Looking up profiles::logstash::config in YAML backend
DEBUG: Thu May 29 13:12:56 -0400 2014: Looking for data source node/hostname
DEBUG: Thu May 29 13:12:56 -0400 2014: Found profiles::logstash::config in node/hostname
{"order"=>"10", "content"=>"this is a test"}

[Doug Forster]

Maybe try your hiera command right inside your create_resources.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f9ece3eb-03cd-4a21-8dd1-30170127c83e%40googlegroups.com.

[Brian Wilkins]

Same problem. It seems like I am close. If only it didn't put all the data together as one string.

[Doug Forster]

I think I figured out what was wrong. Dynamic data bindings may be mapping the inputs to the profiles::logstash::config defined type. Also I was in error in modifying your hiera data.

Try this:

create_resources('profiles::logstash::config', hiera_hash('profiles::logstash::config_settings'))

In Hiera:
profiles::logstash::config_settings:

   input_file:
     content: 'this is a test'
     order: '10'

Add a notice line in your profiles::logstash::config type to:
notify("name is ${name}")
notify("content is ${content}")

notify("order is ${order}")

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/ddff314a-d5e5-4f81-8e26-6eb63b665521%40googlegroups.com.

[Brian Wilkins]

Hmm, that's not working either. Doesn't look like it is populating my define now. Hmm

[Doug Forster]

Does it show up properly if you lookup on the server?

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/03325a33-6c90-46d6-8fe1-928456930478%40googlegroups.com.

[Brian Wilkins]

Yep!

$ sudo hiera --debug profiles::logstash::config_settings ::fqdn=hostname

DEBUG: Thu May 29 14:24:29 -0400 2014: Hiera YAML backend starting
DEBUG: Thu May 29 14:24:29 -0400 2014: Looking up profiles::logstash::config_settings in YAML backend
DEBUG: Thu May 29 14:24:29 -0400 2014: Looking for data source node/hostname
DEBUG: Thu May 29 14:24:29 -0400 2014: Found profiles::logstash::config_settings in node/hostname
{"input_file"=>{"content"=>"this is a test", "order"=>"10"}}

[Brian Wilkins]

Solved it using this tip.. it's odd but it works: http://serverfault.com/a/538877/26514

shipper.pp

class profiles::logstash::shipper() {

  $shipper_array = hiera_array('profiles::logstash::config_array')

  define hash_extract() {
    $shipper_hash = hiera_hash('profiles::logstash::config_settings')
    $shipper_config = $shipper_hash[$name]

    profiles::logstash::config {'shipper':
       content => $shipper_config['content'],
       order   => $shipper_config['order'],
    }
    notice($shipper_config['content'])
    notice($shipper_config['order'])
  }

  hash_extract{$shipper_array:}

  class { 'logstash':
    ensure  => 'present',
    version => '1.4.1-1_bd507eb',
    status  => 'enabled',
  }

  include logstash
}


And in my hostname.yaml:

classes:
  - os::repo
  - profiles::logstash::shipper

profiles::logstash::config_array:
  - inputfile

profiles::logstash::config_settings:
  inputfile:

    content: 'this is a test'
    order: '10'

[jcbollinger]

On Thursday, May 29, 2014 2:39:15 PM UTC-5, Brian Wilkins wrote:

Solved it using this tip.. it's odd but it works: http://serverfault.com/a/538877/26514

That's odd only inasmuch as the original problem in that serverfault question was different from yours.  The OP was trying to achieve a similar structure to the one you are working on, though, so it is natural that what worked for him also works for you.

For what it's worth, I think your original problem was here:

[...]

  profiles::logstash::config { $name:
       content => $content,
       order   => $order,
  }

[...]

The variables $content and $order had not been assigned any values in that scope (class profiles::logstash::shipper), which is exactly what the error message said.

I think that create_resources() was a red herring.  It should have been possible to use create_resources() more or less as you originally attempted to do, though you should have specified the fully-qualified name of the resource type ("profiles::logstash::config"), which you did not do.  Indeed, your final data structure appears still amenable to use with create_resources().

 

shipper.pp

class profiles::logstash::shipper() {

  $shipper_array = hiera_array('profiles::logstash::config_array')

  define hash_extract() {
    $shipper_hash = hiera_hash('profiles::logstash::config_settings')
    $shipper_config = $shipper_hash[$name]

    profiles::logstash::config {'shipper':
       content => $shipper_config['content'],
       order   => $shipper_config['order'],
    }
    notice($shipper_config['content'])
    notice($shipper_config['order'])
  }

  hash_extract{$shipper_array:}

  class { 'logstash':
    ensure  => 'present',
    version => '1.4.1-1_bd507eb',
    status  => 'enabled',
  }
  include logstash

Note that the 'include logstash' is completely redundant (but not directly harmful) in that context because Class['profiles::logstash'] is already declared (immediately prior).  It would be much better to declare the class via its fully-qualified name, though, whichever form you use.

 

}

I urge you, however, to avoid nesting classes or defined types.  The semantics are not necessarily what you expect (THAT was the serverfault questioner's issue), it makes the class or definition harder to find, and it contributes to confusion about the actual names of these things.

John

[Brian Wilkins]

FWIW, this is my working solution and I think it is much better:

/etc/puppet/modules/profiles/manifests/logstash/config.pp:

class profiles::logstash::config {
  $brokers = $profiles::logstash::brokers
  $cluster = $profiles::logstash::cluster

  if (!empty($brokers)) and (empty($cluster)) {
    notice("This is a shipper.")
    logstash::configfile { 'output_broker':
      content => template('profiles/logstash/output_broker.erb'),
      order   => 100
    }
  } elsif (!empty($cluster)) and (!empty($brokers)) {
      notice("This is a central indexer.")
      logstash::configfile { 'input_broker':
        content => template('profiles/logstash/input_broker.erb'),
        order   => 10
      }
      logstash::configfile { 'output_es':
        content => template('profiles/logstash/output_es.erb'),
        order   => 100
      }
  }
}

/etc/puppet/modules/profiles/manifests/logstash/install.pp

class profiles::logstash::install() {
  $ensure = $profiles::logstash::enable ? {true => present, default => absent}
  $status = $profiles::logstash::start ? {true => enabled, default => disabled}

  class { '::logstash':
    ensure  => $ensure,
    status  => $status,
    version => $profiles::logstash::version
  }
}

/etc/puppet/modules/profiles/manifests/logstash.pp:

# == Class: profiles::logstash
#
# A basic module to manage logstash
#
# === Parameters
# [*version*]
#   The package version to install
#
# [*brokers*]
#   An array of brokers to use on this node
#
# [*enable*]
#   Should the service be enabled during boot time?
#
# [*start*]
#   Should the service be started by Puppet?
#
# Note: Values here are defaults and can be overriden by Hiera
#       see - /etc/puppet/data/node/<host>.yaml

class profiles::logstash(
   $version = "1.4.1-1_bd507eb",
   $brokers = ["172.16.14.30", "172.16.14.60"],
   $cluster = undef,
   $enable  = true,
   $start   = true
) {
   class{'profiles::logstash::install': } ->
   class{'profiles::logstash::config': } ->
   Class["profiles::logstash"]
}

And my YAML:

classes:
  - roles::logshipper

profiles::logstash::version: '1.4.1-1_bd507eb'
profiles::logstash::enable: true
profiles::logstash::start: false
profiles::logstash::brokers:
  - hostname1
  - hostname2

And my ERB files

/etc/puppet/modules/profiles/templates/logstash/

<% for @host in @brokers %>

input {
  redis {
    host      => "<%= @host %>"
    type      => "redis-input"
    data_type => "list"
    key       => "logstash"
  }
}

<% end %>

/etc/puppet/modules/profiles/templates/logstash/output_broker.erb:

<%
# iterate over brokers array passed in via Hiera and concatenate
# redis hosts for logstash configuration
host_string = "["
@brokers.each_with_index { |host,idx|
   host_string << "\"#{host}\"";
   host_string << "," if idx < @brokers.length-1
}
host_string << "]"
%>

output {
  redis {
    host => <%= host_string %>
    data_type => "list"
    key => "logstash"
  }
}

/etc/puppet/modules/profiles/templates/logstash/output_es.erb:

output {
  elasticsearch {
    cluster   => "<%= cluster %>"
    index     => "logstash-%{+YYYY.MM.dd.HH}"

[Benjamin Chen]

FYI, just wanted to share this as it allows you to create defined types via hiera.

I built a hiera_manifest functionality at my job. Essentially it allows you to call modules, classes and parameters and defined types all via hiera yaml.

I just put it up on Github to share: https://github.com/mlbam/hiera_manifest

This has slight advantages in that parameters are also put in place at the time of invocation of the module, class or type.

One thing to note is the difference in some syntax for resource definitions when using metaparameters. Also, when troubleshooting, the errors stick to the manifest sometimes and you don't get a proper error message for the module that throws the error.