Hiera hashes and arrays in ERB templates

[Andy Taylor]

Hi,

I'm trying to build a module for haproxy which fetches all the configuration data from Hiera to populate the haproxy config file. I've run into a number of issues though when I try to use hashes. Ideally, I want to use something like this:

haproxy_listeners :

 "cluster1" :

  ip : '192.168.0.2'

  port : '80'

  servers : 

   "server1" :

    ip : '192.168.0.3'

    port : '8080'

So a hash of clusters with each cluster containing a nested hash of servers. Is this possible with Hiera/ERB? It's easy enough to iterate over the first hash, but I can't work out how to extract the contents of the nested hash. Or I might just be approaching this in entirely the wrong way... Any help would be much appreciated.

Thanks,

Andy

[llowder]

I haven't used the function myself, but this looks like it would be a good case for a define + create_resources(), which I think is part of stdlib. You might need to restructure the hashes slightly, but I think that will be the best approach.

 

Thanks,

Andy

[Wolf Noble]

not sure if this is the "best" way to do this, but, taking from a different module I have:


hiera yaml:

snmp_filesystems:
root: {
mountpoint: '/',
threshold: '10%'
}
snmp_collector:
test: {
ip: '1.2.3.4',
group: 'readonly',
community: 'somecommunity'
}

class:

class snmp::params(
filesystems = hiera(snmp_filesystems, undef),
collector = hiera(snmp_collector, undef)
{
#…
}

template:

<%
collector = scope.lookupvar('snmp::params::collector')
filesystems = scope.lookupvar('snmp::params::filesystems')
-%>

<% collector.each_pair do |key, hash| -%>
group <%=hash['group']-%> v2c <%=key%>
<%end-%>

# Filesystem monitoring enabled
<% filesystems.each_pair do |key, hash| -%>
# <%=key%>
disk <%=hash['mountpoint']-%> <%=hash['threshold']%>
<%end-%>




does that help at all?


Wolf Noble
UNIX Team Lead
Datapipe Managed IT Services
1.201.792.4847 (international) x2910
1.888.749.5821 (toll free) x2910

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/DPkuzR8Q8soJ.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.


________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.

[Gavin Williams]

I'm with Andy on this one... I'm doing something very similar with my NetApp volume provider (https://github.com/fatmcgav/fatmcgav-netapp/).

I've created a define with the following contents:

define util::netapp::volume (
        $ensure = present,
        $size,
        $aggr = 'aggr1',
        $snapresv = 0,
        $autoincrement = true,
        $snapschedule = {"minutes" => 0, "hours" => 0, "days" => 0, "weeks" => 0}
        ) {

        netapp_notify {"volume_define_${name}":
                message => "Processing Volume ${name}",
        }
        ->
        netapp_volume { "v_${name}":
                ensure => $ensure,
                initsize => $size,
                aggregate => $aggr,
                spaceres => "none",
                snapreserve => $snapresv,
                autoincrement => $autoincrement,
                options => {'convert_ucode' => 'on', 'no_atime_update' => 'on', 'try_first' => 'volume_grow'},
                snapschedule => $snapschedule
        }
        ->
        netapp_qtree { "q_${name}":
                ensure => $ensure,
                volume => "v_${name}"
        }
        ->
        netapp_export { "/vol/v_${name}/q_${name}":
                ensure => $ensure,
                persistent => true
        }

}

I've added a default hash to 'snapschedule' in the options list, but that can be over-ridden from the Hiera data.

Then use the following to pull the data from hiera and call the define:

create_resources( util::netapp::volume, hiera('volumes') )

'Volumes' in hiera yaml looks like:

volumes:
 vol1:
  ensure: present
  size: '500m'
 vol2:
  ensure: present
  size: '20g'
  snapschedule:
   minutes: 0
   hours: 36
   days: 0
   weeks: 0

You can also use the 'hiera' command to test your yaml structure:

$ hiera -c hiera.yaml volumes clientcert=act-star-nactl01
{"vol1"=>{"ensure"=>"present", "size"=>"500m"}, "vol2"=>{"ensure"=>"present", "size"=>"20g", "snapschedule"=>{"days"=>0, "weeks"=>0, "hours"=>36, "minutes"=>0}}}

As you can see from the above output, snapschedule for vol2 is a nested hash. This assumes that your resource provider can support hashes on the relevant param/property ;)

HTH

Gav

[Andy Taylor]

Thanks for your suggestions guys. I did consider using create_resource, but don't see how I can when I'm trying to apply this Hiera data to a single file. To expand on my initial post, what I need to do is create multiple config blocks within one file resource. So this Hiera data:

haproxy_listeners :

 "cluster1" :

  ip : '192.168.0.2'

  port : '80'

  servers : 

   "server1" :

    ip : '192.168.0.3'

    port : '8080'

   "server2" :

    ip : '192.168.0.4'

    port : '8080'

 "cluster2" :

  ip : '192.168.0.5'

  port : '80'

  servers :

   "server3"

     ip : '192.168.0.6'

     port : '8080'

   "server4"

     ip : '192.168.0.7'

     port : '8080'

will result in this being generated in the haproxy config file:

listen cluster1 192.168.0.2:80

       server server1 192.168.0.3:8080

       server server2 192.168.0.4:8080

listen cluster2 192.168.0.5:80

       server server3 192.168.0.6:8080

       server server4 192.168.0.7:8080

So I don't see how create_resources can handle this, as that's for creating multiple Puppet resources, as opposed to multiple blocks within a single file. The only alternative I can think of at the moment is using create_resources with a define which utilizes Augeas, but I don't know how well that will work.

Thanks,

Andy

[Steven Nemetz]

I'm using stdlib to help with this
 
$listeners = hiera('haproxy_listeners', undef)
$listener_keys = keys($listeners)
 
then pass $listener_keys to a define to create all the instances
The entre hash is in memory ($listeners) and the define will have the key it is working on ($name)
So you can access anything in the data structure to build your resources.

Steven

 

---

Date: Fri, 4 Jan 2013 09:05:15 -0800
From: andyta...@gmail.com
To: puppet...@googlegroups.com
Subject: [Puppet Users] Re: Hiera hashes and arrays in ERB templates

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/W3UBJBXuT24J.

[jcbollinger]

On Friday, January 4, 2013 9:11:28 AM UTC-6, Andy Taylor wrote:

The data can be expressed in YAML, loaded into a Puppet variable via Hiera, and processed in an ERB template to construct your file.  In other words, yes, it is possible.

May I ask, however, what the advantage is to encoding the data in YAML and then transcoding it into the haproxy configuration language?  I mean, if you also use bits and pieces of that object elsewhere in your manifests then perhaps it makes sense, but if all you want to do with it is generate the config file then I don't see the point.  Why not just serve up the config as a static or mostly-static file?

Leaving aside the question of merits of your design,

• Note that your example data are a quadruply-nested hash, not just a doubly-nested one as you say.  You have a hash of clusters, each of which is a hash itself and contains a third hash, each of whose values is another hash.
• The same Ruby mechanisms that allow your template to iterate and otherwise inspect the outer hash also work for the inner ones.
• Perhaps, though, you are tripping over the fact that the values in your outer hash are of uniform type and meaning, whereas the values in the next-level hash are of varying type and meaning.  You probably don't actually want to iterate the second-level hash; instead, you want to retrieve specific values from it by key.

So, you might do something of this general form (not intended to resemble an actual haproxy configuration file):

haproxy.conf.erb:
============
<% @clusters.each_pair | cluster, properties | { -%>
<cluster name="<%= cluster %>"
  ip="<%= properties['ip'] %>"
  port="<%= properties['port'] %>"
  >
<%   properties['servers'].each_pair | server, server_props | { -%>
  <server name="<%= server %>"
    ip="<%= server_props['ip'] %>"
    port="<%= server_props['port'] %>"
    />
<%   } -%>
</cluster>
<% } -%>


Once corrected for the inevitable typos, that should produce output similar to
<cluster name="cluster1"
  ip="192.168.0.2"
  port="80"
  >
  <server name="server1"
    ip="192.168.0.3"
    port="8080"
    />
</cluster>


John