"extending" hiera data?

[Robert Poulson]

Hey List,

during the last weeks I finally managed to spend some quality time :) with Puppet and I deepened my knowledge a bit: learned some ruby, wrote my first custom ruby facts, used hiera in my latest module... it's been fun.

There is one concept with Hiera though which is still unclear for me.

Let's say all of our servers have only port 22 open. But I assign my module to a node which installs and configures Apache, so it should add port 80 and 443 to the open ports list.

Or I assign my other module to a node which uses SSO and has to enable the mod_auth_kerb Apache module. On my ELK nodes I'd like to use some more Apache modules as well.

Or a Docker node needs to add some rules to the /etc/sudoers, another application some more rules...

I could name a few more examples, but the point is: I can't always define something static in a hiera value, like open_ports: / apache_modules_enabled: / sudo_commands: etc. but need to dynamically manage these arrays when a module is assigned to a node.

How should be this achieved?

A _very_ primitive analogy is like

  $PATH="$PATH:/usr/local/my_app"

where I don't have to take care of the actual content of $PATH, but can simply extend it. Something similar would be great in Hiera (or elsewhere) instead of using defines for every resource I have.

Thanks

Rp

[Peter Kristolaitis]

This is probably a case of trying to use Hiera for stuff you shouldn't be using it for.  It's important to keep in mind that Hiera is not really a replacement for manifest files -- or even better, modules that follow a role & profile architecture.  You can hack Hiera to do that, but you're going to rip your hair out debugging it, and there are better architectural ways of dealing with it anyways.   create_resources() seems really cool when you first learn about it.  That fades quickly as you scale.  ;)

To use your firewall example, "ports opened" depend on the profile of the system (e.g. what software is installed).  All nodes that have Apache need port 80 and 443 opened, for example.  All SSH servers need port 22.

In a "roles & profiles" model, every node probably gets a "baseline firewall" profile assigned to it.  This is where you would set up iptables policies, default-drop rules, etc.   If you're using the puppetlabs-firewall module, you'll probably tell it to purge all non-Puppet-managed firewall rules.

Then in every profile class that requires an open network port, you would have it add the appropriate firewall rule definition.   The SSH profile class adds a firewall rule for port 22.  The Apache class adds firewall rules for ports 80 and 443.

This way, when Puppet is configuring your system, the logic is "I have Apache installed, so I need ports 80 and 443 opened", instead of "I'm node ABC and I have port 80 and 443 opened, but I might or might not have Apache installed", which is a situation you can easily get into if you're shoving as much as you can into Hiera.

This approach seems like much more work at first, because breaking out all your logic into profile modules can be quite verbose and time-consuming.  Sometimes you will get a profile class that does nothing but install a single package (e.g. if you have multiple profiles that require a single Apache module).  That's OK.  It will be MUCH easier and less error-prone to manage over the long run, and you'll appreciate the architecture if you start using exported resources and other features of Puppet.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CANwwCtz0nz6%2BzZMX4ksfrfy%3D8sMjmsSi6vrSpZqS49fVUVO-dA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Poulson]

Hi Peter,

thank you for the valuable input!

I read your answer yesterday many times and asked myself: "yes, it's clear that I'm doing it wrong, but what is the Right Way™ then?".

But many hours later, I might have one of those "aha moments" :)

I always wanted to write modules - or use them in a manner - which define the state of the node after the module gets assigned to it. The modules either contain the information themselves (like the mounts, apache modules etc.) or get these as a parameter; and they always got assigned directly to the node.

But it seems to me that in this "roles & profiles" model, one uses the modules more like if they were just various libs; they're not assigned to the nodes per se, but are just "called" (included) in the node classifiers with the parameters.

This way, I don't have to open the port 5601 for Kibana in my ELK module, but include the firewall class with port 5601 as a parameter within the ELK profile.

Or something similar. Or something different. But I think I got your point :)

The good news is, I have very much work ahead of me so I don't mind to start creating new modules, and re-doing the existing ones in this philosophy.

Thanks; and feel free to correct me if I'm still wrong :)

Rp

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CANwwCtz0nz6%2BzZMX4ksfrfy%3D8sMjmsSi6vrSpZqS49fVUVO-dA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/efe67a83-ce29-07ef-3e69-1bcdb32f2d05%40alter3d.ca.

[Rich Burroughs]

You're right on with the libs part. The modules are assigned to the nodes but using the parameter values that are passed in.

There's a great talk from Gary Larizza from Puppet Conf a couple of years ago where he lays this all out. You write modules with parameterized classes (or use ones from the Forge), and they become like APIs that you use in your profiles/roles.

http://garylarizza.com/blog/2014/10/23/puppetconf-2014-talk/

I highly recommend it. It's funny and informative.

And Peter's response was great. I agree completely. I think of Hiera mainly as a place to put data that's different between environments (load balancer IPs, creds, etc.). I have never liked create_resources and I'm not crazy about the pattern of using Hiera like an ENC.

But everyone can Puppet their own way, and people may love patterns I dislike.

Rich

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CANwwCtz0nz6%2BzZMX4ksfrfy%3D8sMjmsSi6vrSpZqS49fVUVO-dA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/efe67a83-ce29-07ef-3e69-1bcdb32f2d05%40alter3d.ca.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CANwwCtyX0jER-zaQwXfuYhxs8hz5ngMKqbSu%2Bv8RnDAS72s8yw%40mail.gmail.com.

[Peter Kristolaitis]

No problem!   Feeling lost is a little normal -- Puppet is a deceptively complex tool.  You will probably go through MANY iterations of architecture, module layout, etc, as you ramp up your Puppet skills.  When I first started with Puppet, I wrote a very clever ENC that did some internal-business-logic stuff.  It worked great, and I was very proud of myself.  Fast forward a few months, and I realized that I had basically reinvented Hiera, in a way that wasn't as good as Hiera, and all I had to do to use pure Hiera was write a custom fact and use it in my Hiera hierarchy.

The stuff I described is a little hard to understand if you haven't dealt with modules before, because it's very abstract.  However, it seems like you're on the right track.  Group everything to make an "ELK server" into a profile module -- including packages, configuration files, and network ports, then just include that into a role module that gets assigned to a node.  If there is special "data" (say, database credentials), that should come from Hiera.  You may find that you may need to (or it might just be easier to) split out your ELK module into its 3 components, then have the ELK module tie them all together.

However, I would caution you against completely rewriting everything from scratch -- there are a lot of really great modules on the Forge -- Elasticache, for example, has a really good module that is provided by the vendor.  Don't reinvent the wheel.  :)   Just write enough 'glue' code to tie everything together to meet your business needs.

I would start by watching the video linked to by Rich in his reply to this thread.  I haven't watched the whole thing, but what I've watched is a pretty awesome intro to a lot of best practices.  There are a couple other good videos of the roles & profiles model floating around that may be worth your time to watch as well.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CANwwCtyX0jER-zaQwXfuYhxs8hz5ngMKqbSu%2Bv8RnDAS72s8yw%40mail.gmail.com.