Puppet Module Best Practice (Roles/Profiles)

[Lesley Kimmel]

Hi all;

I'm a Linux engineer who went through a typical growth period with Puppet and finally landed on the Roles and Profiles pattern which generally works well.

I have a coworker that started on after me and doesn't like this pattern and having to update profiles or base modules when new functionality is needed; especially for quick one-off things.

So he's basically started creating one class containing 'create_resource' declarations for the standard Puppet resource types (file, user, group, exec, etc.). Then he just adds all of the appropriate parameters in hashes in Hiera. He's convinced this is the right way to do it since he hasn't yet ran into a scenario where this doesn't work easily.

I told him if it was the right way then all the smart people working with and developing Puppet would have put it out as the best practice. However, I can't seem to come up with a really great scenario that will convince him. Can anyone share thoughts on scenarios where this patter will blow up [hard]?

Thanks!

[Dan White]

There appears to be contradiction here. 

If “one-off” changes for “new functionality” are needed, why are the changes going into a base/profile module ?

Would some more specific and detailed examples be possible ?

—————————————————————————————————

"Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us."

Bill Waterson (Calvin & Hobbes)

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/c46b1e54-f3c1-4429-b270-68189e6937f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lesley Kimmel]

As a trivial example maybe we suddenly found that we needed to add a configuration file to a bunch of systems quickly. Because my coworker's class includes a generic call to 'create_resources("file")' we can arbitrarily add any file with any content by simply adding some data to Hiera. Realistically, we should want to get this particular file integrated into a profile at some point. However, since it's so quick and easy to just add it to Hiera my coworker believes it is the better solution.

However, that is somewhat besides the point. I guess the question may more accurately be asked as "Why is using profiles with focused base modules generally considered a best practice but a very generic, but flexible class (as described) is not?"

On Thursday, July 18, 2019 at 8:49:25 PM UTC-5, LinuxDan wrote:

There appears to be contradiction here. 

If “one-off” changes for “new functionality” are needed, why are the changes going into a base/profile module ?

Would some more specific and detailed examples be possible ?

—————————————————————————————————

"Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us."

Bill Waterson (Calvin & Hobbes)

On Jul 18, 2019, at 8:59 PM, Lesley Kimmel <lesley...@gmail.com> wrote:

Hi all;

I'm a Linux engineer who went through a typical growth period with Puppet and finally landed on the Roles and Profiles pattern which generally works well.

I have a coworker that started on after me and doesn't like this pattern and having to update profiles or base modules when new functionality is needed; especially for quick one-off things.

So he's basically started creating one class containing 'create_resource' declarations for the standard Puppet resource types (file, user, group, exec, etc.). Then he just adds all of the appropriate parameters in hashes in Hiera. He's convinced this is the right way to do it since he hasn't yet ran into a scenario where this doesn't work easily.

I told him if it was the right way then all the smart people working with and developing Puppet would have put it out as the best practice. However, I can't seem to come up with a really great scenario that will convince him. Can anyone share thoughts on scenarios where this patter will blow up [hard]?

Thanks!

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet...@googlegroups.com.

[Rob Nelson]

There are times auto-loading resources from hiera data makes sense, such as mlocal users or sudoer.d snippets (though PLEASE use .each instead of create-resources!). These resources should absolutely be resources you in no way manage anywhere else. For sudoer.d, you might “include sudo; $sudoers.each ...” in a base class and then use deep merges to populate the hash in hiera, and that’s fine.

Where you can easily run into problems with this is with resource types that are often managed elsewhere. Files, especially. It’s so easy to have a hash of files that you keep adding to, and one day you end up with catalog compilation failing because of duplicate resources because some “one-off” fix you made now conflicts with a component module you use (ask me how I know!). Even still, it IS possible to do this with careful maintenance.

I would ask why, though? The underlying concept of roles and profiles is to use composition to create your desired state through layers. Having a catch-all class works against such composition, you lose granularity quickly. Your hiera data and hierarchy can get very complex as you try to deliver certain resources to certain nodes via YAML, when it would be a lot simpler to deliver those same resources through a properly composed role written in Puppet DSL.

A simple (contrived) example might two nodes that get all their config through a profile::base class via create_resources/.each. One is a web server, the other a database server. The first needs an Apache config file, an sshd.config file, and an application config file. The second needs a my.cnf and an sshd.config file. You could place the sshd.config data in default.yaml, the Apache and app configs in node_service/app1.yaml, and the MySQL file in node_service/database.yaml. So far, so good.

Some time goes by and you decide to experiment with nginx and app1. You can leave the app config in node_service/app1.yaml, but you have to move the Apache config somewhere and add another layer to the hierarchy for the webserver. Then  more time goes on and you see new config options for nginx that are only available in rhel8, so you have to add a hierarchy level for osfamily version+webserver. This rabbit hole gets deep, quickly.

Alternatively, you could assign role::MySQL (profile::base, profile::MySQL) and role::app1::apache (profile::base, profile::Apache, profile::app1) to the two nodes. When you check out nginx, you create role::app1::nginx (profile::base, profile::nginx, profile::app1). Each config file is included via the specific profile, and the data hierarchy is simpler (default, osfamilyd/). This is far easier to hold in your head, troubleshoot, and extend IMO.

I definitely think composition has significant benefits over using catch-all parameters, though specific “collected” parameters can be leveraging safely. I also think that managing YAML gets ugly quickly, not to mention it is more reminiscent of certain other CM systems 😁. In any case, I hope this helps you evaluate the pros and cons of each approach and how to find the solution that works for you!

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/8143b443-4a28-45b4-9931-d200e3e91223%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Rob Nelson

[Dan White]

Thanks for the detail, Rob.

I was racking my brain to compose a response with minimum success so far.

------------------------------------------------
“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”  (Bill Waterson: Calvin & Hobbes)

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAC76iT8sW9MMN%3DvL0OsFv25DGcU%3DPypPBZ2v%3D7m5fUaG%2ByN4Bg%40mail.gmail.com.

[jcbollinger]

On Thursday, July 18, 2019 at 7:59:26 PM UTC-5, Lesley Kimmel wrote:

Hi all;

I'm a Linux engineer who went through a typical growth period with Puppet and finally landed on the Roles and Profiles pattern which generally works well.

I have a coworker that started on after me and doesn't like this pattern and having to update profiles or base modules when new functionality is needed; especially for quick one-off things.

So he's basically started creating one class containing 'create_resource' declarations for the standard Puppet resource types (file, user, group, exec, etc.). Then he just adds all of the appropriate parameters in hashes in Hiera. He's convinced this is the right way to do it since he hasn't yet ran into a scenario where this doesn't work easily.

All technical considerations notwithstanding, this is a problem that should be addressed by your manager or group leader.  I estimate chances of logic-ing your junior colleague into doing things differently to be low, no matter what argument you present.  They don't have the experience or vision to inform a different opinion than they have formed, and they are focused on expediency.  But there is enormous benefit to taking a consistent approach within the team.  The manager / leader should recognize that, and it is reasonable to hope that the junior member will at least accept that, even if they don't like the extra steps they need to take to be consistent.

Of course, management might want reasons for continuing with the roles & profiles approach instead of switching to what amounts to pure-Hiera manifests.  The problem with this is that it is hard to give concrete examples before you run into actual issues, and you would prefer to avoid such issues in the first place.  IT-savvy managers might be influenced by the observation that preferring Roles & Profiles over all-Hiera can be characterized as favoring composition over inheritance for customizing and deriving machine configurations.  They might also appreciate that encapsulating units of configuration in Puppet classes constitutes a modular approach that is easier to manage and maintain in the aggregate, over time.  Your coworker's approach is reasonable for emergency fixes that need to roll out absolutely as fast as possible, but if you build your infrastructure management as a tower of such jerry-rigs then down the road you are probably facing increasingly difficult maintenance problems and ultimately a large-scale rewrite / refactoring.

 

I told him if it was the right way then all the smart people working with and developing Puppet would have put it out as the best practice. However, I can't seem to come up with a really great scenario that will convince him. Can anyone share thoughts on scenarios where this patter will blow up [hard]?

It's going to become more and more prone to duplicate resource declaration errors as the number of resources declared this way grows.  It's going to require either an increasingly deep and complex Hiera hierarchy, or else a great deal of per-individual-node configuration data.  There is every reason to expect a lot of data duplication to emerge in such a situation.  Note, however, that you don't necessarily need to go all the way to roles and profiles to address most of those considerations.  Many of them can be addressed to a significant extent simply by thoughtful design of your main classes and modules.

Overall, it's a question of scope.  The problems addressed by Roles & Profiles are fairly high-level ones of organizing and maintaining a large and diverse manifest set for infrastructure that itself has a lot of diversity.  For simple situations it is probably a bit overkill, but well-designed classes and modules are still much to your benefit.

John

[Luke Bigum]

On Friday, 19 July 2019 01:59:26 UTC+1, Lesley Kimmel wrote:

Hi all;

I told him if it was the right way then all the smart people working with and developing Puppet would have put it out as the best practice. However, I can't seem to come up with a really great scenario that will convince him. Can anyone share thoughts on scenarios where this patter will blow up [hard]?

Thanks!

Everyone else's replies touch on all the major points, so can't add too much more there.  I do have an ageing blog post though, and it might be helpful... The post describes a contrived scenario based on a real-world problem; it starts with an engineering team working mostly out of Hiera YAML files, and then over time getting into a situation where that design pattern was causing too much pain, and thus the conversion to a role-profile pattern.  All the internal technical secrets are obfuscated away behind this contrived scenario, but something very close to this happened to us - a medium sized company that's been using Puppet for 8+ years.

The problem being solved is to do with creating networking on Linux servers, so it's a great deal more complicated than just adding a package to a machine. However if you can understand the concepts and the challenges faced, it might work as a real-world example for your Junior staff member... Maybe... 30% chance :-)

The first half of the page is the scenario / blog post (which is the bit you want), the next is the docs for the example module itself describing our solution (which won't be useful to you):

https://github.com/lukebigum/puppet-networking-example