Roles/profile design

[Josh]

As we are starting to re-factor our puppet modules using Craig's roles/profile design we have found that this system works well for servers (or groups of servers) who have an entire stack of technology deployed to them or who all are part the same custom application. Clusters of servers typically have a unique set of configuration and fit well into this design.  For example, a web-server in the "app1" cluster may look like:

node webserver1 {

  include role::app1_webserver

}

class role::app1_webserver {

  include profiles::base

  include profiles::webserver:app1

}

class profiles::webserver::app1 {

  class { '::apache':

    a => 'x',

    b => 'y',

  }

  file { "/etc/something":

    ensure => present,

    content => template("apache/blah.erb"),

  }

}

Along with standard apache, there are various other custom/non-standard things that need to be done.  This works well in profiles because it provides a layer of abstraction between the component modules and this extra configuration is common for all servers in the "app1" cluster. 

I'm having trouble understanding how to treat those "one-off" servers who just need things like a standard apache, or a standard mysqld installation, but aren't part of any common application and don't need any custom stuff.  I thought about defining profiles like so:

class profiles::webserver::apache::generic {

  include ::apache

}

I feel like this design just over complicates the profiles logic, but I can't seem to figure out another way to handle this type of scenario.  Furthermore, what if I need a generic webserver with PHP?  Do I need another profile (ie, profiles::webserver::apache::generic::php).  Can anyone give any input on to how I should be handling this?

Thanks,

Josh

[Ramin K]

You're overloading profile because you're not using Hiera or an ENC.
Take this example

class role::app1_fe { # or perhaps ::fe_app1
include profile::apache
include profile::php
}

class profile::apache {
include apache

$mymods = hiera('apache::a2mods', {})
create_resources('apache::a2mod', $mymods)
$myvhosts = hiera('apache::vhosts', {})
create_resources('apache::vhost', $myvhosts)

}

profile::apache can be used by *any* server that needs Apache because by
default it does nothing but the minimal config of Apache. However if you
were to feed it data such as modules to enable or vhosts to load you
could build any Apache server you might need.

hieradata/app1_fe.yaml # this assumes you have a role fact.
---
apache::a2mods:
expires: {}
headers: {}
rewrite: {}
apache::vhosts:
www.example.com: {}

Structures like profile::webserver::app1 indicate you're mixing roles
and profiles. And you also embedding data into your Puppet code rather
than writing code to consume data.

It's a lot to wrap your head around and from my experience it takes 1-2
restructures of your Puppet code to fully understand it.

Ramin

[Josh]

Ramin,

Thanks.  Your description was very helpful. I have finally started to understand how everything should be organized.

My profile::apache class needs to drop some additional templates/files to the host for my Apache configuration.  Where should these files and templates be located?  For example, is it frowned upon to create a "files" or "templates" sub-directory under profiles?  Example?

profiles/files/apache/apachestats.conf

profiles/templates/apache/apachestats.erb

This particular file/template should be associated directly with Apache, but I don't know if it warrants creating an entirely new module just to house one or two files.

Thanks,

Josh

[Ramin K]

That's a gray area in my experience. On my system I maintain an
internal Apache module. Because I don't share it or care about using the
more mainstream puppetlabs-apache module I push a number of local
changes down into the Apache module. This makes the most sense for me
because I want Apache to fully work with *my* stats collection,
monitoring, etc out of the box.
That said my module predates role/profile. If I were starting over, I'd
probably write a more general Apache module and push the site specific
files into the profile. That class might look something like this.

class profile::apache {
include ::apache

include sslcerts

collectd::plugin { 'apache': }
logrotate::simple { 'apache': }
nrpe::simple { 'check_apache': }

$myconfs = hiera('apache::conf', {})
create_resources('apache::conf', $myconfs)

$mymods = hiera('apache::a2mods', {})
create_resources('apache::a2mod', $mymods)
$myvhosts = hiera('apache::vhosts', {})
create_resources('apache::vhost', $myvhosts)

Sslcerts::Cert<||> -> Service['apache']
}

Your template question is a bit harder and is again something on a gray
area in my experience. I like to put vhosts in
modules/$mycompany/templates/apache/ rather than directly in the Apache
module. However I do manage status.conf, logging.conf, ports.conf etc
out of the Apache module and all files reside within.
Again if I were doing it over I might make general templates and pass
them data via a hiera lookups in profile::apache. Those templates would
live in the Apache module and I think that's the right place for them.
However I'm running a single system within a single business unit.
Flexibility to support many use cases often takes a back seat to the
quickest way forward which is why most of my conf files aren't even
templates. Easy to get away with hardcoding everything when you own the
full stack.

In summary role, profile, and module aren't so easy to define as we'd
like. The organization of the business, the system, and the way you
admin the system are all factors in how you ultimately decide to
organize your Puppet code and data.

Ramin

[Josh]

Ramin,

After looking more at your example for configuring apache mods via hiera, I have one problem.

The create_resources will actually just define a resource like so:

apache::mod { 'php' }

However, to install the php module with puppetlabs/apache, I actually need to include the apache::mod::php class, ie:

class { 'apache::mod::php' }

Any ideas on how to make this work correctly?  The hiera data should allow me to choose which apache::mod::subclass modules that should be installed.

[Joseph Swick]

On 12/23/2013 02:52 PM, Josh wrote:
> Ramin,
>
> After looking more at your example for configuring apache mods via hiera, I
> have one problem.
>
> The create_resources will actually just define a resource like so:
>
> apache::mod { 'php' }
>
> However, to install the php module with puppetlabs/apache, I actually need
> to include the apache::mod::php class, ie:
>
> class { 'apache::mod::php' }
>
> Any ideas on how to make this work correctly? The hiera data should allow
> me to choose which apache::mod::subclass modules that should be installed.
>
> Thanks,
>
> Josh
>

Hi,
How are you declaring your classes to include from with in hiera?

Is it similar to this?

---
classes:
- apache
- apache::mod
apache::someparamater: value
apache::mod::php: blah


If so, you should be able to do:

---
classes:
- apache
- apache::mod::php
apache::my_favorite_parameter: value
apache::mod::php::php_parameter: some_other_vaule

I haven't tried that exact thing with the apache module, but it does
work for other modules with sub-classes that I've been working with.
That is assuming that you're using the 'classes' array with the
hiera_include function. We use the create_resources function to create
wrappers for defines, while regular classes are included via the
hiera_include and classes array.

Our setup was pretty much taken directly from the hiera documentation:

http://docs.puppetlabs.com/hiera/1/puppet.html#assigning-classes-to-nodes-with-hiera-hierainclude

There are some gotchas that come up with the hiera merge behavior
depending on how complex you're hiera layout becomes. For example, we
had to set the hiera merge_behavior to deeper for us to get some of the
desired results that we were looking for.

--
Joseph Swick <joseph...@meltwater.com>
Operations Engineer
Meltwater Group

[Josh]

Joseph,

I'm not currently defining classes with hiera.  The host is assigned a role, which includes a profile, which installs includes ::apache.  I guess this may be something that we need to look at for these types of scenarios.

[Ramin K]

Something like the following might work.

class profile::app1_site
class {'apache::mod::php':}
package { 'php5-brcypt': }
# etc etc
}

I've avoided parameterized classes for historical reasons, but Joseph's
method should work. It does require some restructuring.

fwiw not a fan of the apache::mod set of functionality unless the
module is part of Apache core. I've noticed there is now a
puppetlabs-passenger module to handle the complexities apache::mod {
'passenger': } could not. Assuming your PHP config is more than just
installing the Apache module I recommend following the same logic and
making your own PHP module with a module and local conf define. Wrap
that in profile::php with the needed create_resources and add it to the
role.

Ramin

> Joseph Swick <joseph...@meltwater.com <javascript:>>
> Operations Engineer
> Meltwater Group
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/c7ba4374-1b14-4dfc-8165-97122836141a%40googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Josh]

Joseph,

So, the problem with this method appears to be that once you specify "hiera_include('classes')" in the environment's site.pp, Puppet appears to try and make Hiera the ONLY source for node classification.  I rely on roles from my ENC and profiles for classification as well.

[Joseph Swick]

On 12/26/2013 12:48 PM, Josh wrote:
> Joseph,
>
> So, the problem with this method appears to be that once you specify
> "hiera_include('classes')" in the environment's site.pp, Puppet appears to
> try and make Hiera the ONLY source for node classification. I rely on
> roles from my ENC and profiles for classification as well.
>
> Josh
>

That's certainly another gotcha with hiera and classes that's good to
know. We went to using hiera exclusively for our node classification,
so we didn't run into that issue as we're not using any other ENCs with
our current configuration.

[jcbollinger]

On Thursday, December 26, 2013 12:25:38 PM UTC-6, Joseph Swick wrote:

On 12/26/2013 12:48 PM, Josh wrote:
> Joseph,
>
> So, the problem with this method appears to be that once you specify
> "hiera_include('classes')" in the environment's site.pp, Puppet appears to
> try and make Hiera the ONLY source for node classification.  I rely on
> roles from my ENC and profiles for classification as well.
>
> Josh
>

That's certainly another gotcha with hiera and classes that's good to
know.  We went to using hiera exclusively for our node classification,
so we didn't run into that issue as we're not using any other ENCs with
our current configuration.

It's good to know only if it's in fact true.  The hiera_include() function is simply a wrapper around Puppet's 'include' function that selects the classes to include via an hiera() lookup.  (Look for yourself if you don't believe me: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/parser/functions/hiera_include.rb .)  Neither hiera_include() nor the underlying include() function inherently interferes with classification via ENC (or node block).

With that said, I like hiera for node classification, and I think there's much to be said for using only hiera for that purpose.


John