Craig Dunn's Roles/Profiles/Components & Conflicts

[mjus...@gmail.com]

Hi all,

We use the roles/profiles/components model originally suggested by Craig Dunn fairly heavily.  In our case:

• The role is a business name, like "Application X App Server"
• The profile is the technical name, like "Base Components" or "Webserver"
• The components are either wrapper classes around modules or modules themselves, like "PHP" or "Apache".

For the most part, this works well.  We can have, for example:

• MyFace Application Server
• Base Components
• SSSD
• Sudo
• NTP
• PHP Webserver
• PHP
• Apache
• PHP-FPM
• Memcache

However, we're running into trouble how to handle the situation where you're running a box with multiple functions... for example, WordPress and Drupal.  In that case, how do you handle configuration conflicts?  On the surface, it seems like we would create a more generic profile like "PHP Webserver" (like I did in the above example).  If I do this, however, I lose the ability to define profile specific variables such as firewall rules, cron jobs, etc.

Any thoughts on this?

Best,

Matt

[jcbollinger]

If you follow Dunn to the letter then each node has exactly one role.  Therefore, if you want a box with multiple functions then you need to define a role that encompasses all of them.  That's a useful exercise in itself, as it makes you think about why those functions are all served by the same machine.

Suppose, then, that you have a role for what you're talking about, something such as "Satellite Office Omnibus Server" or whatever.  You then proceed as normal for the pattern, by defining one or more profiles for machines having that role.  Those profiles define all the needed components, just as your other profiles do.  Presumably the profiles for such roles will be larger than most.

I think the key hurdle here is defining appropriate roles.  "Multi-function computer" isn't a role, it's just a vague description.  Given genuine, coherent roles for these computers, I think everything else will fall out fairly naturally.


John

[Christopher Wood]

(inline)

On Wed, May 14, 2014 at 10:22:02PM -0700, mjus...@gmail.com wrote:
> Hi all,
> We use the roles/profiles/components model originally suggested by Craig
> Dunn fairly heavily.  In our case:
>

> * The role is a business name, like "Application X App Server"
> * The profile is the technical name, like "Base Components" or
> "Webserver"
> * The components are either wrapper classes around modules or modules

> themselves, like "PHP" or "Apache".
>
> For the most part, this works well.  We can have, for example:
>

> * MyFace Application Server
>
> * Base Components
>
> * SSSD
> * Sudo
> * NTP
>
> * PHP Webserver
>
> * PHP
> * Apache
> * PHP-FPM
> * Memcache

>
> However, we're running into trouble how to handle the situation
> where you're running a box with multiple functions... for example,
> WordPress and Drupal.  In that case, how do you handle configuration
> conflicts?  On the surface, it seems like we would create a more generic
> profile like "PHP Webserver" (like I did in the above example).  If I do
> this, however, I lose the ability to define profile specific variables
> such as firewall rules, cron jobs, etc.
> Any thoughts on this?
> Best,
> Matt

You might be interested in this thing, it helped me a great deal:

https://ask.puppetlabs.com/question/1655/an-end-to-end-roleprofile-example-using-hiera/

Other tips:

-only includes and chaining in roles/profiles (no resources)
-never use resource style declarations for classes anywhere
-all data in hiera
-make lots of use of include directories in configurations
-examples: /etc/httpd/conf.d, /etc/profile.d

My further rhubarbing:

Sounds like you'd end up with three roles to start:

wordpress server
drupal server
wordpress+drupal server

A bit of thinking about how to break out commonalities into other hiera levels and you're there.

Later on, perhaps try combining those into a single "phpttpd" role where hiera flags set which php configuration bits are installed into the aforementioned include directories, with suitable chaining freeing you from too much explicit notify/subscribe.

Which model you use depends on how you use the servers and whether it's easier for your admins to think of them all as generic servers with pseudo-random features per host, or hosts with one specific function. I've preferred the first for most of my stuff, but another department prefers the second style given their operational constraints.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [1]puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> [2]https://groups.google.com/d/msgid/puppet-users/c1157f75-5a08-4e13-8739-abef8bd23cf4%40googlegroups.com.
> For more options, visit [3]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:puppet-users...@googlegroups.com
> 2. https://groups.google.com/d/msgid/puppet-users/c1157f75-5a08-4e13-8739-abef8bd23cf4%40googlegroups.com?utm_medium=email&utm_source=footer
> 3. https://groups.google.com/d/optout

[Ramin K]

On 5/14/2014 10:22 PM, mjus...@gmail.com wrote:
> Hi all,
>
> We use the roles/profiles/components model originally suggested by Craig
> Dunn fairly heavily. In our case:
>

> * The role is a business name, like "Application X App Server"
> * The profile is the technical name, like "Base Components" or "Webserver"
> * The components are either wrapper classes around modules or modules

> themselves, like "PHP" or "Apache".
>
> For the most part, this works well. We can have, for example:
>

> * MyFace Application Server
> o Base Components
> + SSSD
> + Sudo
> + NTP
> o PHP Webserver
> + PHP
> + Apache
> + PHP-FPM
> + Memcache

>
> However, we're running into trouble how to handle the situation
> where you're running a box with multiple functions... for example,
> WordPress and Drupal. In that case, how do you handle configuration
> conflicts? On the surface, it seems like we would create a more generic
> profile like "PHP Webserver" (like I did in the above example). If I do
> this, however, I lose the ability to define profile specific variables
> such as firewall rules, cron jobs, etc.
>
> Any thoughts on this?

As always John's response earlier in the thread was spot on.
Determining role rather than roles is the basis of your problem.

The way I design role/profile today is to think in terms of creating a
data schema that describes my system. Then I assemble roles, profiles,
and modules to consume that schema. That schema is realized in the
hiera.yaml. A simple flexible hiera.yaml might look like:

:hierarchy:
- hosts/%{clientcert}
- env/%{environment}/%{role}
- role/%{role}
- env/%{environment}
- common

I believe that role (or some unique fact, enc lookup, whatever) is the
right place to insert function specific data because it is unique to
that group of servers. Some resources like sourcing firewall rules based
on multiple profiles are more likely to lead to conflicts particularly
if there are overlapping functionality. Some resources like vhosts are
unlikely to collide though load order might be unexpected.
As a side note attempting to do role/profile without Hiera, ENC,
Foreman, or some sort of external data source is doomed to fail in my
opinion.

I'd also like to disagree slightly with Christopher who also posted in
this thread. Your profile:: classes are the perfect place for all sorts
of site specific nonsense including resources. In fact if you're not
sufficiently embarrassed of your profile classes you're probably not
taking full advantage of that abstraction layer. I wrote about how I
use/abuse them here,
https://ask.puppetlabs.com/question/5235/what-goes-in-the-profile-part-of-roleprofile/

Ramin

[Christopher Wood]

(inline)

(I think I had this exact conversation when discussing profiles at work.)

This is usually where I say: If you have site-specific things you should modularize those and add the relevant include statements and chaining to the profile, with the data in hiera.

For us that includes a site level in hiera so that hosts at different sites get different puppetmasters (server not ca_server), ntp servers, resolvers, and so on. This helps us use the mnemonic that anything in node yaml indicates either an oddity or an error. This will also let us quickly include a module if we need it somewhere else.

Of course, your mileage may vary, mine sure has at times.

> Ramin

>
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/53750B41.40806%40badapple.net.
> For more options, visit https://groups.google.com/d/optout.

[Gary Larizza]

Just to add, I've written a bit about this at http://bit.ly/puppetworkflows2 and you might want to take a look

Profiles are TECHNOLOGY-specific, so a profile for wordpress, and one for drupal, and one for java, and etc...  Roles are BUSINESS LOGIC-specific, so a role for your cluster nodes, or a role for your application servers, and etc (however you refer to your machines internally).  This allows you to swap out profiles (technology-specific components) in the event that different roles require different technology components.

Hope that helps a bit.

 

Any thoughts on this?

Best,

Matt

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/c1157f75-5a08-4e13-8739-abef8bd23cf4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Gary Larizza

Professional Services Engineer

Puppet Labs

[Jesse Cotton]

I work with Matt and am filling in for him since I posed this question to him originally.

Our confusion really lies around how you layout profiles for a multi-function box. For example, suppose you have a role, "CMS App Server" that will host various CMS like Wordpress, Drupal, and others. They are all built on top of the same technologies, Apache, PHP, and MySQL. I don't believe you can build a separate profile for each CMS b/c they will conflict (at least within Puppet). Each will require a certain set of php modules and settings, apache modules and settings, etc. So do you build a single profile like profile::wordpress_drupal_cms3_cm4 or do you build a profile::apachefpm profile? The later seems more logical to me however you lose the ability to define profile specific hiera variables b/c profile::apachefpm is generic.

[Ramin K]

On 5/15/2014 12:14 PM, Christopher Wood wrote:
> (inline)
>
> On Thu, May 15, 2014 at 11:45:21AM -0700, Ramin K wrote:
>>
>> I'd also like to disagree slightly with Christopher who also
>> posted in this thread. Your profile:: classes are the perfect place
>> for all sorts of site specific nonsense including resources. In
>> fact if you're not sufficiently embarrassed of your profile classes
>> you're probably not taking full advantage of that abstraction
>> layer. I wrote about how I use/abuse them here,
>> https://ask.puppetlabs.com/question/5235/what-goes-in-the-profile-part-of-roleprofile/
>
> (I think I had this exact conversation when discussing profiles at
> work.)
>
> This is usually where I say: If you have site-specific things you
> should modularize those and add the relevant include statements and
> chaining to the profile, with the data in hiera.
>
> For us that includes a site level in hiera so that hosts at different
> sites get different puppetmasters (server not ca_server), ntp
> servers, resolvers, and so on. This helps us use the mnemonic that
> anything in node yaml indicates either an oddity or an error. This
> will also let us quickly include a module if we need it somewhere
> else.
>
> Of course, your mileage may vary, mine sure has at times.

Oh good, disagreement. I think much harder when that happens :-)

We probably agree more than disagree, but the statement that bothered
me was that profile was a place of pure includes and chains. Sure
everything should be modularized, but often the time to do it is
eventually and sometimes never. Generalization should only happen once
you need more than one and fully understand the problem. Perhaps we
could mimic Knuth and say "Premature generalization is a distraction
from automating your production system."

I think there is a distinction between small systems and large ones
which affect both our point of views. I run a smallish system that I own
completely. Without dissenting opinions about what should be on servers
I'm free to make sweeping changes to how we manage a daemon with hardly
anyone noticing let alone caring. And because I'm manpower limited, I'm
unlikely to do generalization unless absolutely needed.
In a larger system many sites, divisions, etc are going to do things
differently and generalization is required much earlier and often. Also
with a larger staff the additional code and complexity is worth it if it
keeps everyone from reinventing the wheel.

Ramin

[Joaquin Menchaca]

Hello: This might be a dumb question, but can you build a system manually with the various CMSes?  Are there a cross-set of technologies (Apache, PHP, MySQL) that will work on all CMSes in your organization?  I would think in terms in how you would build these systems manually w/o puppet in some organized way, then see how this can fit into puppet.   If you can build some reference platform that can host some or all of these, then you have your basic profile for that wonder-web-box system.

Complexity might come in when you start separating, such as putting php in it's own module and apache in another, then sequencing them in profile.  You can use augues to scissor in php specific configurations (or alternatively creative erb ruby code to other configurations).  This is what I do for a nginx + php-fm configuration.

class profile::webserver {  class  { "nginx": } -> class  { "php": } }

[Jesse Cotton]

Yes it is feasible :) So really the server is just a PHP app server. What makes it useable for all of the CMSs is that the correct set of apache and php modules are installed, and it has the correct firewall rules, cron jobs and backup jobs configured. With this in mind, my inclination is to build a cms_app_server profile but that seems more like a role then a profile.

[David Schmitt]

Hi,

To answer the original qustion with a somewhat cheap truism: It is
possible, but not required that profiles are composable. E.g.
profile::apache_server and profile::mysql_server should be trivially
composable, because they manage disjunct sets of resources. On the other
hand, profile::apache_server and profile::nginx_server might not be
composable, because both need to use port 80, which cannot be shared.

So to the question of building composable profiles, if your profiles
often need a specific set of php modules, it might ease the situation by
creating a profile::fat_php_installation, which contains all required
php modules and becomes the focal point for intra-cms-php-development. A
different approach would be finegrained classes for each php module,
which can be included multiple times across many profiles. A totally
different approach would be to create local php installations for each
CMS which do not talk to each other at all, leaving the CMS consumer to
choose the exact patch level and composition required.


Surely, there are several other possibilities. Which one is the "right"
one really depends on the structure of your development team and systems.


Regards, David

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com

> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/be9a1775-2d62-49f2-8b25-ae072729c059%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/be9a1775-2d62-49f2-8b25-ae072729c059%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Joaquin Menchaca]

Out of curiosity, what exactly are all the components you are trying to install, e.g. CMSes? Drupal? WordPress? Joomla?  Is that database going to be on the same system?

Note, segue here before I divulge a solution, perhaps sacrilegious, is that the role/profile might not be the best design pattern for, as it seems to be a top-down with functional decomposition, rather than a bottom-up approach (e.g. starting with apache, app-engine  child class, then cms  child class, each one inheriting properties and actions of the parent).  There are other OOP design patterns (yet to be documented) that could work for Puppet.  Still, here's a way to maybe get this to work with roles/profiles:

You could write maybe a fat_php module (not profile), but has parameters to turn on/off the features you want through parametrization.  Thus you can have this fat_php module support all the requirements to install everything under the sun, then a profile will toggle them on and off through parameters.  Thus, a package's ensure attribute can be set to "present" depending on what CMS the profile will build. 

The role would call whatever profile CMSes that are needed for that role, so it can call two or more CMS profiles.  Each one will toggle various options, such as a package's ensure to "present", appropriate to the that CMS.  When the catalog is compiled, and 2+ CMSes were enabled, packages will only be installed once.  If any custom code should have some way to avoid collisions, if it has already been invoked previously, so you need to be careful to ensure it is idempotent, especially for exec { command: }.

When you advanced the code base to use Hiera, you can just use class includes, to have pure idempotence, and avoid specifying parameters through class { param=value }, which might cause some unexpected, ala honey badger ("don't give a shit"), results when classes might try to get installed twice or out of order between profiles included in the roles.

Similarly, you can have a fat MySQL module which has the needed schema and account configurations needed to make it work with whatever CMS.  These sets of configurations and actions are toggled on/off, and which will be set by the profile (or hiera config related to the profile).

[jcbollinger]

On Friday, May 16, 2014 10:55:35 AM UTC-5, Jesse Cotton wrote:

I work with Matt and am filling in for him since I posed this question to him originally.

Our confusion really lies around how you layout profiles for a multi-function box. For example, suppose you have a role, "CMS App Server" that will host various CMS like Wordpress, Drupal, and others. They are all built on top of the same technologies, Apache, PHP, and MySQL. I don't believe you can build a separate profile for each CMS b/c they will conflict (at least within Puppet). Each will require a certain set of php modules and settings, apache modules and settings, etc. So do you build a single profile like profile::wordpress_drupal_cms3_cm4 or do you build a profile::apachefpm profile? The later seems more logical to me however you lose the ability to define profile specific hiera variables b/c profile::apachefpm is generic.

You can declare the same class multiple times, provided that you use the 'include' function or its cousins to do so.  That is a tremendously useful technique for managing configurations where multiple facilities share requirements on some of the same core resources.  Using 'include' instead of resource-like class declarations means you must feed data to your classes via hiera instead of via your manifests, but separating your data from your manifests is good practice anyway.  That's what Christopher Wood was pointing you toward, and it is good advice.

It is thus possible to find and/or write composable classes and modules for managing the components you want.  As David Schmitt observes, composability is not automatic, but I don't see why your particular case of an apache-based PHP app server with firewall rules and a specific collection of PHP and apache modules should present any special problem.

Thus, the answer to your question is "neither."  You build a drupal profile, and a wordpress profile, etc., and you use them all together.


John

[Jesse Cotton]

On Monday, May 19, 2014 10:18:11 AM UTC-4, jcbollinger wrote:

On Friday, May 16, 2014 10:55:35 AM UTC-5, Jesse Cotton wrote:

I work with Matt and am filling in for him since I posed this question to him originally.

Our confusion really lies around how you layout profiles for a multi-function box. For example, suppose you have a role, "CMS App Server" that will host various CMS like Wordpress, Drupal, and others. They are all built on top of the same technologies, Apache, PHP, and MySQL. I don't believe you can build a separate profile for each CMS b/c they will conflict (at least within Puppet). Each will require a certain set of php modules and settings, apache modules and settings, etc. So do you build a single profile like profile::wordpress_drupal_cms3_cm4 or do you build a profile::apachefpm profile? The later seems more logical to me however you lose the ability to define profile specific hiera variables b/c profile::apachefpm is generic.

You can declare the same class multiple times, provided that you use the 'include' function or its cousins to do so.  That is a tremendously useful technique for managing configurations where multiple facilities share requirements on some of the same core resources.  Using 'include' instead of resource-like class declarations means you must feed data to your classes via hiera instead of via your manifests, but separating your data from your manifests is good practice anyway.  That's what Christopher Wood was pointing you toward, and it is good advice.

We're aware of most of this and agree with most of this. However when you always call include, you lose the ability to say a particular hiera variable is attached to the profile. For example

If you define:

class profile::apache_phpfpm {

  include ::apache

}

with the following in hiera:

apache::keepalive = 1

keepalive = 1 applies anywhere apache is included

vs

class profile::apache_phpfpm (

  $keepalive = 1

) {

  class { ::apache:

    keepalive => $keepalive

  }

}

profile::apache_phpfpm::keepalive = 1

So with the later you can build a somewhat self-contained profile. With the former you have to set variables "globally" or on a node.

 

It is thus possible to find and/or write composable classes and modules for managing the components you want.  As David Schmitt observes, composability is not automatic, but I don't see why your particular case of an apache-based PHP app server with firewall rules and a specific collection of PHP and apache modules should present any special problem.

Thus, the answer to your question is "neither."  You build a drupal profile, and a wordpress profile, etc., and you use them all together.

So below is an example for Drupal. It could literally be cloned for Wordpress and Joomla. Unfortunately, b/c of the class declarations, it is not composable.

class profile::drupal (

  $apache_listen = ['80'],

  $apache_name_virtual_hosts = ['*:80'],

  $apache_modules = ['fastcgi'],

  $apache_fastcgi_servers = {

    'www' => {

      host => '127.0.0.1:9000',

      faux_path => '/var/www/php.fcgi',

      alias => '/php.fcgi',

      file_type => 'application/x-httpd-php'

    }

  },

  $phpfpm_pools = {

    'www' => {

      listen  => '127.0.0.1:9000',

      user => 'apache',

      pm_max_requests => 500,

      catch_workers_output => 'no',

      php_admin_value => {},

      php_value => {}

    }

  },

  $php_modules = [],

  $firewall_rules = {},

  $backup_jobs = {},

  $cron_jobs = {}

) {

  include ::apache

  ::apache::listen { $apache_listen: }

  ::apache::namevirtualhost { $apache_name_virtual_hosts: }

  ::apache::mod { $apache_modules: }

  create_resources(::apache::fastcgi::server, $apache_fastcgi_servers)

  include ::php::fpm::daemon

  create_resources(::php::fpm::conf, $phpfpm_pools)

  ::php::module { $php_modules: } ~> Service['php-fpm']

  # So the apache user is created before

  # php-fpm starts

  Class['::apache'] -> Class['::php::fpm::daemon']

  create_resources(firewall, $firewall_rules)

  create_resources(::duplicity::job, $backup_jobs)

  create_resources(::cron::job, $cron_jobs)

}

 

John

[Joaquin Menchaca]

You could do global variables, but you couldn't one set the hierarchy to the profile level?  Thus you isolate variables in preference to the profile, rather than a global mismash.  Stick with parametrization for now if this works, refactor later after getting it working, as you'll have a functional base state.
 

We're aware of most of this and agree with most of this. However when you always call include, you lose the ability to say a particular hiera variable is attached to the profile. For example

If you define:

class profile::apache_phpfpm {

  include ::apache

}

with the following in hiera:

apache::keepalive = 1

keepalive = 1 applies anywhere apache is included

vs

class profile::apache_phpfpm (

  $keepalive = 1

) {

  class { ::apache:

    keepalive => $keepalive

  }

}

profile::apache_phpfpm::keepalive = 1

I am not sure about this... Personally, I get scared of having too many sub-classes beyond install.pp, config.pp, service.pp, etc. It might make those files verbose, but then it is easier to contain and debug.  Afterwards, parts into separate pieces, if this helps.

Is cron a separate class?  Personally, I put my bottom level modules into their product role, which may require cron functionality.  I avoid crafting classes based on their functionality, as this would be the same as having a class called package that accepts an array packages to be installed.  It gets hard to identify what is involved in installing a product or scheduling cleanup tasks for the same product; the code to implement a particular product gets scattered across many files, rather than self-contained in one module.
 

[Jesse Cotton]

Perhaps I am overlooking something. How would you construct the hierarchy so you could set apache::keepalive for this profile?

class profile::apache_phpfpm {

  include ::apache

}

You could do global variables, but you couldn't one set the hierarchy to the profile level?  Thus you isolate variables in preference to the profile, rather than a global mismash.  Stick with parametrization for now if this works, refactor later after getting it working, as you'll have a functional base state.
 

We're aware of most of this and agree with most of this. However when you always call include, you lose the ability to say a particular hiera variable is attached to the profile. For example

If you define:

class profile::apache_phpfpm {

  include ::apache

}

with the following in hiera:

apache::keepalive = 1

keepalive = 1 applies anywhere apache is included

vs

class profile::apache_phpfpm (

  $keepalive = 1

) {

  class { ::apache:

    keepalive => $keepalive

  }

}

profile::apache_phpfpm::keepalive = 1

I am not sure about this... Personally, I get scared of having too many sub-classes beyond install.pp, config.pp, service.pp, etc. It might make those files verbose, but then it is easier to contain and debug.  Afterwards, parts into separate pieces, if this helps.

Is cron a separate class?  Personally, I put my bottom level modules into their product role, which may require cron functionality.  I avoid crafting classes based on their functionality, as this would be the same as having a class called package that accepts an array packages to be installed.  It gets hard to identify what is involved in installing a product or scheduling cleanup tasks for the same product; the code to implement a particular product gets scattered across many files, rather than self-contained in one module.
 

I am not sure I understand what you're suggesting here. The point of the create_resources calls at the very bottom is so we can attach backup jobs, cron jobs, firewall rules to the profile. You seem to be suggesting we should assign these at the role level. Is that correct? I see valid reasons for doing it either way but experience may dictate one is better then the other.

Yes, apache, php, cron, duplicity, and firewall are all separate classes.

[jcbollinger]

On Monday, May 19, 2014 1:42:08 PM UTC-5, Jesse Cotton wrote:

On Monday, May 19, 2014 10:18:11 AM UTC-4, jcbollinger wrote:

On Friday, May 16, 2014 10:55:35 AM UTC-5, Jesse Cotton wrote:

I work with Matt and am filling in for him since I posed this question to him originally.

Our confusion really lies around how you layout profiles for a multi-function box. For example, suppose you have a role, "CMS App Server" that will host various CMS like Wordpress, Drupal, and others. They are all built on top of the same technologies, Apache, PHP, and MySQL. I don't believe you can build a separate profile for each CMS b/c they will conflict (at least within Puppet). Each will require a certain set of php modules and settings, apache modules and settings, etc. So do you build a single profile like profile::wordpress_drupal_cms3_cm4 or do you build a profile::apachefpm profile? The later seems more logical to me however you lose the ability to define profile specific hiera variables b/c profile::apachefpm is generic.

You can declare the same class multiple times, provided that you use the 'include' function or its cousins to do so.  That is a tremendously useful technique for managing configurations where multiple facilities share requirements on some of the same core resources.  Using 'include' instead of resource-like class declarations means you must feed data to your classes via hiera instead of via your manifests, but separating your data from your manifests is good practice anyway.  That's what Christopher Wood was pointing you toward, and it is good advice.

We're aware of most of this and agree with most of this. However when you always call include, you lose the ability to say a particular hiera variable is attached to the profile. For example

If you define:

class profile::apache_phpfpm {

  include ::apache

}

with the following in hiera:

apache::keepalive = 1

keepalive = 1 applies anywhere apache is included

vs

class profile::apache_phpfpm (

  $keepalive = 1

) {

  class { ::apache:

    keepalive => $keepalive

  }

}

profile::apache_phpfpm::keepalive = 1

So with the later you can build a somewhat self-contained profile. With the former you have to set variables "globally" or on a node.

That is what your data hierarchy is for.  Hiera does not limit you to only global and per-node data.  You can define as many hierarchy levels as you need, each grouping your nodes in its own way (though it usually makes the most sense for the groups at each level to be subsets of the groups at the next lower level).

And the foregoing is based on using only the built-in YAML back end.  Hiera supports pluggable back ends, usable together or separately.  A custom back end can employ whatever lookup or computation you want to serve whichever data you choose.

 

Either you're missing something or I am.  I see nothing in that class that would inherently preclude it being composed.  In particular, the two class declarations it contains both use 'include', not the resource-like class declaration syntax.  If there is a barrier to composition it would be related to composition with another class that declares some of the same resources.  That problem has a solution, however: factor out the multiply-declared resources into their own class or classes, which the then-composable classes declare instead of declaring the resources directly.


John

[Christopher Wood]

There's a point. Though regardless of my level of understanding I definitely will need more than one (more below.)

> I think there is a distinction between small systems and large ones
> which affect both our point of views. I run a smallish system that I
> own completely. Without dissenting opinions about what should be on
> servers I'm free to make sweeping changes to how we manage a daemon
> with hardly anyone noticing let alone caring. And because I'm
> manpower limited, I'm unlikely to do generalization unless
> absolutely needed.
> In a larger system many sites, divisions, etc are going to do
> things differently and generalization is required much earlier and
> often. Also with a larger staff the additional code and complexity
> is worth it if it keeps everyone from reinventing the wheel.

I think you've hit the nail right on the head. My thinking is definitely influenced by the need to do things at (medium-ish) scale and I won't be interrupted by some random server's nagios tantrum. If it was a choice between perfect profiles and being able to get everything else done too I would definitely make your choice instead (as I have before). As long as it's in git/puppet somebody can read it later and figure out what I was up to.

> Ramin
>
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/537654FF.2000502%40badapple.net.

[Jesse Cotton]

We're aware of most of this and agree with most of this. However when you always call include, you lose the ability to say a particular hiera variable is attached to the profile. For example

If you define:

class profile::apache_phpfpm {

  include ::apache

}

with the following in hiera:

apache::keepalive = 1

keepalive = 1 applies anywhere apache is included

vs

class profile::apache_phpfpm (

  $keepalive = 1

) {

  class { ::apache:

    keepalive => $keepalive

  }

}

profile::apache_phpfpm::keepalive = 1

So with the later you can build a somewhat self-contained profile. With the former you have to set variables "globally" or on a node.

That is what your data hierarchy is for.  Hiera does not limit you to only global and per-node data.  You can define as many hierarchy levels as you need, each grouping your nodes in its own way (though it usually makes the most sense for the groups at each level to be subsets of the groups at the next lower level).

Understood. However without creating a custom fact, I am not aware of a way to say 'apache::keepalive = 1' only applies when a node has the profile 'profile::apache_phpfpm'. 

And the foregoing is based on using only the built-in YAML back end.  Hiera supports pluggable back ends, usable together or separately.  A custom back end can employ whatever lookup or computation you want to serve whichever data you choose.

We're really trying to avoid this.

Duplicate declaration: Apache::Listen[80] is already declared in file ...

Yes, the class could be broken into separate classes but this only exacerbates the issue of assigning variables based on the profile.

How are others constructing their hierarchies when using the Role & Profile pattern? Are you assigning variables based on a nodes role or profiles? If so, how? Custom facts, parameterized classes as we've done in the example above?

John

[Christopher Wood]

It sounds like there's some generalizable rule where some hosts get certain settings (ex: keepalive) setting and some don't. Perhaps this could be broken out into another hiera layer? Cheapest possible example:

:hierarchy:
- 'nodes/%{fqdn}'
- 'roles/%{role}'
- 'common'

There's this workflow:

puppetmaster uses ENC to get the node's role (named "mytype" here)
the role is used to name the class for hiera_include (role::mytype)
mytype.yaml is also a yaml file as part of that hiera level

Then in the yaml for "phpfpm" and "needkeepalive" you have "apache::keepalive: true" and in common.yaml you have "apache::keepalive: false".

> And the foregoing is based on using only the built-in YAML back end. 
> Hiera supports pluggable back ends, usable together or separately.  A
> custom back end can employ whatever lookup or computation you want to
> serve whichever data you choose.
>
> We're really trying to avoid this.
>  
>
>  
>
>  
>
> It is thus possible to find and/or write composable classes and
> modules for managing the components you want.  As David Schmitt
> observes, composability is not automatic, but I don't see why your
> particular case of an apache-based PHP app server with firewall
> rules and a specific collection of PHP and apache modules should
> present any special problem.
>
> Thus, the answer to your question is "neither."  You build a drupal
> profile, and a wordpress profile, etc., and you use them all
> together.
>
> So below is an example for Drupal. It could literally be cloned for
> Wordpress and Joomla. Unfortunately, b/c of the class declarations, it
> is not composable.
> class profile::drupal (
>   $apache_listen = ['80'],
>   $apache_name_virtual_hosts = ['*:80'],
>   $apache_modules = ['fastcgi'],
>   $apache_fastcgi_servers = {
>     'www' => {

>       host => '[1]127.0.0.1:9000',

>       faux_path => '/var/www/php.fcgi',
>       alias => '/php.fcgi',
>       file_type => 'application/x-httpd-php'
>     }
>   },
>   $phpfpm_pools = {
>     'www' => {

>       listen  => '[2]127.0.0.1:9000',

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [3]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [4]https://groups.google.com/d/msgid/puppet-users/b6df42fd-0a33-4b63-ba8c-b56b54986d3e%40googlegroups.com.
> For more options, visit [5]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. http://127.0.0.1:9000/
> 2. http://127.0.0.1:9000/
> 3. mailto:puppet-users...@googlegroups.com
> 4. https://groups.google.com/d/msgid/puppet-users/b6df42fd-0a33-4b63-ba8c-b56b54986d3e%40googlegroups.com?utm_medium=email&utm_source=footer
> 5. https://groups.google.com/d/optout

[jcbollinger]

On Tuesday, May 20, 2014 12:54:15 PM UTC-5, Jesse Cotton wrote:

Understood. However without creating a custom fact, I am not aware of a way to say 'apache::keepalive = 1' only applies when a node has the profile 'profile::apache_phpfpm'. 

There is no good way to say that to Puppet.  It reflects a non-idiomatic thought process.  Moreover, it's unlikely to be what you really want to say -- you might in the future want other nodes to have keepalive enabled, too.  You should instead consider how to assign apache::keepalive: 1 to exactly those nodes that get profile::apache_phpfpm.

There are at least two ways you could approach that problem:

1. If you are using hiera to assign roles and/or profiles to nodes then it should be entirely straightforward to override apache::keepalive at the same hierarchy level where you assign profile::apache_phpfpm.
2. You can perform a resource parameter override in profile::apache_phpfpm.

If it is viable, the former is preferable to the latter.  A resource parameter override is a bit messy because it relies on implementation details of module 'apache'.  (It does not work to override class parameters; you must instead override one or more parameters of a bona fide resource, which can be of either defined or native type.)

 

And the foregoing is based on using only the built-in YAML back end.  Hiera supports pluggable back ends, usable together or separately.  A custom back end can employ whatever lookup or computation you want to serve whichever data you choose.

We're really trying to avoid this.

I don't blame you.  A custom back-end should certainly be a last resort.  As you keep that alternative in reserve, however, don't forget that such a custom component does not necessarily need to replace the YAML back-end; hiera can use two (or more) at the same time.

 

As I said, a duplicate resource.  As I said, factor it (and the others) out.  I think it's excellent practice to avoid resource declarations directly in profile classes, unless possibly of resources that are assuredly specific to the profile in which they appear.

 

Yes, the class could be broken into separate classes but this only exacerbates the issue of assigning variables based on the profile.

But that's just it: you don't want to assign data based (strictly) on profile.  Or if you do, then understand that it is inherently inconsistent with composable profiles.  Since the node characteristic that determines the combination of profiles in use is its role, it is on that basis that you want to assign data.  Indeed, that's what you already said with respect to apache::keepalive.  Note that with hiera, that doesn't mean you must avoid assigning data based on profile; it just means you must (also) be able to assign data based on role, with the role-based data having higher priority.


John

[Jesse Cotton]

But that's just it: you don't want to assign data based (strictly) on profile.  Or if you do, then understand that it is inherently inconsistent with composable profiles.  Since the node characteristic that determines the combination of profiles in use is its role, it is on that basis that you want to assign data.  Indeed, that's what you already said with respect to apache::keepalive.  Note that with hiera, that doesn't mean you must avoid assigning data based on profile; it just means you must (also) be able to assign data based on role, with the role-based data having higher priority.

I understand this better now however the problem we were really struggling with was how to assign variables based on role or profile. Ie, how do you incorporate the the role or profile into the hierarchy so a variable only applies to nodes that have a particular role or profile. Your first suggestion is good although thats not how we're currently assigning classes.