node definition too big in mutualized hosting context

[mezca...@gmail.com]

Hi,

I use puppet to manage servers in a mutualized hosting context and I get some trouble concerning the node definitions.

For example on dns nodes I have to declare many `bind::zone` resources like this:

bind::zone { ['website1.com']:
  expire => 604800,
  minimum => 3600,
  ttl => 38400,
  …
  records => [
    'www        IN    A    xxx.xxx.xxx.xxx'
    …
        ]
}

bind::zone { ['website2.com']:
  expire => 604800,
  minimum => 3600,
  ttl => 38400,
  …
  records => [
    'www        IN    A    xxx.xxx.xxx.xxx'
    …
        ]
}

… and hundred others like this.

Finally, the file containing node definition became very big and difficult to maintain.
I have the same issue for a webserver with apache::vhost resources for example.

For the moment I use a workaround like this:

import vhosts/*.pp

And in each files I put a condition to make a restriction for a particular nodes like this:

if $::fqdn =~ /^web(\d*)\.example\.net$/ {
    apache::vhost { 'website2':
    …
    }
}

I suppose that many puppet users use puppet to manage nodes in mutualized hosting context and I would ask if someone have a better solution for this issue?

Regards,

Thomas

[Mason Turner]

Assuming all the dns servers are getting the same config, We break big chunks like that out into their own modules. So:

node dns1.myco.com {

  include myco::dns

}

node web1.myco.com {

  include myco::web

}

If the servers have different configs, you can make myco::dns::common, myco::dns::this myco::dns::that myco::dns::other.

We also use a "role" fact instead of a regex on the fqdn for case statements, but even those are fragile. I prefer to explicitly include things in the node definitions. It's a bit verbose, but its also clear for anyone else that has to maintain the code.

— Mason Turner

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Ramin K]

On 8/7/2013 6:55 AM, mezca...@gmail.com wrote:
> Hi,
>
> I use puppet to manage servers in a mutualized hosting context and I get
> some trouble concerning the node definitions.
>
> For example on dns nodes I have to declare many `bind::zone` resources
> like this:
>
> bind::zone { ['website1.com']:
> expire => 604800,
> minimum => 3600,
> ttl => 38400,

> �

> records => [
> 'www IN A xxx.xxx.xxx.xxx'

> �
> ]

> }
>
> bind::zone { ['website2.com']:
> expire => 604800,
> minimum => 3600,
> ttl => 38400,

> �

> records => [
> 'www IN A xxx.xxx.xxx.xxx'

> �
> ]
> }
>
> � and hundred others like this.

>
> Finally, the file containing node definition became very big and
> difficult to maintain.
> I have the same issue for a webserver with apache::vhost resources for
> example.
>
> For the moment I use a workaround like this:
>
> import vhosts/*.pp
>
> And in each files I put a condition to make a restriction for a
> particular nodes like this:
>
> if $::fqdn =~ /^web(\d*)\.example\.net$/ {
> apache::vhost { 'website2':

> �

> }
> }
>
> I suppose that many puppet users use puppet to manage nodes in
> mutualized hosting context and I would ask if someone have a better
> solution for this issue?

You might consider using create_resources with hiera. Here's a simple
example.

yaml data
---
apache::a2mods:
expires: {}
gnutls: {}
headers: {}
rewrite: {}
apache::vhosts:
statsstage.example.com:
priority: '99'
a_template: 'apache/vhosts/stats.example.com.erb'
stage.example.com:
priority: '00'

And the class the queries Hiera and pumps the data through various defines.

class profile::apache {

include ::apache
include logrotate::apache

$mymods = hiera('apache::a2mods', {})
create_resources('apache::a2mod', $mymods)

$myvhosts = hiera('apache::vhosts', {})
create_resources('apache::vhost', $myvhosts)

}

Ramin

[mezca...@gmail.com]

Le mercredi 7 août 2013 16:13:26 UTC+2, Mason a écrit :

Assuming all the dns servers are getting the same config, We break big chunks like that out into their own modules. So:

node dns1.myco.com {

  include myco::dns

}

node web1.myco.com {

  include myco::web

}

If the servers have different configs, you can make myco::dns::common, myco::dns::this myco::dns::that myco::dns::other.

We also use a "role" fact instead of a regex on the fqdn for case statements, but even those are fragile. I prefer to explicitly include things in the node definitions. It's a bit verbose, but its also clear for anyone else that has to maintain the code.

— Mason Turner

 
Thanks for your answer.

In your solution you put resources declarations in an external class and the node definition stay small but your class which contain all resources declarations should be very big.
Finally you just moved the issue in an other place or maybe I miss something?

Regards,

Thomas

[mezca...@gmail.com]

You might consider using create_resources with hiera. Here's a simple
example.

yaml data
---
apache::a2mods:
   expires: {}
   gnutls:  {}
   headers: {}
   rewrite: {}
apache::vhosts:
     statsstage.example.com:
         priority:   '99'
         a_template: 'apache/vhosts/stats.example.com.erb'
     stage.example.com:
         priority:   '00'

And the class the queries Hiera and pumps the data through various defines.

class profile::apache {

   include ::apache
   include logrotate::apache

   $mymods = hiera('apache::a2mods', {})
   create_resources('apache::a2mod', $mymods)

   $myvhosts = hiera('apache::vhosts', {})
   create_resources('apache::vhost', $myvhosts)

}

Ramin

Thanks for your answer.

In your example using hiera, you put all vhosts config data in one yaml file. This solution seems to be maintainable in this case but if I use this solution for bind::zone datas the yaml file still be very big and difficult to maintain.
Is it possible with hiera to split this config data in multiple files which contain each bind::zone resource for example?

Regards,

Thomas

[jcbollinger]

On Thursday, August 8, 2013 4:48:02 AM UTC-5, mezca...@gmail.com wrote:

In your example using hiera, you put all vhosts config data in one yaml file. This solution seems to be maintainable in this case but if I use this solution for bind::zone datas the yaml file still be very big and difficult to maintain.

Is it possible with hiera to split this config data in multiple files which contain each bind::zone resource for example?

Hiera does not come with built-in support for what I think you want.  It does, however, support pluggable data source back ends, and it ought to be fairly straightforward to build a custom one that handles whatever data format you prefer.

Still, I think you're being a bit inconsistent here.  I don't see a big difference between splitting your zone data among multiple classes or modules as Mason suggested on one hand, and splitting them among multiple Hiera data files on the other.

Your core problem is that you have big data, and nothing we can suggest will change that.  So, what criteria are important to you in comparing potential solutions?  Without a better idea of what you're after, we're just shooting in the dark.


John

[mezca...@gmail.com]

Your core problem is that you have big data, and nothing we can suggest will change that.  So, what criteria are important to you in comparing potential solutions?  Without a better idea of what you're after, we're just shooting in the dark.

I have big data and I understand I can't make it disappear but I just want to know if there is any way to not put all these datas in the same file.
Maybe I misunderstood but the solution of Ramin or Mason seem to just "move" all these datas in a other place/file.

In short, I have to declare few hundred of resources on a node and each resources potentialy have specific param/datas.
And my criteria is that I don't want to put all these declarations in a single file because it is too difficult to maintain.

Thomas

[jcbollinger]

On Thursday, August 8, 2013 11:40:45 AM UTC-5, mezca...@gmail.com wrote:

Your core problem is that you have big data, and nothing we can suggest will change that.  So, what criteria are important to you in comparing potential solutions?  Without a better idea of what you're after, we're just shooting in the dark.

I have big data and I understand I can't make it disappear but I just want to know if there is any way to not put all these datas in the same file.
Maybe I misunderstood but the solution of Ramin or Mason seem to just "move" all these datas in a other place/file.

Yes, you have misunderstood especially Mason's response.
 

In short, I have to declare few hundred of resources on a node and each resources potentialy have specific param/datas.
And my criteria is that I don't want to put all these declarations in a single file because it is too difficult to maintain.

You can divide your resource declarations among multiple classes and have each node declare (only) the classes it needs.  This is part of what Mason was trying to tell you.  Each class should be recorded in a separate file anyway, so that achieves splitting up your data among multiple files.  Each class may contain as few or as many resource declarations as you like.  Also, if there is any overlap between the resources needed by different nodes then you can reduce the overall volume of data by merging the common bits into classes used by multiple nodes:

modules/zones/manifests/website1_com.pp:
----
class zones::website1_com {

  bind::zone { 'website1.com':
    expire => 604800,
    minimum => 3600,
    ttl => 38400,

    …

    records => [
      'www        IN    A    xxx.xxx.xxx.xxx'

      …
        ]
  }
}

modules/zones/manifests/website2_com.pp:
----
class zones::website2_com {

  bind::zone { 'website2.com':
    expire => 604800,
    minimum => 3600,
    ttl => 38400,

    …
    records => [
      'www        IN    A    yyy.yyy.yyy.yyy'
      …
          ]
  }
}

manifests/site.pp (or wherever you store your node definitions):
----
node /.*\.website1\.com$/ {
  include zones::website1_com
}

node /.*\.website2\.com$/ {
  include zones::website2_com
}

You can do the same thing with vhosts, of course.

Alternatively, if individual nodes are not unreasonably large by themselves, then it might suit you to just split your node declarations into separate files.  That's actually pretty common:

manifests/site.pp:
----
import 'nodes/*.pp'

manifests/nodes/website1_com.pp
----
node /.*\.website1\.com$/ {

  bind::zone { 'website1.com':
    expire => 604800,
    minimum => 3600,
    ttl => 38400,

    …

    records => [
      'www        IN    A    xxx.xxx.xxx.xxx'

      …
        ]
  }

  apache::vhost { ... }

  ...
}

manifests/nodes/website2_com.pp
----
node /.*\.website2\.com$/ {

  bind::zone { 'website2.com':
    expire => 604800,
    minimum => 3600,
    ttl => 38400,

    …
    records => [
      'www        IN    A    yyy.yyy.yyy.yyy'
      …
        ]
  }

  apache::vhost { ... }

  ...
}

That's similar to what you're doing now, but it's safer.  Of course, you can combine those approaches, too.

Do note that Puppet's 'include' function is not analogous to similarly-named directives in some programming languages.  It does not interpolate files; rather, it declares that the named class should be included in the target node's catalog.


John

[MezcalitoSCOP grenoble]

Hi Jhon,

Sorry, I just remark that I completely miss your answer a long time ago.

Le vendredi 9 août 2013 15:54:35 UTC+2, jcbollinger a écrit :

You can divide your resource declarations among multiple classes and have each node declare (only) the classes it needs.  This is part of what Mason was trying to tell you.  Each class should be recorded in a separate file anyway, so that achieves splitting up your data among multiple files.  Each class may contain as few or as many resource declarations as you like.  Also, if there is any overlap between the resources needed by different nodes then you can reduce the overall volume of data by merging the common bits into classes used by multiple nodes:

Ok I understand this example: defining zones classes files in a dedicated module and declaring these classes in the node definition. Its probably the best solution for me.

But you speak about declaring classes only in nodes that need it and in my case their is only one node (a DNS server) that must handle all bind zones. So I have to declare all classes in the same node file, right?

Thanks again for the help and sorry again for my late answer.

Regards,

Thomas

[jcbollinger]

To "declare" a class means this

include my_module::my_class

or this

class { 'my_module::my_class':
  # maybe some parameters ...
}

.

You can also use the 'require' or 'contain' function in place of 'include' when such a declaration appears within the definition of another class; these two forms also have effects on class relationships.  For a class to be applied to a given node, that class must be included in the node's catalog, and that is the main function of a class declaration such as the examples above.  The needed declaration(s) may appear directly in the node block (if any) selected for the node, or at top scope, or in the definitions of other classes assigned to the node, or any mix of those.

The definitions of the required classes do not need to appear in the same manifest file, nor should they do.  Class "definitions" are these:

class zones::website1_com {

  bind::zone { 'website1.com':
    expire => 604800,
    minimum => 3600,
    ttl => 38400,

    …

    records => [
      'www        IN    A    xxx.xxx.xxx.xxx'

      …
        ]
  }
}

Does that clear things up?


John