Best Practices/Style: add stuff to a file for each host?

[Robin Lee Powell]

(I'm going a bit more for philosophical discussion than practicality
here, maybe. Do at least feel free to think in terms of what Puppet
*should* do rather than tha fastest way to solve this problem.)

There's a pattern I've run into a lot recently mhere a config file
needs to be built based on information from a number of puppet
managed hosts. Assume here than I mean "node" in the puppet sense
when I say "host". *shrug*

Use cases:

- backups, where each host has directories that need to be backed
up, and things need to be done on the individual backup client
hosts to handle that, and *also* stuff needs to be done on the
backup master

- deploy configuration, where each host has a deploy role and it
makes most sense to talk about the deploy role in each host's
puppet config, but the file that manages the deployment is on
the deploy master host

- VM configuration, where information about a VM needs to affect
data/configuration stored on the host that holds that VM

All the same general pattern of action-at-a-distance: configuration
on a number of hosts affecting the master config file on a single
host.

Also, multi-part config files: in at least some of these cases, the
information from each host generates some configuration file output,
perhaps from a template, but all of those bits of config file need
to be merged together into one master config file.

These don't seem, to me, to be things puppet handles natively, and
(see my other posts) new types don't seem to be being added all that
often.

So, my question is this: what's the right way to handle this sort of
situation?

Feel free to break the action-at-a-distance bit and the
multipart-config-file-merge bit into seperate pieces, if you like.

Thanks so much for your time!

-Robin

--
http://singinst.org/ : Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is false"
is "na nei". My personal page: http://www.digitalkingdom.org/rlp/

[Al @ Lab42]

Hi

On Tuesday, January 11, 2011 5:51:07 AM UTC+1, Robin Lee Powell wrote:

(I'm going a bit more for philosophical discussion than practicality
here, maybe.  Do at least feel free to think in terms of what Puppet
*should* do rather than tha fastest way to solve this problem.)

There's a pattern I've run into a lot recently mhere a config file
needs to be built based on information from a number of puppet
managed hosts.  Assume here than I mean "node" in the puppet sense
when I say "host".  *shrug*

Use cases:

  - backups, where each host has directories that need to be backed
    up, and things need to be done on the individual backup client
    hosts to handle that, and *also* stuff needs to be done on the
    backup master

  - deploy configuration, where each host has a deploy role and it
    makes most sense to talk about the deploy role in each host's
    puppet config, but the file that manages the deployment is on
    the deploy master host

  - VM configuration, where information about a VM needs to affect
    data/configuration stored on the host that holds that VM

All the same general pattern of action-at-a-distance: configuration
on a number of hosts affecting the master config file on a single
host.

This is exactly the case when you need exported resources and, therefore, storeconfigs.
http://projects.puppetlabs.com/projects/1/wiki/Exported_Resources

Also, multi-part config files: in at least some of these cases, the
information from each host generates some configuration file output,
perhaps from a template, but all of those bits of config file need
to be merged together into one master config file.

You can build a file based on different "fragments" at least in 2 ways:
- When you specify an array of templates , when using the content => argument, these templates are actually appended in the defined order.
- With the puppet-concat module by Rip https://github.com/ripienaar/puppet-concat you can build up files based on different fragments (and you can define single fragmensts as exported resources so that they fit the need you expressed). Incidentally, I've started to use it just yesterday, to build named.conf in a bind module, and does exactly what it says.

Hope it helps
Af

[Nick Moffitt]

Al @ Lab42:

> You can build a file based on different "fragments" at least in 2 ways:
> - When you specify an array of templates , when using the content =>
> argument, these templates are actually appended in the defined order.
> - With the puppet-concat module by Rip
> https://github.com/ripienaar/puppet-concat you can build up files

> based on different fragments (and you can define single fragments as

> exported resources so that they fit the need you expressed).

I recently adopted puppet-concat, and found it my new big hammer for
almost everything. I had previously set up a define for a simple
header/body/footer structure where only those who cared would supply a
custom body template name as an argument. When it came time to add more
smarts, it was trivial to swap out the file resource in the define and
insert both a concat and a concat::fragment.

I now still have the old three-template API as well as a "provide any
fragments you like, but know that the header/body/footer have order
01/50/99" mechanism available.

One could also use Augeas, but I do not feel that it would fit the
original poster's desire to find a strong and helpful standard.
Augeas doesn't do well when you need to assert the state of a file
overall (which is at the heart of puppet's declarative nature) but with
responsibility divided among systems. It does much better when you need
to surgically assert one or two lines or stanzas in an otherwise
unmanaged file.

I'll just say that Augeas is named after a man who let his stables fill
up with so much crap that Hercules had to divert two whole rivers to
pressure-hose it clean again. You can't make this stuff up.

--
"These people program the way Victorians dress.
It takes two hours and three assistants to put on
your clothes, and you have to change before dinner.
But everything is modular." -- Miles Nordin, on PAM

[Robin Lee Powell]

On Tue, Jan 11, 2011 at 05:20:38AM -0800, Al @ Lab42 wrote:
>
> You can build a file based on different "fragments" at least in 2
> ways:
>
> - When you specify an array of templates , when using the content
> => argument, these templates are actually appended in the defined
> order.

Can you use exported resources to generate such an array?

[Robin Lee Powell]

On Tue, Jan 11, 2011 at 12:07:30PM -0800, Robin Lee Powell wrote:
> On Tue, Jan 11, 2011 at 05:20:38AM -0800, Al @ Lab42 wrote:
> >
> > You can build a file based on different "fragments" at least in
> > 2 ways:
> >
> > - When you specify an array of templates , when using the
> > content => argument, these templates are actually appended in
> > the defined order.
>
> Can you use exported resources to generate such an array?

Anybody? I can see how to generate resources from lots of hosts and
run it on one host, but I can't see how to generate a config file
out of all of that information.

[Felix Frank]

On 01/12/2011 05:08 PM, Robin Lee Powell wrote:
> On Tue, Jan 11, 2011 at 12:07:30PM -0800, Robin Lee Powell wrote:
>> On Tue, Jan 11, 2011 at 05:20:38AM -0800, Al @ Lab42 wrote:
>>>
>>> You can build a file based on different "fragments" at least in
>>> 2 ways:
>>>
>>> - When you specify an array of templates , when using the
>>> content => argument, these templates are actually appended in
>>> the defined order.
>>
>> Can you use exported resources to generate such an array?
>
> Anybody? I can see how to generate resources from lots of hosts and
> run it on one host, but I can't see how to generate a config file
> out of all of that information.

If you use a concat mechanism as described in this thread, you can have
each of your machines export a respective snippet and the target machine
realizes those and concatenates them to your config file.

HTH,
Felix

[Al @ Lab42]

On Wednesday, January 12, 2011 5:08:07 PM UTC+1, Robin Lee Powell wrote:

On Tue, Jan 11, 2011 at 12:07:30PM -0800, Robin Lee Powell wrote:
> On Tue, Jan 11, 2011 at 05:20:38AM -0800, Al @ Lab42 wrote:
> >
> > You can build a file based on different "fragments" at least in
> > 2 ways:
> >
> > - When you specify an array of templates , when using the
> > content => argument, these templates are actually appended in
> > the defined order.
>
> Can you use exported resources to generate such an array?

Anybody?  I can see how to generate resources from lots of hosts and
run it on one host, but I can't see how to generate a config file
out of all of that information.

-Robin

I give you an example I use for Nagios.

On each node I've something like:
nagios::host { $ fqdn }

This calls the custom define (ignore the various frills such the grouptag selector that manages different Nagios servers according to custom logic or the various nagios::params variables, that are just module's internal variables defined in the separated nagios::params class and concentrate on the fact this this defines just exports a file resource: @@file ):
define nagios::host (
    $ip = $fqdn,
    $short_alias = $fqdn,
    $use = 'generic-host',
    $nagios_parent = '',
    $ensure = 'present',
    $hostgroups = 'all' ) {

    require nagios::params

    @@file { "${nagios::params::customconfigdir}/hosts/${name}.cfg":
        mode    => "${nagios::params::configfile_mode}",
        owner   => "${nagios::params::configfile_owner}",
        group   => "${nagios::params::configfile_group}",
        ensure  => "${ensure}",
        require => Class["nagios::extra"],
        notify  => Service["nagios"],
        content => template( "nagios/host.erb" ),
        tag     => "${nagios::params::grouptag}" ? {
            ''       => "nagios_host",
            default  => "nagios_host_$nagios::params::grouptag",
        },
    }

}


On the Nagios server I 've (in order to collect all the exported reources with the relevant tag)
    case $nagios::params::grouptag {
        "": {
        File <<| tag == "nagios_host" |>>
        File <<| tag == "nagios_service" |>>
#       File <<| tag == "nagios_hostgroup" |>>
        }
        default: {
        File <<| tag == "nagios_host_$nagios::params::grouptag" |>>
        File <<| tag == "nagios_service_$nagios::params::grouptag" |>>
#       File <<| tag == "nagios_hostgroup_$nagios::params::grouptag" |>>
        }
    }

Now, I could place (and maybe I will) , in the nagios::host define something like:

    @@concat::fragment { "nagios_hosts_${name}.cfg":
        target => "${nagios::params::customconfigdir}/hosts/nagios_hosts.cfg",
        ensure  => "${ensure}",
        notify  => Service["nagios"],
        content => template( "nagios/host.erb" ),
        tag     => "${nagios::params::grouptag}" ? {
            ''       => "nagios_host",
            default  => "nagios_host_$nagios::params::grouptag",
        },

and in the nagios server class collect these fragments with:
    case $nagios::params::grouptag {
        "": {
        Concat::fragment <<| tag == "nagios_host" |>>
        Concat::fragment <<| tag == "nagios_service" |>>
#       Concat::fragment <<| tag == "nagios_hostgroup" |>>
        }
        default: {
        Concat::fragment <<| tag == "nagios_host_$nagios::params::grouptag" |>>
        Concat::fragment <<| tag == "nagios_service_$nagios::params::grouptag" |>>
#       Concat::fragment <<| tag == "nagios_hostgroup_$nagios::params::grouptag" |>>
        }
    }

In the same server I should also define the base concat file with something like:
concat { "${nagios::params::customconfigdir}/hosts/nagios_hosts.cfg":
        mode => 644,
        owner => root,
        group => root,
}


Consider that I've not tested this , and there could be syntax or other errors around .
Also I'm not so sure that is safe to have the double colons in the define name when collecting it ( Concat::fragment ) but the principle is that.

Remember also that all the variable or facts you use when exporting a resource (such as $fqdn ) are referred and valued on host that exports it and not to the one that collects them.

Hope it helped without adding confusion

Al