Re: [Puppet Users] sourceselect all

[Martin Alfke]

Hi Bernardo,


On 12.11.2012, at 20:00, Bernardo Costa wrote:

> I am using puppet-2.6.17 from a centos 5.x box and after reading the documentation (http://docs.puppetlabs.com/references/2.6.17/type.html#file) I feel the need of using the sourceselect parameter set to all for copying/concatenating contents from multiple files to a single one. But it miserably fails on this issue. Below is an example of configuration I have tried without success. The idea is to create a base class for common contents of an automount map and also two specialized classes for particular mount points of each node. I believe with "sourceselect => all" it would be possible but I couldn't see it working. Any ideas of why ...?
>
> File {
> ensure => file,
> group => "root",
> owner => "root",
> mode => "0644",
> }
>
> class automount {
> file { "/var/tmp/auto.direct":
> source => "puppet:///files/automount/basic.conf",
> sourceselect => "all",
> }
> }
>
> class automount::itchy inherits automount {
> File['/var/tmp/auto.direct'] {
> source => [ "puppet:///files/automount/itchy.conf", "puppet:///files/automount/basic.conf" ],
> sourceselect => "all",
> }
> }
>
you want to add the new sources in the inherited classes?

e.g. in automount::itchy

source +> ["puppet:///files/automount/itchy.conf", "puppet:///files/automount/basic.conf"],

Sidenote:
when using modules the source has another syntax:

puppet:///modules/<modulename>/<filename>

you can omit the "files" path

HTH

Martin

[Bernardo Costa]

Martin, thank you for your prompt answer. As soon as I get this working I'll try to use modules instead of files.
But anyway, trying to switch from "source =>" to "source +>" didn't make any difference even with "sourceselect => all". The example above still just fetches the first one listed in source attribute. Is it only available at some particular version of puppet maybe ?

File {
    ensure => file,
    group  => "root",
    owner => "root",
    mode  => "0644",
}

class automount {
    file { "/var/tmp/auto.direct":

        sourceselect => all,

        source => [ "puppet:///files/automount/basic.conf" ],
    }  
}
 

class automount::itchy inherits automount {
    File['/var/tmp/auto.direct'] {

        sourceselect => all,

        source +> [ "puppet:///files/automount/itchy.conf", "puppet:///files/automount/basic.conf" ],
    }  
}

class automount::scratchy inherits automount {
    File['/var/tmp/auto.direct'] {
        sourceselect => all,
        source +> [ "puppet:///files/automount/scratchy.conf", "puppet:///files/automount/basic.conf" ],
    }  
}

node default {
    include automount
}
 
node 'scratchy' inherits default {
    include automount::scratchy
}

node 'itchy' inherits default {
    include automount::itchy

[jcbollinger]

On Monday, November 12, 2012 1:00:14 PM UTC-6, Bernardo Costa wrote:

I am using puppet-2.6.17 from a centos 5.x box and after reading the documentation (http://docs.puppetlabs.com/references/2.6.17/type.html#file) I feel the need of using the sourceselect parameter set to all for copying/concatenating contents from multiple files to a single one. But it miserably fails on this issue. Below is an example of configuration I have tried without success. The idea is to create a base class for common contents of an automount map and also two specialized classes for particular mount points of each node. I believe with "sourceselect => all" it would be possible but I couldn't see it working. Any ideas of why ...?

The 'sourceselect' parameter doesn't do what you think it does.  It applies only to File resources that manage directories recursively, where it directs Puppet to combine the contents of all the specified 'source' directories that exist.  Furthermore, "combine" in that case applies at the level of identifying which files to manage, from which sources; it does not produce concatenation under any circumstances.

 

File {
    ensure => file,
    group  => "root",
    owner => "root",
    mode  => "0644",
}

class automount {
    file { "/var/tmp/auto.direct":
        source => "puppet:///files/automount/basic.conf",
        sourceselect => "all",
    }  
}
 
class automount::itchy inherits automount {
    File['/var/tmp/auto.direct'] {
        source => [ "puppet:///files/automount/itchy.conf", "puppet:///files/automount/basic.conf" ],
        sourceselect => "all",
    }  
}

class automount::scratchy inherits automount {
    File['/var/tmp/auto.direct'] {

        source => [ "puppet:///files/automount/scratchy.conf", "puppet:///files/automount/basic.conf" ],
        sourceselect => "all",
    }  
}

node default {
    include automount
}
 

node 'itchy' inherits default {
    include automount::itchy
}

node 'scratchy' inherits default {
    include automount::scratchy
}

Some of your options are:

1. Make puppet:///files/automount/scratchy.conf and the like complete configuration files rather than supplemental file fragments
2. Use 'content' instead of 'source', and create the content via the template() function (even if only to perform the concatenation when you want it)
3. Use the Concat module to build your files from fragments where needed (though this is really a wrapper around the above)

Note that although option (1) will produce some duplication of your config files, it could greatly simplify your manifests.  Consider:

class automount {
  file { "/var/tmp/auto.direct":
    source => [

      "puppet:///files/automount/${hostname}.conf",
      'puppet:///files/automount/basic.conf"]
  }
}

No subclasses or special node blocks needed.

In fairness, options (2) and (3) could also be written to avoid subclassing and special node blocks, but not so simply and easily as it can be done with option (1).


John

[Bernardo Costa]

Ok John, it looks like sourceselect exists for handling files like the ones under /etc/profile.d which is not the case for automount. Good to know. About the options I have, I'd prefer not doing #1 for two reasons: one is that many mountpoints would be duplicated toward many files as they are common for all hosts or for a great number of them, and two is that I'd like to reduce the number of different automount maps managed which makes me avoid doing a one map for each host policy. But the cost of doing things this way is managing hosts as a group of them and having to treat the automount maps as incremental although it is not like this.

I think I have seen an example of #2 and just could make it work int this way but I can't remember of having seen any example of #3. Is there any advantage on trying way #3 ? If I do it, I believe I'll ne to install this concat module. Where is it available ?

[jcbollinger]

On Tuesday, November 13, 2012 11:37:12 AM UTC-6, Bernardo Costa wrote:

Ok John, it looks like sourceselect exists for handling files like the ones under /etc/profile.d which is not the case for automount. Good to know. About the options I have, I'd prefer not doing #1 for two reasons: one is that many mountpoints would be duplicated toward many files as they are common for all hosts or for a great number of them, and two is that I'd like to reduce the number of different automount maps managed which makes me avoid doing a one map for each host policy.

Let me be sure you understand that my suggestion (1) results in exactly one file being managed on each client, and it requires a per-node file on the master only for those nodes that require a file different from some standard one.  It is therefore a fairly good choice both when most of your nodes want the same mountpoints (because you need special-case files only for a few nodes), and at may be a good choice when most nodes are different -- even if there is a shared core of common mount points -- because you'll need special-case files for most nodes anyway.  Of course, you may be somewhere in between, or the suggestion may just not suit you.

 

But the cost of doing things this way is managing hosts as a group of them and having to treat the automount maps as incremental although it is not like this.

I don't understand what you mean by that.  In particular, your use of "incremental" makes me suspect a misunderstanding: my option (1) does not involve concatenation of source files.  Puppet will sync the target file with the first file from its 'source' list that exists.  Thus, for those nodes that have a corresponding puppet:///files/automount/${hostname}.conf file, that file will be the source used, and for all other nodes puppet:///files/automount/basic.conf would be used instead.

 

I think I have seen an example of #2 and just could make it work int this way but I can't remember of having seen any example of #3. Is there any advantage on trying way #3 ? If I do it, I believe I'll ne to install this concat module. Where is it available ?

Option 3 (using the Concat module) makes the most sense if you are building the file from a relatively small number of possible blocks (each of which could define any number of mount points).  If your mount points fall into natural groups by role, location, department, or such then modeling them as file fragments via Concat might really clarify your manifests.

Alternatively, it might work out well to have the "basic" mount points in one static fragment, and everything else in one optional, template-built fragment.

On the other hand, although Concat is a useful module, it could be a lot heavier than you really need.  You'll need to decide for yourself whether it's worth it.  You should be able to find some talk about it in the archives of this group, and you can find the module itself on the Forge.


John

[Bernardo Costa]

Well, what I could understand from your proposition (1) is that I should elect one of the hosts as representative of a group of hosts and make all other hosts inherit its configuration like the example I am showing below.

class automount {
  file { "/var/tmp/auto.direct":
    source =>"puppet:///files/automount/${hostname}.conf",

   }
}

node default {
        include automount
}

node itchy1, itchy2, itchy3 inherits default {
  file { "/var/tmp/auto.direct":
    source =>"puppet:///files/automount/itchy.conf",
   }
}

What I'd like to avoid using this scheme of concatenating templates is changing many files when only one entry is changed. Supposing many hosts mount a specific directory, if the provider file of this directory changes, then I'd need to use a sed script to change only one entry. In fact, this is a complex issue. Maybe the best thing is not putting every direct mountpoints in only one map/file but splitting them in maybe two, three or four. For this, I'd need a auto.master file like the one below.

/- /etc/auto1.direct
/- /etc/auto2.direct
/- /etc/auto3.direct
/- /etc/auto4.direct

Your idea of defining a basic map and just override the basic if a particular map exists with the name of the host is interesting but for my case, I'd need almost one particular map for each host. Most of the hosts have particular direct mountpoints. Although this is a fairly complex thing to configure in all hosts, I am trying to find the most simple and elegant solution :) For now, I am trying to avoid using concat module as I prefer to have the bare minimum of puppet features and not many dependences on external tools. Anyway, thank you for you help.

[Jakov Sosic]

If I understood you correctly, you have some standard blocks that need
to be present on most of the hosts, but almost every host has different
final concatenated configuration file?

For example, you have blocks:

block1
block2
block3
block4
host1block1
host1block2
host2block1
host3block1

You could name the files something like that on the master, and then use
generate() in your manifest. You would pass $::fqdn to generate script,
which runs on master, takes all of the blocks and builds final file
which will be shipped to a node. For example:

if generate('/etc/puppet/modules/costa/scripts/generate_file.sh',$::fqdn) {
file {'/var/tmp/auto.direct':
source => "puppet:///files/${::fqdn}/auto.direct",
}
}

generate_file.sh script could look something like this:

#!/bin/bash

# standard for all hosts:
cat block1 >> /etc/puppet/files/$HOSTNAME/auto.direct
cat block2 >> /etc/puppet/files/$HOSTNAME/auto.direct
cat block3 >> /etc/puppet/files/$HOSTNAME/auto.direct
cat block4 >> /etc/puppet/files/$HOSTNAME/auto.direct

HOSTNAME="$1"
for i in `ls /etc/puppet/modules/costa/files/fragments | grep $HOSTNAME`; do
cat $i >> /etc/puppet/files/$HOSTNAME/auto.direct
done


If that solution is not good for your particular case, I suggest you
rewrite automount startup script, so that on each (re)start and refresh
it removes automount.conf and concatenates all files from "automount.d"
into new config file. That way your puppet manifest would be really
clean and readable, and you would only push building blocks as files and
not the finished configuration file.

That kind of solution Samba developers suggest to those who dare to ask
when and if they will develop/implement "samba.d" behaviour :)