module design: different module sections manipulating files in single directory

[dkoleary]

Hey;

I suspected this was going to be a problem and, sure enough, it is.  

Here's the scenario:  puppet server 4.5:  I have ~ 1200 hosts on which I want specific files in /root/bin on all hosts.  A reasonably large subset of those should have additional files in /root/bin as part of an home-grown application management process.  To be clear, none of the files from the 'all-host' group overlap with any of the files from the 'some-hosts' group.

The all-host group is easy enough::

  file { '/root/bin':
    ensure  => 'directory',
    owner   => 'root',
    group   => 'root',
    mode    => '0700',
    recurse => true,
    source  => 'puppet:///modules/myroot/rootbin',
    require => File['/root'],
  }

So, that's worked for weeks now.  In my company's slow migration to puppet management, I'm finally to the point of adding some custom application related files to /root/bin.  On the surface, the some-hosts group is pretty easy too::

    file { 'webconfbin':
      ensure  => 'directory',
      path    => '/root/bin',
      owner   => 'root',
      group   => 'root',
      mode    => '0700',
      recurse => true,
      source  => 'puppet:///modules/myroot/webconf',
    }

As I suspected, that resulted in the bright red error message about 'resource /root/bin already declared'.  The two options that I can think of aren't particularly appetizing:

1.  Add the files from some-hosts to all-hosts resulting in the app management files being everywhere.  These files, themselves, don't represent a security issue, but it's not a very clean approach.

2.  Use individual file resources.  I could get away with that approach on this one; but, when I run into a similar issue with dozens or 100s of files, I'd hate to be specifying all those file resources.

Realizing I probably took a wrong turn in my initial design and figuring someone else has to have had run into this problem before, I'm asking the experts.  What's the right way to have a set of files on all hosts and a different set of files on a subset of all hosts in the same directory?

Thanks for any hints/tips/suggestions.

Doug O'Leary

[Dan Mahoney]

I don't often comment on the puppet stuff, but yours made me need to chime
in and say this:

Recurse is an ugly, awful, terrible hack and should be deprecated.

I don't say that with any knowledge of the way it evolved or what its
future support status is, but if you look at it -- it's effectively an
expansion macro that turns into dozens or hundreds of File[] resources
(and interally -- can and MUST grow your manifest internally in-memory to
that size).

Judging by the fact that you're using a /bin directory to distribute
things, which I assume are binaries, or scripts, the thing that makes
sense here (especially with 1200 hosts) is to look in to using your OS's
package manager to accomplish this task -- where you could, possibly,
break out the installables by host-class. (i.e. the files in
yoursite-db.rpm would require the files in yoursite-common.rpm)

You could, then, use puppet to manage a local installable of that
distributable, with a notify action that ran the installer -- or you could
use the builtin package resource type with a local repo, and use require
=> latest to upgrade that, if you have a high change delta.

(Where I say RPM, subsitute OS package manager of choice, obviously).

Yes, this steps outside of puppet, but on its face, puppet is *config*
management, and what you seem to be trying to do, is not.

--

[Rob Nelson]

Doubt,

I agree with Dan, packaging is the answer. And before you say it - yes, packaging sounds scary at first, but it doesn't have to be. Check out FPM - https://github.com/jordansissel/fpm/wiki - to generate a package in the correct format. You can very easily package static files that way, and use file resources with `source => template(...)` for any dynamic files required.

Hosting the files is pretty easy if you're using yum, as yumrepos are built in. You can host them on a node with profile::yumrepo (https://github.com/puppetinabox/controlrepo/blob/production/dist/profile/manifests/yumrepo.pp and https://github.com/puppetinabox/controlrepo/blob/production/hiera/puppet_role/yumrepo.yaml), throw the rpms in /var/www/html/puppetrepo/el7, and then ensure your base profile distributes that repo (https://github.com/puppetinabox/controlrepo/blob/production/dist/profile/manifests/base.pp#L29-L38). That code is dated and needs a little improvement (stop using `create_resources()`!) but should get you started quickly. I'm sure there's an equivalent for Apt.

Rob Nelson
rnel...@gmail.com

[dkoleary]

Hey;

Thanks for the responses.  I hadn't thought of packages.  I will start exploring that option

Thanks again.

Doug O'Leary

[Dan Mahoney]

On Mon, 15 Aug 2016, dkoleary wrote:

> Hey;

> Thanks for the responses.  I hadn't thought of packages.  I will start
> exploring that option

What would work, as a short term option:

* A class that just owns /root/bin

* Separate classes that create things in directories under /root/bin (say
/root/bin/webserver), and requires the above class. Similar to how a core
class may own /etc/apache/conf.d, but others may own things under.

* If you absolutely need them, symlinks, either via individual file
resources, or a script that basically goes through and updates them, as a
notify exec.

What you haven't said, is how many files are involved in your recurse, so
it's hard to say how well this would work for you.

Also an ugly hack, but may be more easily available than learning to
package overnight.

-Dan

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visithttps://groups.google.com/d/msgid/puppet-users/9b736b27-6c5b-42f3-b265-da72
> 50a682f4%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

--

"You're not normal!"

-Michael G. Kessler, referring to my modem online time.


--------Dan Mahoney--------
Techie, Sysadmin, WebGeek
Gushi on efnet/undernet IRC
ICQ: 13735144 AIM: LarpGM
Site: http://www.gushi.org
---------------------------

[dkoleary]

Hey

I went with my option #2 for this exercise. I will definitely be looking into the package management as it'll come up. In fact, it may have already come up in a different request. For this one, though, I have something like 5 scripts in the 'all-hosts' list and 3 in the subset so individual file resources were easy enough. It works. Not pretty by any stretch but that's what process improvement's for.

Thanks again for the responses.

Doug O'Leary