Strategies for "boring" packages

[J.T. Conklin]

At work, we've written about 120 modules in our puppet code repository.
About two dozen are "interesting", in that they have lots of parameters
and configuration that is specific to our environment. The balance are
"boring", rather they are mostly boilerplate with minimal configuration.
For example, our modules abstract the differences in package and service
names between RedHat and Debian based systems.

However, there is some disagreement amongst our puppeteers about how to
handle these "boring" modules. One side objects to the amount of boiler-
plate and duplication, and would prefer that we simply define packages
in our role/profile modules. The other side claims that abstracting
package and service names is value enough to justify the overhead, and
that "boring" packages often become "interesting" over time as new
requirements for flexibility and customization develop over time. Each
group is firmly convinced that their opinion is the right one.

So I throw the question to the puppet community... What strategies do
you use for "boring" modules so you're not overwhelmed by hundreds of
small boilerplate modules?

Thanks for sharing,

--jtc

[Thomas Müller]

Am Dienstag, 19. April 2016 03:07:34 UTC+2 schrieb J.T. Conklin:

At work, we've written about 120 modules in our puppet code repository.
About two dozen are "interesting", in that they have lots of parameters
and configuration that is specific to our environment.  The balance are
"boring", rather they are mostly boilerplate with minimal configuration.
For example, our modules abstract the differences in package and service
names between RedHat and Debian based systems.

I tend to prefere a module if there is any difference. So I would go with modules in your case. Just for the same reasoning as you wroter later on. 

I think it helps with the roles/profiles pattern too, as you can include the module wherever you want. whereas you would need to include a profile from a profile when there's no module - which is something what i'd like to avoid.

- Thomas

[Luke Bigum]

In my mind the "purest" way would be to go individual modules for each package/service combination. If the only requirement is that you are handling the differences between Red Hat and Debian flavours, then a module per package/service. These modules would be wholly self contained and rely on some of the standard set of Facter facts. And then you could publish them :-) It would also avoid future duplicate resource declarations where someone's embedded "packageX" into one profile, and it clashes with "packageX" in another profile.

I can see the argument for putting package installs and service starts into a profile but only if it's global for every operating system. So if there was profile::webserver that needed Package[openssl] and that was correct for all operating systems, then fine. However if you have to start doing conditional logic to find the right name of Package[openssl] for Red Hat and Debian, then profile::webserver is not the place. profile::webserver is a container of business logic that relates wholly and only to your business and your team. The exact implementation of Package[openssl] has nothing to do with profile::webserver, as long as openssl gets there somehow, that should be all you care about at the Profile level. Implementing Package[openssl] really depends on the operating system Facts alone, and this should be in it's own module... and... all of a sudden your profile::webserver is operating system agnostic, which is cool.

Question - why is it taking your team getting annoyed at generating boilerplate code? Surely you have some sort of "puppet module create" wrapper script or you use https://github.com/voxpupuli/modulesync? If you've got so much overhead adding boiler plate code for your boring modules then I think you're tackling the wrong problem... If you can bring the boiler plate code problem down to 1-2 minutes, it's got to only take another 5-10 minutes tops to refactor one package{} and one service{} resource out of the profile and into it's own module, and then your team argument kind of goes away.

Question - why are you writing 120 modules yourself? Are there really no other implementations of these things on the Forge or GitHub?

--
Luke Bigum

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/87shyio252.fsf%40wopr.acorntoolworks.com.
For more options, visit https://groups.google.com/d/optout.
---

LMAX Exchange, Yellow Building, 1A Nicholas Road, London W11 4AN
http://www.LMAX.com/

Recognised by the most prestigious business and technology awards

2016 Best Trading & Execution, HFM US Technology Awards
2016, 2015, 2014, 2013 Best FX Trading Venue - ECN/MTF, WSL Institutional Trading Awards

2015 Winner, Deloitte UK Technology Fast 50
2015, 2014, 2013, One of the UK's fastest growing technology firms, The Sunday Times Tech Track 100
2015 Winner, Deloitte EMEA Technology Fast 500
2015, 2014, 2013 Best Margin Sector Platform, Profit & Loss Readers' Choice Awards

---

FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved.

This message and its attachments are confidential, may not be disclosed or used by any person other than the addressee and are intended only for the named recipient(s). This message is not intended for any recipient(s) who based on their nationality, place of business, domicile or for any other reason, is/are subject to local laws or regulations which prohibit the provision of such products and services. This message is subject to the following terms (http://lmax.com/pdf/general-disclaimers.pdf), if you cannot access these, please notify us by replying to this email and we will send you the terms. If you are not the intended recipient, please notify the sender immediately and delete any copies of this message.

LMAX Exchange is the trading name of LMAX Limited. LMAX Limited operates a multilateral trading facility. LMAX Limited is authorised and regulated by the Financial Conduct Authority (firm registration number 509778) and is a company registered in England and Wales (number 6505809).

LMAX Hong Kong Limited is a wholly-owned subsidiary of LMAX Limited. LMAX Hong Kong is licensed by the Securities and Futures Commission in Hong Kong to conduct Type 3 (leveraged foreign exchange trading) regulated activity with CE Number BDV088.

[J.T. Conklin]

Luke Bigum <luke....@lmax.com> writes:
> In my mind the "purest" way would be to go individual modules for each
> package/service combination. If the only requirement is that you are
> handling the differences between Red Hat and Debian flavours, then a
> module per package/service. These modules would be wholly self
> contained and rely on some of the standard set of Facter facts. And
> then you could publish them :-) It would also avoid future duplicate
> resource declarations where someone's embedded "packageX" into one
> profile, and it clashes with "packageX" in another profile.
>
> I can see the argument for putting package installs and service starts
> into a profile but only if it's global for every operating system. So
> if there was profile::webserver that needed Package[openssl] and that
> was correct for all operating systems, then fine. However if you have
> to start doing conditional logic to find the right name of
> Package[openssl] for Red Hat and Debian, then profile::webserver is
> not the place. profile::webserver is a container of business logic
> that relates wholly and only to your business and your team. The exact
> implementation of Package[openssl] has nothing to do with
> profile::webserver, as long as openssl gets there somehow, that should
> be all you care about at the Profile level. Implementing
> Package[openssl] really depends on the operating system Facts alone,
> and this should be in it's own module... and... all of a sudden your
> profile::webserver is operating system agnostic, which is cool.

I agree 100%.

> Question - why is it taking your team getting annoyed at generating
> boilerplate code? Surely you have some sort of "puppet module create"
> wrapper script or you use https://github.com/voxpupuli/modulesync? If
> you've got so much overhead adding boiler plate code for your boring
> modules then I think you're tackling the wrong problem... If you can
> bring the boiler plate code problem down to 1-2 minutes, it's got to
> only take another 5-10 minutes tops to refactor one package{} and one
> service{} resource out of the profile and into it's own module, and
> then your team argument kind of goes away.

Lately we've been creating a new module every 3-4 weeks. So it's been
faster to copy an existing module, run a perl script that renames the
module, packages, and services, than it would be to write/adapt a script
to generate new modules from a template + parameters. It only takes me a
minute or two to create a new module. The counter-argument is that it
only takes a few seconds to add a "package { 'foo': }" to a profile, and
that a module per package/service leads to a unmanagable set of hundreds
of modules.

While I'm in the camp that separate modules for each package/service are
a good thing, I started this thread in order to double check my opinion.

> Question - why are you writing 120 modules yourself? Are there really
> no other implementations of these things on the Forge or GitHub?

In some cases we've found existing modules, we even use a few. But in
the general case, we've found it useful to write our own modules so they
have the same "look and feel" i.e. use the same sets of parameters and
facts with the same semantics. Our basic package/service boilerplate is
based off the example42 modules (at least as they were ~2-3 years ago).

--jtc

[Ramin K]

At the previous job I preferred to promote packages to full modules
from a holding module.

class vpackages {
include vpackages::params

@package { 'curl': tag => 'utils',}
@package { 'htop': tag => 'utils',}
@package { 'iftop': tag => 'utils',}
@package { 'whois': tag => 'utils', name =>
$vpackages::params::whois, }

# Mysql packages
@package { 'percona-toolkit': tag => 'mysql', }
@package { 'mysql-client': tag => 'mysql', name =>
$vpackages::params::mysqlclient, }
}

mysql-client is good example.

class mysql::install {
package { 'mysql-server':
ensure => present,
name => $mysql::params::packagename,
}

# realize packages like the client
Package<| tag == 'mysql' |>
}

Later our Mysql installations were more complicated in several different
roles with more versions. At that point it made sense to move
mysqlclient into it's own parameterized module.

What I liked about the vpackages modules is that it's a visual
representation of the shared namespace.

Ramin

[jcbollinger]

On Monday, April 18, 2016 at 8:07:34 PM UTC-5, J.T. Conklin wrote:

So I throw the question to the puppet community... What strategies do
you use for "boring" modules so you're not overwhelmed by hundreds of
small boilerplate modules?

Although it's orthogonal to the question, whether you create separate modules for group things together into large collection modules, you could look to the Example42 Tiny Puppet module to minimize your boilerplate.  That is its express purpose.  I don't have enough information to determine how well it actually would meet your needs, but it's certainly worth a look.  If indeed it succeeds in cutting your "boring" modules to a few lines of code then that might change the equation for you.


John

[Dirk Heinrichs]

Am 19.04.2016 um 02:47 schrieb J.T. Conklin:

What strategies do you use for "boring" modules so you're not overwhelmed by hundreds of small boilerplate modules?

I use a simple module with one class for mass-installing packages and another one for mass-uninstalling, like:

class linux_base::install ($packages = undef) {
  $packages.each |String $pkg| {
    package { $pkg: ensure => latest, }
  }
}

where $packages is an array of package names (strings), which can easily be provided for each (group of) host(s) via Hiera.

The ::uninstall class has "purge" instead of "latest". Of course, input array for both classes must be mutually exclusive, if both are used at the same time.

HTH...

    Dirk

--

Dirk Heinrichs, Senior Systems Engineer, Engineering Solutions
Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach
Tel: +49 2226 1596666 (Ansage) 1149
Email: d...@recommind.com
Skype: dirk.heinrichs.recommind
www.recommind.com