How to handle multiple modules requiring the same packages

[bel]

We have a database module and an application module that both require
gcc. When I include both modules in a node, I get errors about the
package being already defined.

How would I address this?

[Nigel Kersten]

Move the gcc package definition to a third module/class.

Tell both the db and app module that they require Class['gcc'].

Does that make sense?

 

--
Nigel Kersten
Product Manager, Puppet Labs

[Luke Bigum]

It sounds like you've got this written down twice:

package { "gcc": ensure => installed }

Causing a duplicate resource definition. To get around this, move the
gcc package install into a module of it's own, called say "gcc", then in
your database and application modules simply "include gcc" to pull in
your new gcc module as a requirement. You will also need to look at
requirements (http://docs.puppetlabs.com/learning/ordering.html).

If you prefer to declare your classes like this:

class { "gcc": }

Then you have to think about how to organise your node definitions a
little bit differently as you'll run into the same Duplicate Definition
error.

--
Luke Bigum
Information Systems
+44 (0) 20 3192 2520
Luke....@lmax.com | http://www.lmax.com
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN

The information in this e-mail and any attachment is confidential and is intended only for the named recipient(s). The e-mail may not be disclosed or used by any person other than the addressee, nor may it be copied in any way. If you are not a named recipient please notify the sender immediately and delete any copies of this message. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any view or opinions presented are solely those of the author and do not necessarily represent those of the company.

[Ramin K]

Usually creating a separate class is best, but for odd single packages
I'll add them to a single class as virtual resources.

oneoffpackages {
@package {'mysql-client'}
@package { 'perl-Getopt-Long-Descriptive': }
}

Then when I want to use that package I'll realize it.

nrpe::delllperc {
blah blah
realize Package['perl-Getopt-Long-Descriptive']
}

collectd::redis {
blah blah
realize Package['perl-Getopt-Long-Descriptive']
}

hostgroup::frontend {
blah blah
realize Package['mysql-client']
}

mysql {
blah blah
realize Package['mysql-client']

I'm not sure it's any cleaner or easier to maintain, but it's another
idea.

Ramin

[Steve Shipway]

You can make the package resource definition conditional.

class foo {
if ! defined( Package[gcc] ) {
package { gcc: ensure=>installed; }
}
}
class bar {
if ! defined( Package[gcc] ) {
package { gcc: ensure=>installed; }
}
}

Or, define a new class for the package(s) and include that

class pkg::gcc {
package { gcc: ensure=>installed; }
}
class foo {
include pkg::gcc
}
class bar {
include pkg::gcc
}

Second method is more elegant IMHO; but if its a one-off then first might be simpler.

Steve

Steve Shipway
University of Auckland ITS
UNIX Systems Design Lead
s.sh...@auckland.ac.nz
Ph: +64 9 373 7599 ext 86487

[jcbollinger]

On Jan 17, 3:11 am, Steve Shipway <s.ship...@auckland.ac.nz> wrote:
> You can make the package resource definition conditional.
>
> class foo {
>  if ! defined( Package[gcc] ) {
>   package { gcc: ensure=>installed; }
>  }}
>
> class bar {
>  if ! defined( Package[gcc] ) {
>   package { gcc: ensure=>installed; }
>  }
>
> }
>
> Or, define a new class for the package(s) and include that
>
> class pkg::gcc {
>   package { gcc: ensure=>installed; }}
>
> class foo {
> include pkg::gcc}
>
> class bar {
> include pkg::gcc
>
> }
>
> Second method is more elegant IMHO; but if its a one-off then first might be simpler.

The first is simpler only if you get lucky, and you don't intend to
modify your manifests ever again. Use of the "defined" function
introduces a parse-order dependency, and the additional work you need
to do to ensure that that dependency is always fulfilled overcomes any
simplicity advantage that might otherwise exist.

If you don't do that work, correctly, then you may still luck into the
code working, but you risk in the future making some seemingly
unrelated change that breaks it.

All-in-all, I'm inclined to consider any use whatever of the 'defined'
function to be a kludge, and a shaky one at that.


John

[Felix Frank]

Hi,

On 01/17/2012 04:11 PM, jcbollinger wrote:
> The first is simpler only if you get lucky, and you don't intend to
> modify your manifests ever again.

basically correct, but you can even play it safe: You must make sure
each and every invocation of the resource in question is protected by
such an if defined(), ever.

Yes, it's terrible design.

Is there a good reason that this function is even retained in recent
versions of puppet? I have yet to encounter an instance where it can be
used cleanly.

Cheers,
Felix

[jcbollinger]

On Jan 18, 5:52 am, Felix Frank <felix.fr...@alumni.tu-berlin.de>
wrote:

> Hi,
>
> On 01/17/2012 04:11 PM, jcbollinger wrote:
>
> > The first is simpler only if you get lucky, and you don't intend to
> > modify your manifests ever again.
>
> basically correct, but you can even play it safe: You must make sure
> each and every invocation of the resource in question is protected by
> such an if defined(), ever.

And in that case you must also make sure that the package is defined
with the same parameters everywhere, and that it stays that way.

> Yes, it's terrible design.

Indeed. Even for a one-off it's more complicated to use the
conditional approach correctly than to just put the package definition
in a class.

> Is there a good reason that this function is even retained in recent
> versions of puppet? I have yet to encounter an instance where it can be
> used cleanly.

I assume the function remains for backwards compatibility. That's a
pretty strong motivation for keeping it, but the function could at
least be deprecated.


John

[Nigel Kersten]

On Wed, Jan 18, 2012 at 3:52 AM, Felix Frank <felix...@alumni.tu-berlin.de> wrote:

Is there a good reason that this function is even retained in recent
versions of puppet? I have yet to encounter an instance where it can be
used cleanly.

I'll start another thread about this, but I'd quite like us to deprecate it for Telly.

 

[Alessandro Franceschi]

I add my 2 cents.

The alternatives proposed in the replies are ok for me, just would like to express an alternative approach that I''ve used in some situations.

It's due to different factors:

- Many packages to manage (but not necessarily)

- Option to include a module in an existing module set, where you don't know if and how the packages you want to use are already defined as resources.

- The fact that these packages generally don't need extra management (service or configuration files) and they do not harm if they are just installed and "forgotten" 

So the approach is to install them not as Puppet resources but via an exec:

class oracle::packages {

    require oracle::params

    file { "install_oracle_dependency.sh":
        mode    => 750, owner => root, group => root,
        content => $operatingsystem ? { 
            centos => template("oracle/install_oracle_dependency.sh-redhat"),
            redhat => template("oracle/install_oracle_dependency.sh-redhat"),
            debian => template("oracle/install_oracle_dependency.sh-debian"),
            ubuntu => template("oracle/install_oracle_dependency.sh-debian"),
            suse   => template("oracle/install_oracle_dependency.sh-suse"),
            },
        path    => "${oracle::params::workdir}/install_oracle_dependency.sh",
    }

    exec { "install_oracle_dependency.sh":
        subscribe   => File["install_oracle_dependency.sh"],
        refreshonly => true,
        timeout     => 3600,
        command     => "${oracle::params::workdir}/install_oracle_dependency.sh",
    }

}

where  the template install_oracle_dependency.sh-redhat is something like

#!/bin/sh
# File Managed by Puppet
# Installs the packages required for installing Oracle applications
<% if $architecture=="i386" %>
yum install -y binutils compat-db compat-libstdc++ compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel elfutils-libelf-devel-static gcc gcc-c++ glibc glibc-common glibc-devel glibc-headers kernel-headers ksh libaio libaio-devel libgcc libgomp libstdc++ libstdc++-devel make numactl-devel pdksh sysstat unixODBC unixODBC-devel
yum groupinstall -y "X Window System"
<% else %>
yum install -y binutils compat-db compat-libstdc++ compat-libstdc++.i386 compat-libstdc++-33 compat-libstdc++-33.i386  elfutils-libelf elfutils-libelf-devel elfutils-libelf-devel-static gcc gcc-c++ glibc glibc.i386 glibc-common glibc-devel glibc-devel.i386 glibc-headers kernel-headers ksh libaio libaio.i386 libaio-devel libaio-devel.i386 libgcc libgcc.i386 libgomp libstdc++ libstdc++.i386 libstdc++-devel make numactl-devel pdksh sysstat unixODBC unixODBC.i386 unixODBC-devel.i386
yum groupinstall -y "X Window System"
<% end %>

Opinions on this approach?

al

[jcbollinger]

1) Many of the packages you mention are pretty fundamental. If you
want to manage them then it would be better to group them into a
module for your standard environment, instead of allowing various
other modules to manage them.

2) It would be better to let the package management system handle
finding and installing depenencies. Where the software you want to
install is not available pre-packaged and cannot easily be packaged
locally, it would still be better to build a requirements-only package
(e.g. "oracle-dependencies") and manage that.

3) If you want to name the packages explicitly in your Puppet
configuration, then it would be better to declare them in some central
place as virtual Package resources. Modules that want them would then
*realize* them instead of declaring them. It is not a problem for a
virtual resource to be realized many times for the same node.


John

[bel]

I find it confusing that Puppet defines modules as reusable code yet,
when it comes to packages, you have to have prior knowledge of what
other package resources are defined.

Thank you for all the proposed solutions but does anyone know if
Puppet has a best practice they recommend in this case? Otherwise, I'm
going with defining new classes for the packages.

> s.ship...@auckland.ac.nz

[och ach]

same problem here. just find that razor module has static def of package curl.
what is the best practice?

for now i have deleted the definition from module and included my virtual packages definitions but would love to solve it not dirty fix it

[Felix Frank]

Whoo, the triumphant return of a thread from over two years ago ^^

Still, this has come up on the development list again, too, recently.

There has not yet been significat progress and we are still stuck with
the workarounds from the olden days. You can try with the
ensure_resource() function (in stdlib?).

My gut says that there won't be a superior alternative very soon, seeing
as there is not even an armature yet (that I know of, but I don't
monitor those).
Still, this is a known issue that is on people's minds, and I'm
confident that it will be tackled once the current development cycle has
settled.

HTH,
Felix

[Mike Frisch]

I am surprised there is no viable solution to this seemingly basic problem... I maintain a comprehensive set of Puppet modules for our internal software and while it hasn't been an issue yet, having two modules attempt to install the same package (ie. openssh-clients on RHEL) is certainly not out of the question. For my own maintained modules, this isn't a problem, but if I incorporate a third-party module that does the same, there's no guarantee that one of us won't have to change.

I have started using ensure_resource() but notice that with a Package resource that "ensure => installed" and "ensure => present" are treated as two distinct instantiations of the Package resource, even if they're treated as equivalent in Puppet.

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/dOCIZ8-Gfgw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/533584AC.7000503%40alumni.tu-berlin.de.
For more options, visit https://groups.google.com/d/optout.

[Bruno Léon]

For this kind of stuff i created a package module that only ships packages potentially shared (python lib, curl, etc...).
I  virtually déclare un thé module ans realie un the main modules

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/df5d4469-4bd5-4f48-b7d4-a221aadc0edd%40googlegroups.com.

[Peter Bukowinski]

I do the same currently. I have a module that installs/updates an in-house-developed scientific application with many os package (yum) and python module (pip) dependencies. I recently wrote a second module for another scientific application with overlapping dependencies and saw duplicate resource declaration errors when I first tested it.

My workaround was to pool the package requirements from both modules into a separate module and simply include it in my application modules. It sollved the issue, but nodes that only need one of the two applications will get more packages than they require. I suppose I should tag each package with the name of the application that requires it and use virtual resources to only realize the necessary packages, but I'm satisfied with the current solution since the overlap is pretty significant.

--

Peter Bukowinski

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJgpziV%2BNwR2Jht%2BtkY8zibsctombqQ7HVqhv13-K9QudK822g%40mail.gmail.com.

[jcbollinger]

On Friday, March 28, 2014 9:18:20 AM UTC-5, Felix.Frank wrote:

Whoo, the triumphant return of a thread from over two years ago ^^

Still, this has come up on the development list again, too, recently.

There has not yet been significat progress and we are still stuck with
the workarounds from the olden days. You can try with the
ensure_resource() function (in stdlib?).

Or, since any fix at this point is going to be dirty, you could just stick with the dirty fix you already have.  I rate it much superior to ensure_resource() or (equivalent) "if defined".

 

My gut says that there won't be a superior alternative very soon, seeing
as there is not even an armature yet (that I know of, but I don't
monitor those).

Sadly, I have to agree with Felix's gut.

 

Still, this is a known issue that is on people's minds, and I'm
confident that it will be tackled once the current development cycle has
settled.

I'll indulge in a little optimism there.



John

[Felix Frank]

Oh right.

Yes, of course, it *can* (and should!) be solved cleanly if you are in
control of all involved modules. It gets hairy were third party modules
enter the fray.

Cheers,
Felix

[jcbollinger]

On Friday, March 28, 2014 10:14:56 AM UTC-5, Mike Frisch wrote:

I am surprised there is no viable solution to this seemingly basic problem... I maintain a comprehensive set of Puppet modules for our internal software and while it hasn't been an issue yet, having two modules attempt to install the same package (ie. openssh-clients on RHEL) is certainly not out of the question. For my own maintained modules, this isn't a problem, but if I incorporate a third-party module that does the same, there's no guarantee that one of us won't have to change.

It is a basic problem in the sense that it is reasonably likely to be encountered if you rely on an uncurated collection of modules or on modules from multiple collections.  It breaks modularity in that modules cannot be implemented wholly without consideration of other modules.  But be sure to recognize that the problem is deeper than duplicate resource definition: it goes to the reason that Puppet disallows duplicate declarations in the first place.

Most resources -- and packages in particular -- are more complicated than a binary absent/present (a.k.a defined) switch can capture.  The "if ! defined" approach, including functions such as ensure_resource(), can prevent Puppet from emitting duplicate declaration errors if used pervasively, but it prevents Puppet from diagnosing conflicting resource requirements.  Consider this:

class site::security {
  if ! defined Package['vino'] {
    package { 'vino': ensure => 'absent' }
  }
}

class myapp::dependencies {
  if ! defined Package['vino'] {
    package { 'vino': ensure => 'installed' }
  }
}


Now suppose one of my nodes declares both classes.  Puppet will not complain, neither during catalog compilation nor during catalog application, but it's anybody's guess whether vino will be installed or not when Puppet is done.  And it might go back and forth over time as the manifest set is maintained.

To be sure, I chose the installed/absent conflict as a particularly striking one, but the same principle applies to any other difference between resource properties declared in different places.  (For example, if one of two or more declarations of one package specifies a version, and another specifies a different version or none.)

 

I have started using ensure_resource() but notice that with a Package resource that "ensure => installed" and "ensure => present" are treated as two distinct instantiations of the Package resource, even if they're treated as equivalent in Puppet.

Ensure_resource() is evil.  Do not use it.  Och's solution is one of the better alternatives, because it avoids multiple declaration of any resource (without regard to conditionals).  The other reasonably good approach is what Nigel suggested back in this thread's first life: factor out the package in question into a separate module (where it is declared concretely), and everywhere declare the appropriate class instead of directly declaring the package.



John

[Justin Rowles]

At risk of bumping a now truly ancient thread (which I found because I have a 3rd party module and one of my own which both want to control a specific package), I don't see why puppet should complain when two identical (or at least non-contradictory) invocations of the same resource are found.

[Lowe Schmidt]

It most likely has to do with creating the graph of dependencies, if we have one Package['a'] with two branches of deps it can't guarantee that it is acyclic.

--

Lowe Schmidt | +46 723 867 157

On 13 June 2016 at 12:21, Justin Rowles <justin...@bgch.co.uk> wrote:

At risk of bumping a now truly ancient thread (which I found because I have a 3rd party module and one of my own which both want to control a specific package), I don't see why puppet should complain when two identical (or at least non-contradictory) invocations of the same resource are found.

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7a6df4fc-3d34-442b-83d2-7f3c38015973%40googlegroups.com.

[jcbollinger]

On Monday, June 13, 2016 at 5:21:42 AM UTC-5, Justin Rowles wrote:

At risk of bumping a now truly ancient thread (which I found because I have a 3rd party module and one of my own which both want to control a specific package), I don't see why puppet should complain when two identical (or at least non-contradictory) invocations of the same resource are found.

You have necro'd a thread that was already necro'd once before.  I guess if history is any indicator, we can expect it to be necro'd again some time around July, 2018.  But let's see whether we can defy history, shall we?

In any case, there is no answer to the question of why Puppet works as it does in this regard that will be of any help to you.  That is, no answer to that question will help you resolve your present issue.  There has been a great deal of discussion of the general topic over the years; in fact, I think it may be Puppet's single most controversial characteristic.  Yet it endures.

If you really want to discuss why, then that discussion would be better conducted over on puppet-developers.

If you just want to vent, then ok, get it out of your system.

If you have specific questions about how to apply some of the plethora of information and advice that is already available, or if you have some pattern that you think has not been adequately covered, then by all means, please do start a new thread to discuss it.


John