Declare ressources from another stage

[Prunk Dump]

Hello puppet users !

My problem is simple. I use the following classes in three different stages (setup/main/runtime) :

class { 'apt::client':
      stage => 'setup',
}
...
...
class { 'hostpkg': }
...
...
class { 'extrapkg':
      stage => 'runtime',
}

The idea is that the "apt::client" class configure the Debian apt tools. Next the "hostpkg" class install and configure the important packages (ex: those needed to allow the connexion of users). And finally the "extrapkg" class install the secondary packages (internet browser etc ...). But sometimes the "hostpkg" or "extrapkg" packages need some extra apt sources, some apt-pinning or the multiarch. So I created three resources/class :

define apt::client::pinning( $package = $title, $pin, $priority, $ensure = present)
define apt::client::source( $sourcename = $title, $type = 'deb', $uri, $distribution = $apt::client::distribution, $components, $ensure = present)
class apt::client::multiarch

There three resources are contained in the "apt::client" class as they must be run before "apt-get update" (near "apt::client::end"). The problem come when a package in "extrapkg" need for example the multiarch :

-> From the "extrapkg" class I include the "apt::client::multiarch" class to get the multiarch activated
-> As the "extrapkg" class is in the "runtime" stage, the "apt::client::multiarch" is contained in the stage
-> I get a circular dependency as "apt::client::multiarch" need to be run before "apt::client::end" and so after the beginning of the runtime stage as it is included in the "runtime" stage.

How I can declare, inside a stage, a resource or a class that must not be contained in the stage. It is just "required" and must be run inside it's corresponding stage.

If anyone can help !

Baptiste.

[jcbollinger]

On Tuesday, October 4, 2016 at 6:55:36 AM UTC-5, Prunk Dump wrote:

Hello puppet users !

My problem is simple. I use the following classes in three different stages (setup/main/runtime) :

class { 'apt::client':
      stage => 'setup',
}
...
...
class { 'hostpkg': }
...
...
class { 'extrapkg':
      stage => 'runtime',
}

The idea is that the "apt::client" class configure the Debian apt tools. Next the "hostpkg" class install and configure the important packages (ex: those needed to allow the connexion of users). And finally the "extrapkg" class install the secondary packages (internet browser etc ...). But sometimes the "hostpkg" or "extrapkg" packages need some extra apt sources, some apt-pinning or the multiarch. So I created three resources/class :

Do take care to distinguish between operational relationships and management relationships.  Puppet class and resource ordering is about satisfying management relationships -- that is, situations where Puppet can manage resource B only after first ensuring that resource A is in its intended target state.  Avoid declaring unnecessary relationships, for there is no upside at all to having such relationships, and it incurs added risk of problems, such dependency cycles.  In particular, just because package B uses or interacts with package A does not necessarily imply that A must be managed before B, or vise versa.

Especially take care with run stages, for they are nothing more than a convenient mechanism to apply relationships to many pairs of classes at once.  Almost inevitably they apply a lot of unneeded relationships, but they may nevertheless be the best tool for a select few jobs.  Indeed, the Puppet docs say this:

[...] stages should only be used with the simplest of classes, and only when absolutely necessary. Mass dependencies like package repositories are effectively the only valid use case.

I'm inclined to suppose that there may be other valid use cases at certain sites, and I don't think the docs are saying that stages are always the best solution for ensuring that package repositories are managed before all packages, but do take the docs' remark in the spirit in which I think it was intended: run stages are the wrong tool for almost every job.

In your particular case, it may be that class apt::client is indeed one of those rare classes that make sense in a non-default stage, but class extrapkg seems very unlikely to be such a class.  In the unlikely event that you need any puppet-application-ordering relationships between resources declared in class extrapkg and (other) classes and resources in stage main, I see no barrier to declaring those relationships explicitly.  Relying on doing so instead of needlessly placing class extrapkg in a separate stage may solve some of your problems.

 

define apt::client::pinning( $package = $title, $pin, $priority, $ensure = present)
define apt::client::source( $sourcename = $title, $type = 'deb', $uri, $distribution = $apt::client::distribution, $components, $ensure = present)
class apt::client::multiarch

There three resources are contained in the "apt::client" class as they must be run before "apt-get update" (near "apt::client::end").

What resources?  I see (definitions of) two resource types and one class.  Class apt::client may declare instances of the two types; those instances are then contained by the class.  It might also declare and even also contain (in the relationship sense) class apt::client::multiarch, but the declaring and containing are separate considerations.  Under no circumstances, however, should the definitions of these types and this class be lexically contained within the definition of class apt::client.  Each definition should appear in its own file.

 

The problem come when a package in "extrapkg" need for example the multiarch :

-> From the "extrapkg" class I include the "apt::client::multiarch" class to get the multiarch activated

Ok.

 

-> As the "extrapkg" class is in the "runtime" stage, the "apt::client::multiarch" is contained in the stage

Only if class extrapkg contains (relationship sense) class apt::client::multiarch, unless I've missed a change somewhere.  Indeed, lack of the behavior you describe was once cited as a bug, and that bug report was rejected.

If you do this ...

# This should be ok:
include "apt::client::multiarch"

... then you do not automatically put that class in the same run stage as the class declaring it.  If, on the other hand, you ...

# Don't do this with a class that may also be
# declared by other classes in different run stages:
contain "apt::client::multiarch"

... then you indeed do create a risk of dependency cycles such as you describe next.  The run stages docs warn about this specifically.

 

-> I get a circular dependency as "apt::client::multiarch" need to be run before "apt::client::end" and so after the beginning of the runtime stage as it is included in the "runtime" stage.

How I can declare, inside a stage, a resource or a class that must not be contained in the stage. It is just "required" and must be run inside it's corresponding stage.

A resource is always contained by the class that declares it, and will therefore be applied in the same stage as that class.  Run stage assignment is not transitive, except for contained classes.  Other classes' run stages are determined independently of the stage to which the declaring class, if any, is assigned.

If you can rely on apt::client::multiarch to be declared independently of anything in class extrapkg, as it seems maybe you can do, then one alternative is for class extrapkg simply to ignore it.  On the other hand, if you must permit multiple classes to declare that class, and especially if it self-declares containment within apt::client, which appears may be the case, then you should set its 'stage' metaparameter as needed via automated data binding.


John

[Prunk Dump]

A big thank-you John ! You reply is very precise. I learn a lot of things !

In the case of apt::client::multiarch it may be a bug. Because I only "include" the class. Not contain. And I get this dependency cycle :

Error: Could not apply complete catalog: Found 1 dependency cycle:
(Anchor[apt::client::end] => Class[Apt::Client] => Stage[setup] => Stage[main] => Class[Settings] => Class[Settings] => Stage[main] => Stage[runtime] => Class[Apt::Client::Multiarch] => Exec[add arch i386] => Class[Apt::Client::Update] => Exec[aptitude update] => Class[Apt::Client::Update] => Stage[setup])

It is true that :

Class[Apt::Client::Multiarch] => Exec[add arch i386] => Class[Apt::Client::Update] => Exec[aptitude update] => Class[Apt::Client::Update]

But there is no reason that (as it's just included) :

Stage[runtime] => Class[Apt::Client::Multiarch]

Can you help me on a more simpler and concrete example ? 

1) My apt::client class have a defined type :

define apt::client::source

that allow to add sources to /etc/apt/sources.list.d. As an "apt-get update" is needed after adding sources, all the resources defined in my apt::client::source type :
=> require Class['apt::client::audit'] (that check if there are not pending install)
=> notify Class['apt::client::update'] (that run apt-get update)
There as no more dependencies. And the two class (  apt::client::audit and apt::client::update ) are contained in the apt::client class.

2) Now I want to build a "wine" class that install wine from "backports".

=> the wine class require that apt is ready to install packages so naturally wine class need to be run after the apt::client class.

=> But wine also need that the backports sources are added to sources.list. So in the the wine::install class manifest I declare a apt::client::source(...backports sources...) that is required by the Package['wine'] resource.

3) Finally I got a dependency cycle :

apt::client::update => wine::install  (because apt need to be ready)
wine::install => apt::client::source(...backports sources...)  (because the resource is declared inside the wine class and required by Package['wine'])
apt::client::source(...backports sources...) => apt::client::update (as the source need to be added before apt-get update)

You can say to me that I can create a specific apt::client::backports_source class and include it instead of declaring it. But I have many other modules that need many other specific sources.

You can say that I can put the apt::client::source(...backports sources...) in the apt module. But I don't want that this sources be added on node that don't have wine.

I there a way to achieve what I want with puppet ? Maybe with class parameters ?

Thanks !

[jcbollinger]

On Thursday, October 6, 2016 at 9:03:26 AM UTC-5, Prunk Dump wrote:

Can you help me on a more simpler and concrete example ? 

1) My apt::client class have a defined type :

Your apt::client class does not have such a defined type.  Even if the type's definition appears lexically within the class's (which it ought not to do), that does not imply any meaningful ownership or scope.

 

define apt::client::source

that allow to add sources to /etc/apt/sources.list.d. As an "apt-get update" is needed after adding sources, all the resources defined in my apt::client::source type :
=> require Class['apt::client::audit'] (that check if there are not pending install)
=> notify Class['apt::client::update'] (that run apt-get update)
There as no more dependencies. And the two class (  apt::client::audit and apt::client::update ) are contained in the apt::client class.

2) Now I want to build a "wine" class that install wine from "backports".

=> the wine class require that apt is ready to install packages so naturally wine class need to be run after the apt::client class.

=> But wine also need that the backports sources are added to sources.list. So in the the wine::install class manifest I declare a apt::client::source(...backports sources...) that is required by the Package['wine'] resource.

3) Finally I got a dependency cycle :

apt::client::update => wine::install  (because apt need to be ready)
wine::install => apt::client::source(...backports sources...)  (because the resource is declared inside the wine class and required by Package['wine'])
apt::client::source(...backports sources...) => apt::client::update (as the source need to be added before apt-get update)

You can say to me that I can create a specific apt::client::backports_source class and include it instead of declaring it. But I have many other modules that need many other specific sources.

You can say that I can put the apt::client::source(...backports sources...) in the apt module. But I don't want that this sources be added on node that don't have wine.

I there a way to achieve what I want with puppet ? Maybe with class parameters ?

 
The cycle you describe occurs because your relationships are too coarse-grained.  It's not all of class wine::install that must be applied after apt::client::update, nor all of it that must be applied before apt::client::update.  You have one part that must be applied after, one part that must be applied before, and maybe a part that can be applied either before or after.  To resolve that situation you need to split the class into at least two classes, and apply the relationships with each one that are appropriate for it.


John