Apparent unexpectedly synthesized dependency

[James Olin Oden]

I am getting a dependency loop when trying to apply my puppet
manifests, however the loop makes little sense to me as one the
dependencies seems to come from no where.

The cycle is this:

Augeas[Change production to development in virtual hosts] =>
Exec[re-read apache config] =>
Class[App::Apache] =>
Augeas[Change production to development in virtual hosts])

This cycle is started with this resource in Class A:


augeas { 'Change production to development in virtual hosts':
context => '/files/etc/httpd/conf.d/httpd-app.conf',
changes => [
"set VirtualHost[1]/*[self::directive='SetEnv']/arg[2] development",
"set VirtualHost[2]/*[self::directive='SetEnv']/arg[2] development",
],
require => Class['app::apache'],
notify => Exec['re-read apache config'],
}

Exec['re-read apache config'] lives in the class app::apache, and it
looks like:

exec { 're-read apache config':
command => '/sbin/apachectl graceful',
require => Class['app::packages'],
refreshonly => true,
}

So if I were to draw a dep graph I would draw two branches, this one:

Augeas[Change production to development in virtual hosts] =>
Class['app::apache']

And this one:

Augeas[Change production to development in virtual hosts] =>
Exec['re-read apache config'] =>
Class['app::packages']


But I would not draw the one puppet came up with:


Augeas[Change production to development in virtual hosts] =>
Exec[re-read apache config] =>
Class[App::Apache] =>
Augeas[Change production to development in virtual hosts]

In particular I don't see where it is getting that Exec[re-read apache
config] requires Class[App::Apache] (which itself contains
Exec[re-read apache config]). But then it goes a step beyond and
states that Class[App::Apache], the dep I can't explain requires
Augeas[Change production to development in virtual hosts] causing a
loop. Near as I can tell App::Apache makes no requirements that
takes you directly back Augeas[Change production to development in
virtual hosts].

Symbolically, I expect the dependency graphs:

A -> B
A -> C -> D

where B contains C, but in no way expected the graph:

A->C->B->A

Why might this be happening and what am I not understanding?

Thanks...James

[jcbollinger]

On Wednesday, February 4, 2015 at 10:09:04 AM UTC-6, James Oden wrote:

I am getting a dependency loop when trying to apply my puppet
manifests, however the loop makes little sense to me as one the
dependencies seems to come from no where.

The cycle is this:

   Augeas[Change production to development in virtual hosts] =>
   Exec[re-read apache config] =>
   Class[App::Apache] =>
   Augeas[Change production to development in virtual hosts])

 

This cycle is started with this resource in Class A:


    augeas { 'Change production to development in virtual hosts':
        context   => '/files/etc/httpd/conf.d/httpd-app.conf',
        changes   => [
            "set VirtualHost[1]/*[self::directive='SetEnv']/arg[2] development",
            "set VirtualHost[2]/*[self::directive='SetEnv']/arg[2] development",
        ],
        require => Class['app::apache'],
        notify  => Exec['re-read apache config'],
    }

 Exec['re-read apache config'] lives in the class app::apache, and it
looks like:

    exec { 're-read apache config':
        command     => '/sbin/apachectl graceful',
        require     => Class['app::packages'],
        refreshonly => true,
    }

[...]
 

In particular I don't see where it is getting that Exec[re-read apache
config] requires Class[App::Apache] (which itself contains
Exec[re-read apache config]).

That dependency arises from the Exec resource being declared by Class[App::Apache].  You could read it as "Class[App::Apache] has not been fully applied while Exec[re-read apache config] has not yet been applied."

 

 But then it goes a step beyond and
states that Class[App::Apache], the dep I can't explain requires
Augeas[Change production to development in virtual hosts] causing a
loop.   Near as I can tell App::Apache makes no requirements that
takes you directly back Augeas[Change production to development in
virtual hosts].

That's right in the Augeas resource you presented, specifically:

        require => Class['app::apache'],

Class['app::apache'] is too coarse-grained to serve your dependency requirements, as is evident in your expressing a dependency on a resource it declares.  You want your Augeas resource to be applied after Apache's configuration file has been configured, and you want the restart to be triggered when Augeas modifies the file, but both of those resources are managed by Class['app::apache'].

Since we've lately been discussing DSL code style and practices, I'll observe that the prevailing opinion these days seems to be that it is inappropriate for resources to be declared a given scope to be explicitly targeted by relationships declared in any unrelated scope, especially across module boundaries.  That principle would direct that relationships specifically with Exec['re-read apache config'] not be declared outside Class['app::apache'] or any subclasses it may have, and particularly not by classes outside module "app".

One common way to approach the issue would be to establish separate classes for the distinct logical components of Class['app::apache'].  The usual split is one to manage the physical presence of the software on the target system, one to manage its configuration, and one to manage its execution as a service.  These might be represented as classes app::apache::software (or "app::apache::install", which is more conventional but less suitable), app::apache::configuration (or "app::apache::config", which many people prefer), and app::apache::service.  For the record, where I offer alternative names for those components, it is because class names should be nouns describing the thing they manage, not verbs.

Given such a split, your Augeas resource would require Class['app::apache::software'], and notify Class['app::apache::service'].

Note that such a split does not imply removing Class['app::apache'], but most -- perhaps all -- of its work would be reduced to declaring (and declaring containment of) the split-out classes.


John

[jcbollinger]

Addendum:

On Wednesday, February 4, 2015 at 10:09:04 AM UTC-6, James Oden wrote:
 

The cycle is this:

   Augeas[Change production to development in virtual hosts] =>
   Exec[re-read apache config] =>
   Class[App::Apache] =>
   Augeas[Change production to development in virtual hosts])

[...]
 

In particular I don't see where it is getting that Exec[re-read apache
config] requires Class[App::Apache]

Note that the cycle you present does not have a direct link such as you describe with that.  The link actually present runs the other direction: Class['app::apache'] requires Exec['re-read apache config'].  I have already answered why that arises, but I want to make sure you don't think I've answered the wrong question.


John

[James Olin Oden]

On Thu, Feb 5, 2015 at 9:52 AM, jcbollinger <John.Bo...@stjude.org> wrote:
<snip>

>>
>> In particular I don't see where it is getting that Exec[re-read apache
>> config] requires Class[App::Apache] (which itself contains
>> Exec[re-read apache config]).
>
>
>
> That dependency arises from the Exec resource being declared by
> Class[App::Apache]. You could read it as "Class[App::Apache] has not been
> fully applied while Exec[re-read apache config] has not yet been applied."
>

But that implies that any resource of a class has a dependency on its
class. Put another way every element of a container seems to have a
dependency on the container? Sounds very chicken and egg to me (i.e.
a built in cycle).

>
>>
>> But then it goes a step beyond and
>> states that Class[App::Apache], the dep I can't explain requires
>> Augeas[Change production to development in virtual hosts] causing a
>> loop. Near as I can tell App::Apache makes no requirements that
>> takes you directly back Augeas[Change production to development in
>> virtual hosts].
>
>
>
> That's right in the Augeas resource you presented, specifically:
>
>> require => Class['app::apache'],
>

Yes but that is a separate dependency. I have declared:

A->B
A->C
where B contains C

1. That should in no way imply B->A.
2. Having a dependency on something contained in something else seems
odd, but when you think from a packaging perspective that's normal
since what you really need to do is execute this thing contained in
the container. This is why RPM has file dependencies (well it's one
of the reasons).

>
>
> Class['app::apache'] is too coarse-grained to serve your dependency
> requirements, as is evident in your expressing a dependency on a resource it
> declares. You want your Augeas resource to be applied after Apache's
> configuration file has been configured, and you want the restart to be
> triggered when Augeas modifies the file, but both of those resources are
> managed by Class['app::apache'].
>

Yes.

> Since we've lately been discussing DSL code style and practices, I'll
> observe that the prevailing opinion these days seems to be that it is
> inappropriate for resources to be declared a given scope to be explicitly
> targeted by relationships declared in any unrelated scope, especially across
> module boundaries. That principle would direct that relationships
> specifically with Exec['re-read apache config'] not be declared outside
> Class['app::apache'] or any subclasses it may have, and particularly not by
> classes outside module "app".

I understand the reasoning here, however in this case what is actually
going on is:

1) my real dependency is on app::apache which contains something I want to
execute.
2) I am executing something from app::apache.

What I was trying to do then was call the specific "command" I wanted
to run, and express the dependency that "command" had on its
container/package/class.

Now if just "notify => blah" automatically creates a dependency on the
container of blah that is awesome. However the fact that I had
explicitly declared that dependency should at worst only cause a
duplicate dependency then, not somehow convolve a cycle from that
duplicate dependency.

>
> One common way to approach the issue would be to establish separate classes
> for the distinct logical components of Class['app::apache']. The usual
> split is one to manage the physical presence of the software on the target
> system, one to manage its configuration, and one to manage its execution as
> a service. These might be represented as classes app::apache::software (or
> "app::apache::install", which is more conventional but less suitable),
> app::apache::configuration (or "app::apache::config", which many people
> prefer), and app::apache::service. For the record, where I offer
> alternative names for those components, it is because class names should be
> nouns describing the thing they manage, not verbs.
>
> Given such a split, your Augeas resource would require
> Class['app::apache::software'], and notify Class['app::apache::service'].
>
> Note that such a split does not imply removing Class['app::apache'], but
> most -- perhaps all -- of its work would be reduced to declaring (and
> declaring containment of) the split-out classes.

I like that way of approaching things. However, after listening to
your explanation it still feels like there is something amiss in the
handling of dependencies here. Two separate trees in a DAG should not
be merged together (unless I'm totally misunderstanding something).

That said, concerning the approach of notifying a class, when doing
this does this mean that all notifiable resources in the class get the
notify message too?

Thanks...James

[jcbollinger]

On Thursday, February 5, 2015 at 9:49:31 AM UTC-6, James Oden wrote:

On Thu, Feb 5, 2015 at 9:52 AM, jcbollinger <John.Bo...@stjude.org> wrote:
<snip>
>>
>> In particular I don't see where it is getting that Exec[re-read apache
>> config] requires Class[App::Apache] (which itself contains
>> Exec[re-read apache config]).
>
>
>
> That dependency arises from the Exec resource being declared by
> Class[App::Apache].  You could read it as "Class[App::Apache] has not been
> fully applied while Exec[re-read apache config] has not yet been applied."
>
But that implies that any resource of a class has a dependency on its
class.   Put another way every element of a container seems to have a
dependency on the container?   Sounds very chicken and egg to me (i.e.
a built in cycle).

It's more complicated than that.  The whole concept of containment requires some way of representing the boundaries of the container.  Puppet has implemented that in various ways over time.  The Exec -> Class relationship you see in your cycle is more technically a relationship between a contained resource and the end of its container.  The end cannot (notionally) be applied until after all the contained resources have been applied.  Puppet's representation of the beginning of a container is not relevant to your particular case.

Be certain you understand that the dependencies we're talking about here are strictly order-of-application dependencies.  That is, the chain expression "A -> B" expresses that in any given catalog run, correct/successful application of resource B depends on resource A must already have been applied in the same catalog run, but it has no other significance.  There are other forms of dependency relevant to Puppet, too, but they have nothing to do with chaining or 'resource relationships'.

 

>
>>
>>  But then it goes a step beyond and
>> states that Class[App::Apache], the dep I can't explain requires
>> Augeas[Change production to development in virtual hosts] causing a
>> loop.   Near as I can tell App::Apache makes no requirements that
>> takes you directly back Augeas[Change production to development in
>> virtual hosts].
>
>
>
> That's right in the Augeas resource you presented, specifically:
>
>>         require => Class['app::apache'],
>
Yes but that is a separate dependency.   I have declared:

    A->B
    A->C
    where B contains C

No.  You have declared

Class -> Augeas (require)
Augeas -> Exec (notify)

You also get

Exec -> Class (containment)

Voila!  A cycle.

 

1.   That should in no way imply B->A.
2.  Having a dependency on something contained in something else seems
odd,

I don't think it's so odd, but it tends to produce practical problems -- exactly as you are currently experiencing.  It can be done, but the problems get harder and harder to manage as your manifest set scales up and out.

[...]

 

I understand the reasoning here, however in this case what is actually
going on is:

   1)  my real dependency is on app::apache which contains something I want to
        execute.
   2) I am executing something from app::apache.

The central problem is entirely in the explicit declarations you have made.  Your Augeas resource declares that class app::apache must be applied first.  It also declares that a resource contained in that class must be applied later.  It's like saying "I have a sandwich and chips for lunch.  I'll eat my lunch before the meeting, and after the meeting I'll eat my sandwich."

 

What I was trying to do then was call the specific "command" I wanted
to run, and express the dependency that "command" had on its
container/package/class.

Now if just "notify => blah" automatically creates a dependency on the
container of blah that is awesome.

It doesn't, but I'm beginning to suspect that there is some lack of clarity (or perhaps lack of consistency) in this conversation regarding the direction of the relationship that the words "depends on" and "a dependency" is intended to convey.  When I say "A depends on B", I mean B must be applied before A is, so B -> A in terms of Puppet's chain operators.

 

  However the fact that I had
explicitly declared that dependency should at worst only cause a
duplicate dependency then, not somehow convolve a cycle from that
duplicate dependency.

Puppet has no problem with duplicate relationships, but that's not what's going on here.

 

I like that way of approaching things.   However, after listening to
your explanation it still feels like there is something amiss in the
handling of dependencies here.  Two separate trees in a DAG should not
be merged together (unless I'm totally misunderstanding something).

I don't understand what "merging" you think you see, unless it's that all order-of-application requirements for one catalog are represented in the same directed graph.  Every relationship between any pair of resources is represented as an edge in it.  Ordering requirements implied by containment are also expressed as edges in it.  Provided that graph is acyclic, a global order of application for all resources is chosen consistent with it.  Everything needs to be represented in that same graph for it to serve its purpose, so if you are arguing against that then yes, you are totally misunderstanding something.

 

That said, concerning the approach of notifying a class, when doing
this does this mean that all notifiable resources in the class get the
notify message too?

Yes.  Likewise for notifying an instance of a defined type.

Most of what we've been discussing is covered in the language reference, primarily in the section on resource relationships, and a bit in the section on containment.


John