Solving `if defined(Resource['name'])`

[Daniele Sluijters]

Hi everyone,

As I was browsing through a few modules on the Forge I noticed that more and more people seem to have taken to the `if ! defined(Package['name']) {}` to ensure their module doesn't cause a duplicate resource declaration with another module, third party or your own, that is also managing that package.

From a purist perception this should never happen in the first place as there are suddenly two modules declaring the same resource. Unfortunately this is often what it comes down to when you pull in a few modules from the Forge or Github, especially with things like build-essential that a multitude of different things might need. In turn this makes using third-party modules more difficult because they might introduce issues like these that, especially for newcomers to the platform, are troublesome to fix.

I was hoping we could devise a solution, at least for package resources, that would allow multiple declarations of the package and have those declarations resolve themselves. I'm specifically focussing on package here because that's the most common source of trouble, things like an rbenv, rvm and pyenv module all declaring that they need 'build-essential'.

My initial thought was that a package could have two states, hard and soft where a state => hard, definition of a package would always trump any other definitions of a package in a soft state. The question then becomes, how do you deal with multiple soft states of the same package which basically boils down to you can't have multiple declarations of the same thing, so we're back to square one.

My other thought was that perhaps it should be possible to `require => Package['name']` from any resource and if that resource isn't declared in the catalog spawn the resource ourselves but if it is defined that declaration is used. I'm not sure how we could deal with scenario's where someone requires said package but we explicitly define it as absent in our manifests.

Please spare me the "they're being wrong on the internet" speech, I'm well aware of it but the truth of the matter is you'll never be able to coordinate the community in such a way that this issue doesn't pop up. I'm not really concerned about experienced Puppet users/developers that can solve this for themselves, more about the less developer oriented operator that has to use Puppet for whatever reason. Another approach would be to really curate the modules up on the Forge but I doubt anyone feels like doing that or has the time for it.

I'm not sure if it's even desirable or possible to solve this issue in the first place but I was curious if anyone has given any more thought to this?

-- 

Daniele Sluijters

[Felix Frank]

Hi Daniele,

thanks for bringing this up. Again :-)

On 02/19/2014 12:16 PM, Daniele Sluijters wrote:
> I'm not sure if it's even desirable or possible to solve this issue in
> the first place but I was curious if anyone has given any more thought
> to this?

They have. I managed to dig up these two monsters from the past, but
there are more, I believe:

"RFC: Deprecate defined() function for Telly."
https://groups.google.com/forum/#!searchin/puppet-users/cross-module/puppet-users/YncxKD3MSms/fHOhIqat0u4J

"Cross-module dependencies"
https://groups.google.com/forum/#!searchin/puppet-users/cross-module/puppet-users/Fvl0aOe4RPE/-k60j2QSID0J

Take special note of John's "constraint" sketch in the latter.

Cheers,
Felix

[Trevor Vaughan]

One of the things that was proposed in the past was allowing different modules to declare the same resource iff it was declared identically in all cases.

It was shot down as promoting bad practices but so was if defined() and the fact is that we *need* that because I can't have service { 'foo': ensure => running } in two modules.

Yes, of course the correct way to do things is to abstract out each portion that is not on your critical path and have parameters that allow you to selectively manage it thereby alleviating the issues with if defined(), etc...

But, everyone has jobs to do and may not be able to spend much time refactoring for other users.

I *do* think that all Puppet Labs sponsored/approved modules should allow for the following to be selectively managed/included by a module at a minimum:

* Firewall

* PAM

* Service

* Web

In general, I've found that if you make these optional, you stand a pretty good chance of staying out of people's way and letting them manage those portions their way while still taking advantage of your module.

All that said, I still want to be able to declare multiple identical resources and have it "just work".

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/234c5510-c2b2-4786-ac76-68a73bf55e7d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Trevor Vaughan]

Sorry for the double post, but I just remembered the new way of doing if defined(). A function was added to stdlib called ensure_resource https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/lib/puppet/parser/functions/ensure_resource.rb.

Looking at it, and how it is used, it's just another if defined().

Trevor

[Daniele Sluijters]

Thanks for bringing up ensure_resources, I completely forgot about that one. Add it to the list of things I'd rather see solved more permanently.

[John Bollinger]

On Wednesday, February 19, 2014 5:16:00 AM UTC-6, Daniele Sluijters wrote:

Hi everyone,

As I was browsing through a few modules on the Forge I noticed that more and more people seem to have taken to the `if ! defined(Package['name']) {}` to ensure their module doesn't cause a duplicate resource declaration with another module, third party or your own, that is also managing that package.

Oh, dear.  You've brought up one of my pet peeves.

 

From a purist perception this should never happen in the first place as there are suddenly two modules declaring the same resource. Unfortunately this is often what it comes down to when you pull in a few modules from the Forge or Github, especially with things like build-essential that a multitude of different things might need. In turn this makes using third-party modules more difficult because they might introduce issues like these that, especially for newcomers to the platform, are troublesome to fix.

You are exactly right on all counts.

 

I was hoping we could devise a solution, at least for package resources, that would allow multiple declarations of the package and have those declarations resolve themselves. I'm specifically focussing on package here because that's the most common source of trouble, things like an rbenv, rvm and pyenv module all declaring that they need 'build-essential'.

I'm not very fond of the idea of hardwiring special support for Package resources into Puppet (nor for resources of any other specific type).  I still like my "constraints" idea, to which Felix directed your attention, but if something like that is ever implemented then adopting it will require (smallish) modifications to be made to a lot of manifests.

 

My other thought was that perhaps it should be possible to `require => Package['name']` from any resource and if that resource isn't declared in the catalog spawn the resource ourselves but if it is defined that declaration is used. I'm not sure how we could deal with scenario's where someone requires said package but we explicitly define it as absent in our manifests.

A solution might be possible along such lines, and there's no particular reason for it to be specific to the Package resource type.  Puppet could conceivably provide an option for native types to support autodeclaration of referenced but undeclared instances.  If, after evaluating all relevant manifests, it discovered one or more references to a specific resource of an auto-declarable type, then it would create a corresponding resource in the catalog and initialize it with (say) an auto_init() method of the type.

It is true that such a mechanism would not kick in where a resource is explicitly declared absent, but that's a limitation of the (mal)practice of using "if ! defined()", too -- a module compatibility issue that 'constraints' could detect, but that simple declaration checks, in any form, cannot.  A more serious limitation is that you may want catalog compilation to fail when resources are referenced but not anywhere declared.  The catalog compiler should certainly emit warnings in these cases, but I would be happier if autodeclaration had to be explicitly enabled in the master's configuration in order to be used at all.

 

Please spare me the "they're being wrong on the internet" speech, I'm well aware of it but the truth of the matter is you'll never be able to coordinate the community in such a way that this issue doesn't pop up. I'm not really concerned about experienced Puppet users/developers that can solve this for themselves, more about the less developer oriented operator that has to use Puppet for whatever reason. Another approach would be to really curate the modules up on the Forge but I doubt anyone feels like doing that or has the time for it.

I spare you the speech.  No one here needs it, anyway.

 

I'm not sure if it's even desirable or possible to solve this issue in the first place but I was curious if anyone has given any more thought to this?

Yes, indeed, as Felix said.  This is the Big Kahuna of module compatibility problems, and such issues have been on the community's mind at least since the Forge was introduced.


John

[Wolf Noble]

One thing I've seen done is something like the following contrived example:

class core{

    @package { 'build essential':

      ensure => 'installed',

      tag    => 'core_packages'

  }

  Package <| tag == 'core_packages' |>

}

From my probably incomplete understanding, virtualizing and realizing the packages like this allows for multiple classes to declare the resource, and as long as the ensure value is the same across declarations, the declarations can cohaibtate in the same catalog.

Am I missing something that would make this an antipattern as well? (entirely possible)

[Wolf Noble]

autocorrect changed 'buildessential' to 'build essential'..

mental note to not send to puppet-dev from my phone. ;)

[Nan Liu]

You still can't declare two virtual package, only the ability to realize as many times as necessary. This requires module authors to agree on the same set of virtual resource module. I think people just want to say

ensure_package('buildessential') 

Instead of worry about which module on forge has this virtual resource and whether it's the same as another author's choice of virtual package module. If the compiler can recognize the same package and avoid the need for a special function, even better.

Nan

[markus]

> I'm not very fond of the idea of hardwiring special support for
> Package resources into Puppet (nor for resources of any other specific
> type). I still like my "constraints" idea, to which Felix directed
> your attention, but if something like that is ever implemented then
> adopting it will require (smallish) modifications to be made to a lot
> of manifests.

+1 to a constraints style solution. Figuring how to do it without
causing a lot of widespread minor pain would be an added plus. :)

[Daniele Sluijters]

I like a constrains style solution much better myself to, wasn't aware of it at the time.

I think with Puppet 4 we could introduce some additional pain, semver wise we'd be

allowed to do that. If that helped fix this issue once and for all I'm pretty sure the

community would support it too.

[Felix Frank]

Hi,

bumping an old thread with a some news: I've built a proof-of-concept
implementation of John's constraint scheme. Enjoy:
https://github.com/ffrank/puppet/tree/constraints

I'd be grateful for any feedback. Constraints can currently be added as
normal resources to a manifest as shown in the README.

Each is limited to targetting a single resource. I'd be interested in
amending this so that metaparameter-like declarations such as

resource => Package[foo,bar]
resource => [ Package[foo], Service[bar] ]

are supported. There's also some TODOs such as tight validation (i.e.
syntax checks).

An idea for extending the concept is "weak" constrains that won't fail
if a targeted resource is not part of a catalog.

A PR is pending once these additions are complete.

Thanks in advance to anyone who takes a closer look.

Cheers,
Felix

[Andy Parker]

On Mon, Apr 14, 2014 at 4:07 PM, Felix Frank <Felix...@alumni.tu-berlin.de> wrote:

Hi,

bumping an old thread with a some news: I've built a proof-of-concept
implementation of John's constraint scheme. Enjoy:
https://github.com/ffrank/puppet/tree/constraints

I'd be grateful for any feedback. Constraints can currently be added as
normal resources to a manifest as shown in the README.

Each is limited to targetting a single resource. I'd be interested in
amending this so that metaparameter-like declarations such as

  resource => Package[foo,bar]
  resource => [ Package[foo], Service[bar] ]

are supported. There's also some TODOs such as tight validation (i.e.
syntax checks).

That is pretty cool.

 

An idea for extending the concept is "weak" constrains that won't fail
if a targeted resource is not part of a catalog.

A PR is pending once these additions are complete.

Thanks in advance to anyone who takes a closer look.

I'm wondering if there is a way of either doing this in a module (do the constraint check in a provider), or if there is a change we can make to allow this to be in a module. It would be great to be able to have the extension points so that these kinds of things don't have to go into the puppet repository in order to be tried out.

 

Cheers,
Felix

On 02/20/2014 10:17 AM, Daniele Sluijters wrote:
> I like a constrains style solution much better myself to, wasn't aware
> of it at the time.
>
> I think with Puppet 4 we could introduce some additional pain, semver
> wise we'd be
> allowed to do that. If that helped fix this issue once and for all I'm
> pretty sure the
> community would support it too.

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/534C6A4B.8020408%40Alumni.TU-Berlin.de.
For more options, visit https://groups.google.com/d/optout.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

Join us at PuppetConf 2014, September 22-24 in San Francisco

Register by May 30th to take advantage of the Early Adopter discount —save $349!

[Felix Frank]

On 04/15/2014 03:01 AM, Andy Parker wrote:
> I'm wondering if there is a way of either doing this in a module (do
> the constraint check in a provider), or if there is a change we can
> make to allow this to be in a module. It would be great to be able to
> have the extension points so that these kinds of things don't have to
> go into the puppet repository in order to be tried out.

Thanks for the input.

I even *had* been thinking of hacking the constraint evaluation into the
new type's generate hook. This way, the agent would naturally run them
before the transaction, thinking it was a constraint's way to do
generate additional resources.

This has two limitations, though:
1. The cosmetic issue that the error output alludes to the generation of
additional resources and
2. the critical problem that this hook has no way to interrupt the
transaction.

Usually, exceptions that are raised from types and providers are handled
as "resource local" problems. They will fail the resource, but not the
transaction as a whole.

I currently see no way to do this without patching the transaction code.
Of course, we could just introduce an appropriate hook into the core and
do everything else in modules for a time then.

Seeing as the lonterm goal of the feature would be to serve as the
mortar between third party modules, I'd still see it in tier 1, but
that's just my gut feeling.

It may just become a very common dependency such as concat_file I guess.
But can we deprecate defined() in favor of an external module? :-)

Cheers,
Felix

[Trevor Vaughan]

Looking at this (and awesome work Felix) my gut reaction is that this should still be part of the core language.

I don't like the idea of effectively being able to monkey patch the Puppet core flow directly regarding how types and providers interact. I say this mainly because, when I jump between environments, I don't really want to have to figure out what insane collection of "enhancements" were added where and what issues they may be causing.

I'm still going to have to play with this some more and figure out if it meets my use cases before I can comment further.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/534CC263.2070403%40Alumni.TU-Berlin.de.

For more options, visit https://groups.google.com/d/optout.

--

[Andy Parker]

On Mon, Apr 14, 2014 at 10:23 PM, Felix Frank <Felix...@alumni.tu-berlin.de> wrote:

On 04/15/2014 03:01 AM, Andy Parker wrote:
> I'm wondering if there is a way of either doing this in a module (do
> the constraint check in a provider), or if there is a change we can
> make to allow this to be in a module. It would be great to be able to
> have the extension points so that these kinds of things don't have to
> go into the puppet repository in order to be tried out.

Thanks for the input.

I even *had* been thinking of hacking the constraint evaluation into the
new type's generate hook. This way, the agent would naturally run them
before the transaction, thinking it was a constraint's way to do
generate additional resources.

Generate might the appropriate place, or some new hook that we don't have yet. There has been a constant need for lifecycle hooks so that providers can make better decisions.

 

This has two limitations, though:
1. The cosmetic issue that the error output alludes to the generation of
additional resources and
2. the critical problem that this hook has no way to interrupt the
transaction.

Usually, exceptions that are raised from types and providers are handled
as "resource local" problems. They will fail the resource, but not the
transaction as a whole.

Right, and if this fails it needs to show up in the report, so that also needs to be taken into account.

 

I currently see no way to do this without patching the transaction code.
Of course, we could just introduce an appropriate hook into the core and
do everything else in modules for a time then.

Yep, that is what I was thinking. I think some functionality like this would be great to have in the core, but I'd like to make sure that it is proven a bit outside first. If that means that we need to make some changes to the core to add the hooks that are needed, then I actually see that as a win-win: not only do we get this awesome functionality out there and able to be iterated on faster than puppet releases, but we also get some extra hooks to make this kind of stuff easier/possible in the future for others.

 

Seeing as the lonterm goal of the feature would be to serve as the
mortar between third party modules, I'd still see it in tier 1, but
that's just my gut feeling.

It may just become a very common dependency such as concat_file I guess.
But can we deprecate defined() in favor of an external module? :-)

Nope :) But I'd like to make sure that this not only solves the defined() problem, but is a joy to use and has powerful features, which might take a little trial and error.

 

Cheers,
Felix

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/534CC263.2070403%40Alumni.TU-Berlin.de.

For more options, visit https://groups.google.com/d/optout.

[Andy Parker]

On Tue, Apr 15, 2014 at 6:17 AM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

Looking at this (and awesome work Felix) my gut reaction is that this should still be part of the core language.

I don't like the idea of effectively being able to monkey patch the Puppet core flow directly regarding how types and providers interact. I say this mainly because, when I jump between environments, I don't really want to have to figure out what insane collection of "enhancements" were added where and what issues they may be causing.

That is the danger of having a pluggable system. I don't see any way around it :(

 

I'm still going to have to play with this some more and figure out if it meets my use cases before I can comment further.

Which is exactly why I want to get some extension points in so that it can be released as a module.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoVdT4DCxMsHGeTv_2130__3LiVPdqVxpBOgaDw1k%2Bka%2Bg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

[Joshua Hoblitt]

On 04/15/2014 10:38 AM, Andy Parker wrote:
> Generate might the appropriate place, or some new hook that we don't have
> yet. There has been a constant need for lifecycle hooks so that providers
> can make better decisions.

+1

This is a tad tangential but I occasionally fantasize about a mount
provider that can deal with stale NFS mounts. It would be nice to have
a hook to attempt to rescue/retry a resource that failed to apply
without having to inject a bunch of conditional logic into the general case.

-Josh

--

[John Bollinger]

On Monday, April 14, 2014 6:07:55 PM UTC-5, Felix Frank wrote:

Hi,

bumping an old thread with a some news: I've built a proof-of-concept
implementation of John's constraint scheme. Enjoy:
https://github.com/ffrank/puppet/tree/constraints

I'd be grateful for any feedback. Constraints can currently be added as
normal resources to a manifest as shown in the README.

Each is limited to targetting a single resource. I'd be interested in
amending this so that metaparameter-like declarations such as

  resource => Package[foo,bar]
  resource => [ Package[foo], Service[bar] ]

are supported. There's also some TODOs such as tight validation (i.e.
syntax checks).

An idea for extending the concept is "weak" constrains that won't fail
if a targeted resource is not part of a catalog.

A PR is pending once these additions are complete.

Thanks in advance to anyone who takes a closer look.

That is really exciting!

And it's great that the code changes were so modest -- I had imagined that constraints might require much deeper changes than that.  Felix, you're my hero!


John

[Felix Frank]

Thanks for all the positive feedback, guys.

Trevor raises some interesting points, but I guess Andy has it right by
proposing that we should let this unfold outside the core until we deem
it ready.

I have some code on hand to hook us into the transaction, but as always,
building tests is the harder half of the work. I hope that a PR will
appear over the course of Easter.

Furthermore, I shall teach myself about authoring on the Forge and see
to releasing the constraints module stat.

Cheers,
Felix

[Trevor Vaughan]

Do you like dead threads?  like dead threads....

So, I was looking back over this and there's one thing that we desperately need as a community.

We need defined(Class['foo']) and it needs to know about ALL classes in the catalog at compile time, not just the ones that got included before it.

This would save a TON of heartburn in a lot of cases.

For instance, right now, I have to do:

# Don't reverse these two!

include 'foo'
include 'bar'

Where bar has something like:

if defined(Class['foo']) {

  do something that happens if bar has foo
}

In theory, this also rides into unmangling some of the roles/profiles configurations that I've seen posted around where people twist themselves in knots using variables to make sure that their classes are included properly when a simple if defined(Class['foo']) would make all of that disappear.

I would like to suggest that this get fixed in the future parser and that it be singled out as something like  'has_class'.

if has_class('foo') { stuff }

Thoughts?

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/d680773d-57bb-49e5-9875-f543ace773b0%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Henrik Lindberg]

On 2014-17-06 22:14, Trevor Vaughan wrote:
> Do you like dead threads? like dead threads....
>

Only some of them (like this one)...

> So, I was looking back over this and there's one thing that we
> desperately need as a community.
>
> We need defined(Class['foo']) and it needs to know about ALL classes in
> the catalog at compile time, not just the ones that got included before it.
>

By ALL classes I assume you mean all the classes that will have been
evaluated and included in the catalog at the time the catalog
compilation is finished?

> This would save a TON of heartburn in a lot of cases.
>
> For instance, right now, I have to do:
>
> # Don't reverse these two!
> include 'foo'
> include 'bar'
>
> Where bar has something like:
>
> if defined(Class['foo']) {
> do something that happens if bar has foo
> }
>
> In theory, this also rides into unmangling some of the roles/profiles
> configurations that I've seen posted around where people twist
> themselves in knots using variables to make sure that their classes are
> included properly when a simple if defined(Class['foo']) would make all
> of that disappear.
>
> I would like to suggest that this get fixed in the future parser and

> that it be singled out as something like 'has_class'..

>
> if has_class('foo') { stuff }
>
> Thoughts?

It is quite difficult to achieve as a general "see if it is defined now"
since an inclusion can take place in conditional logic, from logic in
ruby code (using define_resources), classes can be parameterized and
instantiated with resource creation syntax (which makes them unknown to
the system until the class / resource type that realizes them is evaluated.

In other words - close to impossible to do this accurately given all the
ways classes can be added to the catalog during the evaluation. At least
as a simple lookup of Class['foo'].

I can imagine a mechanism that kicks in towards the end of the
compilation but that is also complicated - one part of the logic has a
rule that if X is missing it does Y, and another rule adds X if it is
(directly or indirectly missing). We can also add callbacks - say when a
class is added to the catalog - can trigger a code block that is then
evaluated. (This is harder to trigger for something that is not in the
catalog), but maybe this could be another set of callbacks, and they
keep triggering until nothing more is triggered).
(PROLOG anyone? Constraint solver?)

I can also imagine being made more aware of when things do occur and
only take action when certain things are known - a sort of orchestration
of the composition of the catalog. This would enable stating things like
"include this in the catalog when these conditions hold / when these
other things are done". This is not much unlike the callbacks /
constraints scenario above, but in a fashion where it is possible to
reason about what is going to happen (starving, deadlocking, endless
looping). (This is an idea that I think is very powerful and that can
extend to agents as well)

Maybe we can find some compromise. One area that has been discussed is
the domain of queries. The do run after the rest of the compilation has
been made, but they can currently only do certain pre-defined tasks
(like realizing resources). Maybe a better query and more powerful query
mechanism is enough. (Again very similar to registered callbacks with
predicates that trigger at the end - that is basically what the query
mechanism is now (i.e. the spaceship operators).

I do also think that there are other fundamental compositional
mechanisms missing - say the ability to extend things in a fashion that
the extended party is aware of how it is extended (or will be extended
because someone promised / declared this). Naturally, the reverse is
also needed; if module X is included and A, B are in the catalog, then X
will do something without A, and B being aware (a reverse dependency if
you like).

Just some thoughts on the topic.
- henrik

> <mailto:puppet-dev%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-dev/d680773d-57bb-49e5-9875-f543ace773b0%40googlegroups.com.
>
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699

> tvau...@onyxpoint.com <mailto:tvau...@onyxpoint.com>

>
> -- This account not approved for unencrypted proprietary information --
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-dev+...@googlegroups.com

> <mailto:puppet-dev+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoV3th5f-uHM7wqL6LOPoJf-%2BxWUhB%2BJAAhh7Tua2rA8Pw%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoV3th5f-uHM7wqL6LOPoJf-%2BxWUhB%2BJAAhh7Tua2rA8Pw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Trevor Vaughan]

Henrik, you 100% nailed everything that I wanted.

In my mind, I was envisioning the system of callbacks that you described as a stack with pointers (file:line?) portions of the code. When the catalog pass is complete, refer back to the callback stack(s) and process them down until empty.

Basically, partial evaluation of all files and then recursive evaluation until complete.

It seems to me, not having studied the compiler, that this would be more straightforward to implement due to the way that Puppet uses very clear resource stanzas that can be atomically parsed.

It does get hairy when you mix the callbacks with the reverse dependencies. I can see having to iterate over the stack (mesh?) and check for loops and correctness while doing so. It could certainly turn into quite the recursive memory hog if not done properly.

However, I really do think that this is needed in the language as it seems to be a place where people frequently stumble when putting together what should be a straightforward thought process.

Thanks!

Trevor

    send an email to puppet-dev+unsubscribe@googlegroups.com

    <mailto:puppet-dev%2Bunsu...@googlegroups.com>.

    To view this discussion on the web visit
    https://groups.google.com/d/msgid/puppet-dev/d680773d-57bb-49e5-9875-f543ace773b0%40googlegroups.com.

    For more options, visit https://groups.google.com/groups/opt_out.




--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699

tvau...@onyxpoint.com <mailto:tvau...@onyxpoint.com>

-- This account not approved for unencrypted proprietary information --

--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send

an email to puppet-dev+unsubscribe@googlegroups.com

<mailto:puppet-dev+unsub...@googlegroups.com>.

To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoV3th5f-uHM7wqL6LOPoJf-%2BxWUhB%2BJAAhh7Tua2rA8Pw%40mail.gmail.com
<https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoV3th5f-uHM7wqL6LOPoJf-%2BxWUhB%2BJAAhh7Tua2rA8Pw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/lnqqkl%245ai%241%40ger.gmane.org.

For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

[Felix Frank]

Hi,

putting on my manifest design philosopher's hat :-)

On 06/18/2014 04:11 AM, Trevor Vaughan wrote:
> It does get hairy when you mix the callbacks with the reverse
> dependencies. I can see having to iterate over the stack (mesh?) and
> check for loops and correctness while doing so. It could certainly turn
> into quite the recursive memory hog if not done properly.

Yes, as Henrik mentioned, you'd basically need an advanced resolution
mechanism to it right. I predict that there may well be cases that the
system will not just get right.

> However, I really do think that this is needed in the language as it
> seems to be a place where people frequently stumble when putting
> together what should be a straightforward thought process.

While those are cool ideas (and I'm still rooting for a lazy
evaluator...anyone? no?) I'd like to contend the proposition that actual
support for this style of solving problems should be supported (and by
extension, recommended).

We are currently trying to teach users that in order to model a system,
they typically need to find a way to non-ambiguously define what does
and what does not need to be part of the catalog. This may often boil
down to introducing a flag value in Hiera or somesuch.

In other words, there seems to be a consensus that "reflection" (for
lack of a better term) in manifests should be avoided in favor of a
common data source in which to base your queries.

And yes, I do realize that there are edge cases where this is quite
painful in comparison to `if defined(Class[x])`, but I guess this is us
discussing if those are worth the burden of supporting the scheme.

Final aside (although I believe that you are aware of this, still got
the urge to point this out): 'Don't reverse these two' will work on the
node level, but as soon as you're two or three levels of `include` deep,
it's wishful thinking at best. One should never rely on evaluation
order. This is precisely why we need defined() gone.

...rereading, the whole message has a distinct Get Off My Lawn feel to
it - sorry. Guess I'm that dude by now :-)

Cheers,
Felix

[Trevor Vaughan]

Hi Felix,

Yes, I'm very aware that the current recommendations are to very cleanly model your system to be able to non-ambiguously define your catalog.

However, there have always been arguments that the Puppet language should be declarative throughout which makes for more maintainable and intuitive manifests overall.

If you go look for posts surrounding this subject, I'd say that the consensus is because reflection just doesn't work, not because it shouldn't be there.

Frankly, it can solve a LOT of problems for you and make the system designer's life a lot easier.

Scenario 1: A requires B if C but not otherwise.

If I don't have reflection in this case, then I have to have documentation that says, hey, if you put A and C together on a system, remember to add B! This is prone to error and is something that the code should just "take care of" for me. I'm aware that you could use a role/profile model to take care of this but you're just making people write extra code for no really good reason except that the language doesn't actually address this issue. (I'll rant about the irritation of figuring out what code is doing in a role/profile system later).

Scenario 2: A requires B if module B is present but do something else otherwise.

This one is more about system introspection I suppose but we have all of these lovely Module Forge description files and we can't use them in the language! I would love to be able to do the following:

if $::module_mysql <= '1.2' { include 'mysql' } else { service { 'mysql': ensure => running, enable => true } }

You can then also use $::module_mysql to set up automatic Hiera hierarchies that run from version down to a default (hieradata/mysql/1.0, hieradata/mysql/0.9, hieradata/mysql/default) and life is magic and wonderful when modules on the forge change.

Scenario 1 is obviously what we're referring to in this thread and I think, with the way most cloud component architectures seem to be designed, the burden is being placed on the system user as opposed to the system designer in too many cases. And, frankly, some times you just need to get something working and you hit a point where you realize that you'd have to redesign the system to avoid using reflection and you just don't have time for that or it is going to add a LOT of unnecessary code.

Hope all this makes sense this early in the morning!

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/53A14820.3080907%40alumni.tu-berlin.de.

For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

Sorry to reply to myself but I just thought of the perfect example of why I want this.

puppetlabs-firewall

So, pretty much all modules need to call out to a firewall if one is enabled for obvious reasons.

However, you may have very real scenarios where you don't want puppetlabs-firewall managing your rules.

So, the following may apply for any given module that could use puppetlabs-firewall:

a) Have a variable that toggles firewall support off => Bad, nobody wants to declare variables that they don't have to, particularly when missing one means that you're going to end up with an unusable system

b) Have a variable that toggles firewall support on => Better, because you can't accidentally modify the system, but you probably want this 99% of the time so that means you have to declare a lot of variables or again, have an unusable system.

c) Have a global that toggles firewall support per node => Even Better because you can set something one time and call it done. But, you still have to declare a variable when you really shouldn't have to (and booo globals).

d) Use if defined => Aw, parse order dependent. So, declare 'include firewall' first! But that's not really very intuitive.

e) Use the new proposed magic => Woo, it "just works" if *anything* in the catalog includes 'firewall'. This way, if I don't want it, I don't get it in an intuitive fashion.

This also dovetails into my second scenario in the previous e-mail which would mean that I could do if $::module_firewall[:provider] == 'puppetlabs' and $::module_firewall[:version] >= 1.2 { include firewall{ :provider => 'puppetlabs', :version => '1.2' } which could happily include the *correct* firewall module for my custom module.

Yeah, I expanded the example some when thinking about it. It would be nice to be able to include the *correct* module for your code. To do this:

$::module_<foo> would be a hash of items that describes the Modulefile and $::module_foo without a key would just tell you if it is present in your catalog at all.

Also 'include' would not accept hashes and include the specified module. In the case that two are present on the system, you would need to deconflict your include code or somehow be able to set a default for your system (better due to legacy code). Perhaps, a 'default' entry in the Modulefile or somesuch.

Thanks,

Trevor

[Trevor Vaughan]

Grr:

Also 'include' *would* accept hashes and include the specified module. In the case that two are present on the system, you would need to deconflict your include code or somehow be able to set a default for your system (better due to legacy code). Perhaps, a 'default' entry in the Modulefile or somesuch.

[Felix Frank]

On 06/18/2014 12:56 PM, Trevor Vaughan wrote:
> e) Use the new proposed magic => Woo, it "just works" if *anything* in
> the catalog includes 'firewall'. This way, if I don't want it, I don't
> get it in an intuitive fashion.

FWIW, I can see this happen rather easily with a metaparameter like

firewall::rule { "whatever":
lazy => true,
trigger => Class[firewall],
}

*but* this will not work for includes (no, not even using the
resource-like declaration syntax) for much the same reason (I believe)
that class parameter overrides cannot work.

Also, Puppet can do this already, we just call it virtual resources. I
do feel though that many module interoperability use cases of the
firewall variety could be solved using a virtual resources based approach.

But this is essentially derailing the thread. If we find that we *do*
want this for classes, we'll have to kiss *some* frog.

[John Bollinger]

On Wednesday, June 18, 2014 4:22:16 AM UTC-5, Trevor Vaughan wrote:

Hi Felix,

Yes, I'm very aware that the current recommendations are to very cleanly model your system to be able to non-ambiguously define your catalog.

However, there have always been arguments that the Puppet language should be declarative throughout which makes for more maintainable and intuitive manifests overall.

If you go look for posts surrounding this subject, I'd say that the consensus is because reflection just doesn't work, not because it shouldn't be there.

To the extent that I get to say what the consensus is -- and I have been outspoken enough on the topic to suppose I have helped shape that consensus -- I'd say it's that most anything dependent on evaluation order does not work reliably, and reflection shouldn't be there because it cannot be relieved of evaluation-order dependencies.  (Sorry, Henrik, if I'm getting my different orders mixed up.)  To put it another way, I'd say reflection can't work.

If, however, some constrained context could be established in which to perform reflection, then perhaps some form of reflection could be introduced.  For example, it might be possible to inspect which classes are in the catalog from a construct wherein, or at a time when, no further classes can be added.  Adding bona fide resources would be allowed (else what would be the point?), but they would need to actively be prevented from adding classes.

 

Frankly, it can solve a LOT of problems for you and make the system designer's life a lot easier.

It does not solve any problems if it doesn't work reliably, and general reflection cannot work reliably.

 

Scenario 1: A requires B if C but not otherwise.

If I don't have reflection in this case, then I have to have documentation that says, hey, if you put A and C together on a system, remember to add B! This is prone to error and is something that the code should just "take care of" for me. I'm aware that you could use a role/profile model to take care of this but you're just making people write extra code for no really good reason except that the language doesn't actually address this issue. (I'll rant about the irritation of figuring out what code is doing in a role/profile system later).

Scenario 2: A requires B if module B is present but do something else otherwise.

This one is more about system introspection I suppose but we have all of these lovely Module Forge description files and we can't use them in the language! I would love to be able to do the following:

if $::module_mysql <= '1.2' { include 'mysql' } else { service { 'mysql': ensure => running, enable => true } }

You can then also use $::module_mysql to set up automatic Hiera hierarchies that run from version down to a default (hieradata/mysql/1.0, hieradata/mysql/0.9, hieradata/mysql/default) and life is magic and wonderful when modules on the forge change.

Scenario 1 is obviously what we're referring to in this thread and I think, with the way most cloud component architectures seem to be designed, the burden is being placed on the system user as opposed to the system designer in too many cases. And, frankly, some times you just need to get something working and you hit a point where you realize that you'd have to redesign the system to avoid using reflection and you just don't have time for that or it is going to add a LOT of unnecessary code.

Puppet already has a tool for separating the details of a resource from the decision about whether to include it in the catalog: virtual resources.  Those decisions can be tied to whether a given class is declared by having that class realize or (especially) collect the resources.  In many cases that can be done via a wrapper class or definition if the target class itself must not be modified.

Here's an idea: Collectors are already processed very late, so maybe they are late enough -- or could be made late enough -- to provide the kind of context needed for safe reflection.  One might then write something along these lines:

Firewall::Rule<| title == 'whatever' and defined == 'firewall' |>
@firewall::rule { "whatever":
  # ...
}

Of course, the collector wouldn't need actually to be close to the resource declaration, and it might be more general than just one resource, but it could be right there with the resource declaration to provide something much like an if defined() effect.

For that to be safe though, it would be necessary to prevent the collected resources from adding classes to the catalog, probably by failing the catalog if any of them try to do so.  That restriction might be reserved for when the 'defined' key is used in the collector's select expression, provided that such collectors were processed after all others.


John

[Joachim Thuau]

Every time I look at this, I keep thinking about the way Debian implement "feature packages" or "virtual packages". The Debian packages "providing a feature", which can then be depended upon by other packages. Some package that requires a MTA shouldn't care if I use exim, postfix or sendmail, as long as I have some sort of MTA package installed. 

In my case, I have a number of classes that define auth mechanisms (mutually exclusive), and we have other classes that require "some sort of auth". 

We have systems with "nslcd" and some with "sssd". Both provide auth, and can't be installed together. We have other classes that handles what those needs (setup kerberos and what not, for each of those). But then in other classes, we need to make sure we have "some" auth, and maybe a way to figure out which...

I'm looking at this point at combining the two classes into a "meta class" that takes a parameter, and allows me to setup all those things that depends on auth automagically in one class, but I keep thinking that i'm killing reusability at that point. (my magic "auth" class becomes way too specific to be shared)

What I would be after is the ability to have a module "masquerade" as a different name. so I can say "require auth" in some moduleA, and have ModuleAuth1 and ModuleAuth2 both say "I can provide "auth", and let the manager/user of the system define which one they want. 

I wish classes had the ability to do something like:

.../modules/sssd/manifests/init.pp:

class sssd as auth {

  # setup sssd auth here...

}

.../modules/nslcd/manifests/init.pp:

class nslcd as auth {

  # setup nslcd auth here ...

}

.../modules/ssh_authz/manifests/init.pp:

class ssh_authz {

  # something as simple as:

  require auth 

  # that would be difficult to "reconcile" the class name in the parser

  # or something like:

  require Feature['auth']

  # which would explicitly make this class "get me one of those" to be

  # resolved maybe later... but still enforce the dependency...

}

With something like that, you can have someone manage a box with one of the two first classes, and use the third one without having to "care" which scenario they have. If the first two are both included, that's obviously a problem, and can be easily detected (two classes are providing "auth"). if none are included, then a resource is missing. 

Now, all that being said, I have absolutely no knowledge of the way the parser works internally, so this could be very silly, and completely off the realm of possibilities...

[Henrik Lindberg]

On 2014-18-06 22:44, John Bollinger wrote:
>
>
> On Wednesday, June 18, 2014 4:22:16 AM UTC-5, Trevor Vaughan wrote:
>
> Hi Felix,
>
> Yes, I'm very aware that the current recommendations are to very
> cleanly model your system to be able to non-ambiguously define your
> catalog.
>
> However, there have always been arguments that the Puppet language
> should be declarative throughout which makes for more maintainable
> and intuitive manifests overall.
>
> If you go look for posts surrounding this subject, I'd say that the
> consensus is because reflection just doesn't work, not because it
> shouldn't be there.
>
>
>
> To the extent that I get to say what the consensus is -- and I have been
> outspoken enough on the topic to suppose I have helped shape that
> consensus -- I'd say it's that most anything dependent on evaluation
> order does not work reliably, and reflection shouldn't be there because
> it cannot be relieved of evaluation-order dependencies. (Sorry, Henrik,
> if I'm getting my different orders mixed up.) To put it another way,

> I'd say reflection *can't* work.
>
Agree, can never be made reliable. (And evaluation order it is).

> If, however, some constrained context could be established in which to
> perform reflection, then perhaps some form of reflection could be
> introduced. For example, it might be possible to inspect which classes
> are in the catalog from a construct wherein, or at a time when, no
> further classes can be added. Adding bona fide resources would be
> allowed (else what would be the point?), but they would need to actively
> be prevented from adding classes.
>
> Frankly, it can solve a LOT of problems for you and make the system
> designer's life a lot easier.
>
>
>

> It does not solve /any/ problems if it doesn't work reliably, and

> general reflection cannot work reliably.
>

Agree again.

> they are late enough -- or could be /made/ late enough -- to provide the

> kind of context needed for safe reflection. One might then write
> something along these lines:
>
> Firewall::Rule<| title == 'whatever' and defined == 'firewall' |>
> @firewall::rule { "whatever":
> # ...
> }
>
> Of course, the collector wouldn't need actually to be close to the
> resource declaration, and it might be more general than just one
> resource, but it could be right there with the resource declaration to
> provide something much like an if defined() effect.
>

The only thing this does is to move the problem to a new "phase".
Since the purpose is to add resources, naturally you can add defined
types, which in turn creates many resources. The set of rules then
operate on a moving target (constraint solving).

> For that to be safe though, it would be necessary to prevent the
> collected resources from adding classes to the catalog, probably by
> failing the catalog if any of them try to do so. That restriction might
> be reserved for when the 'defined' key is used in the collector's select
> expression, provided that such collectors were processed after all others.
>

I don't think we absolutely have to protect classes from being included.
In a way they are just resources only singletons.

I am thinking that a module can specify constraints as a predicate
(either on other modules being present, other classes, or resources, or
indeed on some abstract "capability"), and if the predicate becomes true
at the end of the catalog production, then it includes one class from
the module.

This adds a much asked for feature; autoinclude. Inclusion in the
catalog by just adding a module to the modulepath. (This is the reverse
dependency I was talking about earlier).

In all of these cases, some kind of constraint solver ability is needed
across the new feature, queries and the forming of dependencies (since
you most certainly need to also have control over the order in which the
"injected" classes/resources are later applied.

The other approach I can think of (I wrote about it earlier) is that you
make explicit statements about things that must have been evaluated
before "you" can be evaluated (i.e. "I need these Promises fulfilled /
defined before I can say what I want", and modules are given the ability
to define such promises. This (as John also points out) requires a
context in which a resolution of available Promises can take place
reliably). Maybe a Promise is nothing else but a class, but that you can
tie your evaluation to the outcome to the evaluation of one or many
other classes. (This is an area I hope to be able to work on ideas after
we have released puppet 4).

Regards
- henrik

[Trevor Vaughan]

Just to make everyone cringe heartily, I've found that I *can* do what I want but not in a way that's really reusable/understandable by others.

It just so happens that you can manipulate the catalog once applied to the system in various parts of custom types and I've hacked around this a bit by creating random resources as necessary in the past.

But I don't recommend this approach to anyone really unless you *really* need it....

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/5affcbe8-4a14-4e7f-95da-1e598f10fd5f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Felix Frank]

On 06/19/2014 03:13 AM, Trevor Vaughan wrote:
> It just so happens that you can manipulate the catalog once applied to
> the system in various parts of custom types and I've hacked around
> this a bit by creating random resources as necessary in the past.

Yes, I cringed. "Well" done ;-)

I stand by my assertion that we can model many such *resource*
crossdependency situations using the virtual resource feature. Enabling
conditional class inclusion is the actual holy grail here, I feel.

On 06/19/2014 12:22 AM, Joachim Thuau wrote:
> What I would be after is the ability to have a module "masquerade" as
> a different name. so I can say "require auth" in some moduleA, and
> have ModuleAuth1 and ModuleAuth2 both say "I can provide "auth", and
> let the manager/user of the system define which one they want.

> ...

> Now, all that being said, I have absolutely no knowledge of the way
> the parser works internally, so this could be very silly, and
> completely off the realm of possibilities...

Eh, no, that sounds actually pretty neat, but should be the subject of a
whole nother thread. I feel that such a feature could be designed
independently from the catalog introspection discussion.

Cheers,
Felix

[John Bollinger]

On Wednesday, June 18, 2014 6:07:02 PM UTC-5, henrik lindberg wrote:

On 2014-18-06 22:44, John Bollinger wrote:
>
> Puppet already has a tool for separating the details of a resource from
> the decision about whether to include it in the catalog: virtual
> resources.  Those decisions can be tied to whether a given class is
> declared by having that class realize or (especially) collect the
> resources.  In many cases that can be done via a wrapper class or
> definition if the target class itself must not be modified.
>
> Here's an idea: Collectors are already processed very late, so maybe
> they are late enough -- or could be /made/ late enough -- to provide the
> kind of context needed for safe reflection.  One might then write
> something along these lines:
>
> Firewall::Rule<| title == 'whatever' and defined == 'firewall' |>
> @firewall::rule { "whatever":
>    # ...
> }
>
> Of course, the collector wouldn't need actually to be close to the
> resource declaration, and it might be more general than just one
> resource, but it could be right there with the resource declaration to
> provide something much like an if defined() effect.
>
The only thing this does is to move the problem to a new "phase".
Since the purpose is to add resources, naturally you can add defined
types, which in turn creates many resources. The set of rules then
operate on a moving target (constraint solving).

I'm thinking about something more restricted than it may seem I've suggested: an ability only to reflect which classes are in the catalog, in a context and time where no new classes can be added (but resources can).  That characterization draws on my position that classes should not be considered resources, at least at DSL level, but you can say instead "(but resources other than classes can [be added])" if you prefer.

 

> For that to be safe though, it would be necessary to prevent the
> collected resources from adding classes to the catalog, probably by
> failing the catalog if any of them try to do so.  That restriction might
> be reserved for when the 'defined' key is used in the collector's select
> expression, provided that such collectors were processed after all others.
>
I don't think we absolutely have to protect classes from being included.
In a way they are just resources only singletons.

The idea was that this limited form of reflection escapes evaluation-order dependencies because it draws on information that will not subsequently change -- to wit, which classes are in the catalog.  That's the reason for preventing new classes from being added.

Now that I think about it, though, I'm no longer sure that would be sufficient.  I think this would introduce a new form of evaluation-order dependency involving the effects of collectors, since the full set of resources selected by any given collector can be determined reliably only after all resources that will be added to the catalog have been added, at least virtually.  On the other hand, maybe this problem isn't new -- it seems like collectors can already suffer from this problem on their own.

And I think any solution to that problem creates its own problems.  Consider this artificial example:

define loop::a ($index = 0) {
  @loop::a { "a$index": index => $index + 1 }
}

@loop::a { 'start': }

Loop::A<| |>

Obviously, you shouldn't write infinite loops, but my point is that on the one hand, whether that is in fact an infinite loop depends on the evaluator implementation, but on the other hand, an evaluator for which it is NOT an infinite loop does not reliably collect all resources that should be collected.  (And on the third hand, it might not even be possible to collect all the resources that should be collected.)

 

I am thinking that a module can specify constraints as a predicate
(either on other modules being present, other classes, or resources, or
indeed on some abstract "capability"), and if the predicate becomes true
at the end of the catalog production, then it includes one class from
the module.

But if a class is added as a result of that mechanism, then it turns out that the predicate wasn't evaluated at the end of catalog production after all.  That's not an issue for just one such constraint, but as soon as you have two or more you are right back to having an evaluation-order problem.  And consider also the issue with collectors that I raised above -- I think that probably applies here, too.

This adds a much asked for feature; autoinclude. Inclusion in the
catalog by just adding a module to the modulepath.

As opposed to adding "include 'mymodule'" to the top level of the starting-point manifest?  Or adding a file containing that to the starting-manifest directory?  What's hard about that?  The latter could even be arranged as part of module installation.

Or why not have a separate module path for autoinclude modules?  Users then select autoinclude behavior by where they put the module -- or even by where they put a symlink to the module.

I think simple autoinclude is a relatively tractable problem, but I guess you're looking at conditional autoinclusion.  Honestly, I'm not very comfortable with that.  Moreover, I'm inclined to think that whatever framework might be devised for evaluating which of the available modules to "auto"include would be no simpler, better, or easier than what we already have.

 

(This is the reverse
dependency I was talking about earlier).

In all of these cases, some kind of constraint solver ability is needed
across the new feature, queries and the forming of dependencies (since
you most certainly need to also have control over the order in which the
"injected" classes/resources are later applied.

I think you're saying there needs to be a way to set up dependencies between classes and resources added to the catalog during the "normal" evaluation phase and those added as a result of this hypothetical coda.  That supposes, I guess, that those cannot be declared either during the normal evaluation pass (e.g. via a collector) or during the additional one.  I'm uncertain whether that's true.

 

The other approach I can think of (I wrote about it earlier) is that you
make explicit statements about things that must have been evaluated
before "you" can be evaluated (i.e. "I need these Promises fulfilled /
defined before I can say what I want", and modules are given the ability
to define such promises. This (as John also points out) requires a
context in which a resolution of available Promises can take place
reliably). Maybe a Promise is nothing else but a class, but that you can
tie your evaluation to the outcome to the evaluation of one or many
other classes. (This is an area I hope to be able to work on ideas after
we have released puppet 4).

Basically that's a way to constrain evaluation order, right?  Such a mechanism would indeed address the core problem, and conditional evaluation can falls out as a special case.  It might even fit in easily with the current (future) evaluator, as it seems like it has some parallels with 'include'.  So, yes, this seems like a promising direction to explore.


John

[Henrik Lindberg]

> suggested: an ability only to reflect which *classes* are in the

> catalog, in a context and time where no new classes can be added (but
> resources can). That characterization draws on my position that classes
> should not be considered resources, at least at DSL level, but you can
> say instead "(but resources other than classes can [be added])" if you
> prefer.
>
> > For that to be safe though, it would be necessary to prevent the
> > collected resources from adding classes to the catalog, probably by
> > failing the catalog if any of them try to do so. That
> restriction might
> > be reserved for when the 'defined' key is used in the collector's
> select
> > expression, provided that such collectors were processed after
> all others.
> >
> I don't think we absolutely have to protect classes from being

> included..

> In a way they are just resources only singletons.
>
>
>
> The idea was that this limited form of reflection escapes
> evaluation-order dependencies because it draws on information that will
> not subsequently change -- to wit, which classes are in the catalog.
> That's the reason for preventing new classes from being added.
>

A new mechanism is then needed, since it is not possible to statically
derive the set of classes that will actually be included in the catalog
and what parameters they will have. It is possible (given a few caveats)
to know the set of classes that can potentially be included, but that is
about it.

> Now that I think about it, though, I'm no longer sure that would be
> sufficient. I think this would introduce a new form of evaluation-order
> dependency involving the effects of collectors, since the full set of
> resources selected by any given collector can be determined reliably
> only after all resources that will be added to the catalog have been
> added, at least virtually. On the other hand, maybe this problem isn't
> new -- it seems like collectors can already suffer from this problem on
> their own.
>
> And I think any solution to that problem creates its own problems.
> Consider this artificial example:
>
> define loop::a ($index = 0) {
> @loop::a { "a$index": index => $index + 1 }
> }
>
> @loop::a { 'start': }
>
> Loop::A<| |>
>
> Obviously, you shouldn't write infinite loops, but my point is that on
> the one hand, whether that is in fact an infinite loop depends on the
> evaluator implementation, but on the other hand, an evaluator for which
> it is NOT an infinite loop does not reliably collect all resources that
> should be collected. (And on the third hand, it might not even be
> possible to collect all the resources that should be collected.)
>

You can do loops now with resources. If you can also evaluate code as a
side effect of collection (or when dependencies are formed), one more
such looping possibility is added. IMO this makes things worse.

Dependencies are evaluated after all classes and resource have been
evaluated. An operand of a dependency may be a query (a.k.a collection).
Therefore, dependencies are added last (I think)). Until that point, all
references are unchecked "future references".

The issues with dependencies and conditional inclusion is that neither
side (the one that causes the inclusion as a side effect, or the side
that is being included) knows enough. (Maybe this is an imaginary
problem and dependencies are more coarse grained).

> The other approach I can think of (I wrote about it earlier) is that
> you
> make explicit statements about things that must have been evaluated
> before "you" can be evaluated (i.e. "I need these Promises fulfilled /
> defined before I can say what I want", and modules are given the
> ability
> to define such promises. This (as John also points out) requires a
> context in which a resolution of available Promises can take place
> reliably). Maybe a Promise is nothing else but a class, but that you
> can
> tie your evaluation to the outcome to the evaluation of one or many
> other classes. (This is an area I hope to be able to work on ideas
> after
> we have released puppet 4).
>
>
>
> Basically that's a way to constrain evaluation order, right? Such a
> mechanism would indeed address the core problem, and conditional
> evaluation can falls out as a special case. It might even fit in easily
> with the current (future) evaluator, as it seems like it has some
> parallels with 'include'. So, yes, this seems like a promising
> direction to explore.
>

Yes it is, and in a safe way. (I think this is the most promising, the
other ways to solve it that have come up all seem to result in a game of
Whack-a-Mole).

And, to expand on this idea a little. I think the exact same mechanism
is applicable when the resources are applied on an agent. i.e. it is an
evaluation plan, workflow, or orchestration if you like.

I doubt I will have time to work on these ideas before Puppet Conf - I
don't have this in presentable form yet, maybe something that can be
shared by then. Some of these ideas have implications that we are
thinking about and working into Puppet 4.0. We also considered these
ideas for the design of the next phase; the catalog builder (replacement
for the current compiler) - that works will take place during the 4x
series with the intent of releasing it as Puppet 5.0.

[Joachim Thuau]

PUP-2811, for the sake of making it on the radar... feel free to start a new thread on this one.  

[Trevor Vaughan]

I'd certainly be interested in working this through more at PuppetConf if there's time.

I also don't see why this needs to wait until 5.0. It should be 100% backward compatible with the 4.x series in theory (it would be nice anyway).

Thanks!

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/lnvots%24eod%241%40ger.gmane.org.

For more options, visit https://groups.google.com/d/optout.