RFC - Resource Defaults and Collection

[Henrik Lindberg]

Hi,
We (the server-team) have started to dig into the area of Resource
Defaults and Collection in our work on 3.7's future parser/evaluator
(and what will become Puppet 4.0).

We believe that there is an opportunity to improve on how Puppet now
works, and make it less confusing. We are not yet completely sure about
edge use-cases and the implications of what we would like to do - so we
would very much like to get your feedback.

Puppet currently allows setting defaults for resources - e.g.

File { mode => '0666' }

* These can also be set for user defined types.

* It is possible to make multiple such statements for the same resource
type, but additional statements may not set the same attributes

* When a default statement is evaluated, it is registered in its closure
scope (i.e. where it is located in source). e.g. a resource created in
Class a, gets defaults that are visible in the scope of Class a.

* All resources defined in that scope will potentially be affected by
those defaults (i.e. if the are found to have missing values at the end
of the compilation).

* At the end of the compilation, all resources will be processed and any
missing values are set from the registered defaults (from the
perspective of the resource's containment scope (i.e. where it was defined).

What are the problems with this?
---

* Defaults are applied after collection, thus if you try to use +> to
add to a virtual resource's attribute, the resource value(s) for the
attribute you are trying to append to will not include the defaults. As
you are also setting/overriding the values with the collection, the
default values are not included in the values, and you have to repeat
them (if they should be included).

* You cannot query (or so we suspect - we have to investigate to be
sure) for values that will be set via the defaults, so the collection
will miss the resources that later (after collection is completed) will
be given a value that would have made it included in the collection.

* It is possible to reopen classes, and introduce additional default
expressions after resources have been instantiated (but not evaluated).
This affects the attribute values the resources will end up having in
the catalog. (At some point later, we will make it an error to reopen a
class, but we don't think we will be able to fix this in time for 4.0).

* Evaluation of Collection Expressions is intertwined with resource
evaluation (same priority in the lazy evaluation queue), and it is
possible to play tricks with virtual resources, defaults and collections
that will (even if there is an attempt at considering potential defaults
when querying) never be able to consider such information that arrived
after the fact.

* Collection can be an operand in a dependency, and thus be evaluated
even later than the application of defaults.

Or, in other words: it is really confusing.

What we want to do:
---

* Make application of defaults eager so that when a resource is
instantiated, it will immediately get the registered and visible
defaults (for missing attributes), at *that* point of time in the
evaluation. This means that defaults become imperative (like variable
assignment).

* When resource defaults are applied eagerly they are also available
when doing collection (instantiation is naturally done before collection
- otherwise there is nothing to collect).

* There are edge cases where collection can be made at a point where all
resources that the query matches (will match) have not yet been created
(i.e. if the virtual resources are instantiated after the query is
evaluated). There seems to be logic that tries to find everything
(lazily at the end), but it is uncertain that it actually works
correctly in all cases. What we propose for the defaults have no effect
on this, it only changes what the resource attributes are when collecting.

* We want to change the function 'defined' to return the state of the
resource instead of just a boolean. Currently the concept of being
"defined" is vague, the function returns true for all kinds of resources
(virtual and exported) as well as those that are included in the
catalog. We want to return some kind of status (that is "truthy" to make
it backwards compatible), but that also tells if the resource is
actually realized (in the catalog). (We agree that using the defined
function is bad practice, but it is required to support certain
use-cases that would otherwise be very painful/impossible to handle).

* The File[foo][mode] syntax for lookup of a resource attribute will be
more sane, as it will correctly produce the value at that point in time
in the evaluation. It is either the value (explicit or default) that
will end up in the catalog for a realized resource, or the explicit or
default value of an unrealized virtual resource that *may* get a
different value when it is later realized (if realized at all). Thus,
with a combination of being able to know if a resource is realized or
not, it is possible to also be certain that the value is the value that
will end up in the catalog).

* In case you wonder, the ability to lookup an attribute is available
server side, when the catalog is produced. Later default values set by
the agent cannot (for obvious reasons) not be computed when the catalog
is being compiled.

We believe that the construct with lazy default application was required
when dynamic scoping was available, but we are honestly not sure about
the rationale behind the current design. There are plenty of
logged issues in Redmine regarding defaults, and collection, but that
did not help us much as they are for a variety of versions, and for when
dynamic scoping was available (plus a number of other issues where not
fixed), so it is very hard to tell which of the old issues that are
still relevant.

If you are doing advanced things with virtual resources, or advanced
composition of defaults where you explicitly depend on the evaluation
order, etc. to get the default values you want we would like to hear
from you, or naturally if you have input on any of the above. Also if
you have logged / or know details of relevant issues that can still be
replicated, we also want to hear from you.

I apologize if the explanations above are short and that there are no
examples. I am happy to write such to illustrate, but I felt that would
only make a long post even longer, and it would be better to show
examples to answer questions.

If there is interest in participating in a hangout to discuss these
matters, please ping, and we will set one up.

Regards
- henrik and andy

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[John Bollinger]

On Saturday, June 28, 2014 9:54:25 AM UTC-5, henrik lindberg wrote:

* Make application of defaults eager so that when a resource is
instantiated, it will immediately get the registered and visible
defaults (for missing attributes), at *that* point of time in the
evaluation. This means that defaults become imperative (like variable
assignment).

I had supposed that defaults worked that way already.  If currently they get applied only at collection time, then does that mean that exported resources don't get any defaults applied?

 

* When resource defaults are applied eagerly they are also available
when doing collection (instantiation is naturally done before collection
- otherwise there is nothing to collect).

* There are edge cases where collection can be made at a point where all
resources that the query matches (will match) have not yet been created
(i.e. if the virtual resources are instantiated after the query is
evaluated). There seems to be logic that tries to find everything
(lazily at the end), but it is uncertain that it actually works
correctly in all cases. What we propose for the defaults have no effect
on this, it only changes what the resource attributes are when collecting.

These edge cases would be where collection of virtual resources of a defined type causes new resources to be declared and/or resource overrides to be applied?  Perhaps also cases where there are two collectors over the same resource type, and one or both apply property overrides?

I think these kinds of cases are problematic in general.  It is unlikely that the catalog builder could ever reliably get them all right right.  In fact, I suspect that there are cases that haven't even a theoretically correct result (with either evaluation scheme for resource defaults).  It is wise to avoid being hung up on that.

 

* We want to change the function 'defined' to return the state of the
resource instead of just a boolean. Currently the concept of being
"defined" is vague, the function returns true for all kinds of resources
(virtual and exported) as well as those that are included in the
catalog. We want to return some kind of status (that is "truthy" to make
it backwards compatible), but that also tells if the resource is
actually realized (in the catalog).

Ok.  Since it's already evaluation-order dependent to use defined() at all, I don't think it's much worse to make it even more evaluation-order dependent.

 

(We agree that using the defined
function is bad practice, but it is required to support certain
use-cases that would otherwise be very painful/impossible to handle).

I think that depends on how generally you define the use cases and/or what constraints you put on them.  But we don't need to go off on that tangent here and now.

 

* The File[foo][mode] syntax for lookup of a resource attribute will be
more sane, as it will correctly produce the value at that point in time
in the evaluation. It is either the value (explicit or default) that
will end up in the catalog for a realized resource, or the explicit or
default value of an unrealized virtual resource that *may* get a
different value when it is later realized (if realized at all).

Not necessarily.  It is possible that a resource override will be evaluated later.

 

Thus,
with a combination of being able to know if a resource is realized or
not, it is possible to also be certain that the value is the value that
will end up in the catalog).

No, it isn't.  With all the overrides and lazy evaluation that can happen, it is never possible to have localized advance knowledge of what any resource parameter's value in the catalog that ultimately will be produced.  Such a prediction may be possible if you have a view of the whole manifest set (e.g. to see that there are no File overrides anywhere in it), but that's inconsistent with a module-based approach to organizing and using manifests.

I see I will have another piece of candy to advise sweet-toothed people to avoid for the sake of their teeth.  However much good that ever does.

 

We believe that the construct with lazy default application was required
when dynamic scoping was available, but we are honestly not sure about
the rationale behind the current design.

Are you conflating the dynamic scope of resource defaults with applying them in lazy fashion, or are these separate issues?  If you are proposing to change the scope of resource defaults, then that makes this a much more significant change than otherwise it would be.

Perhaps the rationale for the current design is to ensure that all the defaults that apply to a given resource declaration are effective.  Consider:

class a {
  File {
    owner => 'alice'
  }
  include 'c'
}

class b {
  File {
    mode => '0660'
  }
  include 'c'
}

class c {
  file { '/tmp/foo': ensure => 'file' }
}

include 'a'
include 'b'
# include 'c'


It would be nice if the properties of File['/tmp/foo'] were not dependent on which of classes 'a', 'b', and 'c' was evaluated first.

 

There are plenty of
logged issues in Redmine regarding defaults, and collection, but that
did not help us much as they are for a variety of versions, and for when
dynamic scoping was available (plus a number of other issues where not
fixed), so it is very hard to tell which of the old issues that are
still relevant.

Dynamic scope is not completely dead.  Perhaps this is the time to kill it (for resource defaults), but please be mindful that doing so is likely to break a lot of code.

I think I have to let this idea sink in a bit before I can form a firm opinion about it.  I started out very positive, but the closer I look at the idea, the more it loses its shine.  That's not to say it wouldn't be a net win, but there seem to be some trade-offs that I need to consider.


John

[Henrik Lindberg]

On 2014-30-06 16:32, John Bollinger wrote:
>
>
> On Saturday, June 28, 2014 9:54:25 AM UTC-5, henrik lindberg wrote:
>
> * Make application of defaults eager so that when a resource is
> instantiated, it will immediately get the registered and visible
> defaults (for missing attributes), at *that* point of time in the
> evaluation. This means that defaults become imperative (like variable
> assignment).
>
>
>
> I had supposed that defaults worked that way already. If currently they
> get applied only at collection time, then does that mean that exported
> resources don't get any defaults applied?
>

Not sure yet. I know defaults are applied very late, when the compiler
does its "finish" - I think it applies to all resources. I also think
that defaults are applied after collection has taken place.

>
> * When resource defaults are applied eagerly they are also available
> when doing collection (instantiation is naturally done before
> collection
> - otherwise there is nothing to collect).
>
> * There are edge cases where collection can be made at a point where
> all
> resources that the query matches (will match) have not yet been created
> (i.e. if the virtual resources are instantiated after the query is
> evaluated). There seems to be logic that tries to find everything
> (lazily at the end), but it is uncertain that it actually works
> correctly in all cases. What we propose for the defaults have no effect
> on this, it only changes what the resource attributes are when
> collecting.
>
>
>
> These edge cases would be where collection of virtual resources of a
> defined type causes new resources to be declared and/or resource
> overrides to be applied? Perhaps also cases where there are two
> collectors over the same resource type, and one or both apply property
> overrides?
>
> I think these kinds of cases are problematic in general. It is unlikely
> that the catalog builder could ever reliably get them all right right.
> In fact, I suspect that there are cases that haven't even a
> theoretically correct result (with either evaluation scheme for resource
> defaults). It is wise to avoid being hung up on that.
>

ok, thanks.

> * We want to change the function 'defined' to return the state of the
> resource instead of just a boolean. Currently the concept of being
> "defined" is vague, the function returns true for all kinds of
> resources
> (virtual and exported) as well as those that are included in the
> catalog. We want to return some kind of status (that is "truthy" to
> make
> it backwards compatible), but that also tells if the resource is
> actually realized (in the catalog).
>
>
>
> Ok. Since it's already evaluation-order dependent to use defined() at
> all, I don't think it's much worse to make it even more evaluation-order
> dependent.
>

:-)

> (We agree that using the defined
> function is bad practice, but it is required to support certain
> use-cases that would otherwise be very painful/impossible to handle).
>
>
>
> I think that depends on how generally you define the use cases and/or
> what constraints you put on them. But we don't need to go off on that
> tangent here and now.
>
>

yep, tabled.

> * The File[foo][mode] syntax for lookup of a resource attribute will be
> more sane, as it will correctly produce the value at that point in time
> in the evaluation. It is either the value (explicit or default) that
> will end up in the catalog for a realized resource, or the explicit or
> default value of an unrealized virtual resource that *may* get a
> different value when it is later realized (if realized at all).
>
>
>
> Not necessarily. It is possible that a resource override will be
> evaluated later.
>

yeah, Doh.

> Thus,
> with a combination of being able to know if a resource is realized or
> not, it is possible to also be certain that the value is the value that
> will end up in the catalog).
>
>
>
> No, it isn't. With all the overrides and lazy evaluation that can
> happen, it is never possible to have localized advance knowledge of what
> any resource parameter's value in the catalog that ultimately will be
> produced. Such a prediction may be possible if you have a view of the
> whole manifest set (e.g. to see that there are no File overrides
> anywhere in it), but that's inconsistent with a module-based approach to
> organizing and using manifests.
>

Yeah, not worth trying to predict.

yeah, that must be it.

> It would be nice if the properties of File['/tmp/foo'] were not
> dependent on which of classes 'a', 'b', and 'c' was evaluated first.
>

What we mean by eager setting of defaults would mean that the resource
gets its defaults when the file { '/tmp/foo': ensure => file} is queued
for evaluation. If the current runtime keeps track of both include
locations for c, and applies the defaults dynamically, then the order of
and be would not matter (with an eager application), but if c was
included before a or b it would.

If we want to keep the order independent then they must be applied late
(or appear to change dynamically throughout the evaluation).

> There are plenty of
> logged issues in Redmine regarding defaults, and collection, but that
> did not help us much as they are for a variety of versions, and for
> when
> dynamic scoping was available (plus a number of other issues where not
> fixed), so it is very hard to tell which of the old issues that are
> still relevant.
>
>
>
> Dynamic scope is not completely dead. Perhaps this is the time to kill
> it (for resource defaults), but please be mindful that doing so is
> likely to break a lot of code.
>

Yea, I realize this.

> I think I have to let this idea sink in a bit before I can form a firm
> opinion about it. I started out very positive, but the closer I look at
> the idea, the more it loses its shine. That's not to say it wouldn't be
> a net win, but there seem to be some trade-offs that I need to consider.
>
>

The two cases that we are concerned about is a) the ability to lookup
resource attribute values, and whether this should return the currently
known default values or not. We can see it as not looking up default at
all, or lookup the default values as known at that point in evaluation.
(Since resource overrides are also in play, and value is actually just a
prediction), and b) collection of virtual resources since it is unclear
what default values have been applied and can be queried...

The eager application of defaults is not a goal in itself.

And thank you for taking time responding on this! Much appreciated.

- henrik

[Henrik Lindberg]

Need to correct myself... see below...

What we cannot predict is what value will be set for realized resources
that have an undef value for an attribute. Those are the only ones that
can be modified with a Resource Override.

If we predict the default, an override may set it to something different
than the default.

An unrealized resource may be completely overridden, so there you need
to know if it is realized or not to be able to know how well the
prediction will hold for any value.

We *are* getting rid of dynamic scoping of resource defaults. This has
already been done in 3.7 and is in effect when using the --parser
future. Defaults follow lexical scoping.

[David Schmitt]

Hi Henrik,

On 2014-06-28 16:54, Henrik Lindberg wrote:
> Hi,
> We (the server-team) have started to dig into the area of Resource
> Defaults and Collection in our work on 3.7's future parser/evaluator
> (and what will become Puppet 4.0).

thanks for taking up this topic. As I'm a heavy user of export/collect,
Defaults were always taboo for my modules, due to many of the reasons
you're listing below.

Do I understand this correctly, that after

File { mode => 0664 }
file { "/tmp/foo":; }
File { owner => bin }
file { "/tmp/bar": mode => 0755; }

the actual resources will look like

file {
# receives mode from first, but not owner from second
"/tmp/foo": mode => 0644, owner => root;
# locally defines mode, but gets owner from second
"/tmp/bar": mode => 0755, owner => bin;
}

and later trying to override any of those attributes (e.g. in a
inherits) will fail?

> * When resource defaults are applied eagerly they are also available
> when doing collection (instantiation is naturally done before collection
> - otherwise there is nothing to collect).

There should be no difference between

File { mode => 0644 }
File <| title == 'example' |>

and

File <| title == 'example' |> { mode => 0644 }

except, perhaps that the latter could create an error when mode is
already set on @file ?

> * There are edge cases where collection can be made at a point where all
> resources that the query matches (will match) have not yet been created
> (i.e. if the virtual resources are instantiated after the query is
> evaluated). There seems to be logic that tries to find everything
> (lazily at the end), but it is uncertain that it actually works
> correctly in all cases. What we propose for the defaults have no effect
> on this, it only changes what the resource attributes are when collecting.

Seems obvious when applying defaults immediately.

> * We want to change the function 'defined' to return the state of the
> resource instead of just a boolean. Currently the concept of being
> "defined" is vague, the function returns true for all kinds of resources
> (virtual and exported) as well as those that are included in the
> catalog. We want to return some kind of status (that is "truthy" to make
> it backwards compatible), but that also tells if the resource is
> actually realized (in the catalog). (We agree that using the defined
> function is bad practice, but it is required to support certain
> use-cases that would otherwise be very painful/impossible to handle).

I think the most common use for "defined" is "is this resource already
part of the catalog up until now". I'm not totally sure why you want to
burden the Defaults story with *this* can of worms, but I think it might
be clearer to deprecate defined in its current form and replace it with
specific functions that inspect the current compilation state.

> * The File[foo][mode] syntax for lookup of a resource attribute will be
> more sane, as it will correctly produce the value at that point in time
> in the evaluation. It is either the value (explicit or default) that
> will end up in the catalog for a realized resource, or the explicit or
> default value of an unrealized virtual resource that *may* get a
> different value when it is later realized (if realized at all). Thus,
> with a combination of being able to know if a resource is realized or
> not, it is possible to also be certain that the value is the value that
> will end up in the catalog).

Even if it's not realized yet, the value can only change iff the current
value is undef, yes? In that case, I'd start with making accesses und
undef values on not-yet-realized resources invalid. Perhaps even detect
when a evaluation-order dependent value is accessed *after* it already
has acquired a value.

> * In case you wonder, the ability to lookup an attribute is available
> server side, when the catalog is produced. Later default values set by
> the agent cannot (for obvious reasons) not be computed when the catalog
> is being compiled.

When/how does the agent apply defaults to resources? Have you ment "the
state the agent reads from the running system"?

> We believe that the construct with lazy default application was required
> when dynamic scoping was available, but we are honestly not sure about
> the rationale behind the current design. There are plenty of
> logged issues in Redmine regarding defaults, and collection, but that
> did not help us much as they are for a variety of versions, and for when
> dynamic scoping was available (plus a number of other issues where not
> fixed), so it is very hard to tell which of the old issues that are
> still relevant.
>
> If you are doing advanced things with virtual resources, or advanced
> composition of defaults where you explicitly depend on the evaluation
> order, etc. to get the default values you want we would like to hear
> from you, or naturally if you have input on any of the above. Also if
> you have logged / or know details of relevant issues that can still be
> replicated, we also want to hear from you.

Another thing I'm avoiding, because it didn't work when I started with
export/collect, which you might want to look into:

define hell($content = $::hostname) {
file { "/tmp/$::name": content => $content; }
}

node b {
@@hell { $::hostname: }
}

node b {
Hell<<||>>
}

will create /tmp/a with the content "b", although I would have expected
the content to be "a" too.

> I apologize if the explanations above are short and that there are no
> examples. I am happy to write such to illustrate, but I felt that would
> only make a long post even longer, and it would be better to show
> examples to answer questions.
>
> If there is interest in participating in a hangout to discuss these
> matters, please ping, and we will set one up.

I don't see the immediate need for one, but would try to attend if invited.

Thanks for your time and work!

Regards, David

[Henrik Lindberg]

On 2014-01-07 8:52, David Schmitt wrote:
> Hi Henrik,
>
> On 2014-06-28 16:54, Henrik Lindberg wrote:

>>...

>> What we want to do:
>> ---
>>
>> * Make application of defaults eager so that when a resource is
>> instantiated, it will immediately get the registered and visible
>> defaults (for missing attributes), at *that* point of time in the
>> evaluation. This means that defaults become imperative (like variable
>> assignment).
>
> Do I understand this correctly, that after
>
> File { mode => 0664 }
> file { "/tmp/foo":; }
> File { owner => bin }
> file { "/tmp/bar": mode => 0755; }
>
> the actual resources will look like
>
> file {
> # receives mode from first, but not owner from second
> "/tmp/foo": mode => 0644, owner => root;
> # locally defines mode, but gets owner from second
> "/tmp/bar": mode => 0755, owner => bin;
> }
>

Yes, that is the idea.

> and later trying to override any of those attributes (e.g. in a
> inherits) will fail?
>

There are two situations for overrides using Resource Override (i.e.
File['tmp/foo'] { ... => ... }). In general, only unset attributes can
be overridden. Unsure if it is allowed currently to override a set
attribute in an inherited class (need to investigate).

>> * When resource defaults are applied eagerly they are also available
>> when doing collection (instantiation is naturally done before collection
>> - otherwise there is nothing to collect).
>
> There should be no difference between
>
> File { mode => 0644 }
> File <| title == 'example' |>
>
> and
>
> File <| title == 'example' |> { mode => 0644 }
>
> except, perhaps that the latter could create an error when mode is
> already set on @file ?
>

The current implementation allows any attribute to be set, even those
that are already set. The principle seems to be that immutability kicks
in when a resource is realized.

So, the two cases you show would not be equivalent. The first would set
mode to '0644' for File[example] if it was not already set, and the
second would enforce that it was set to '0644'.

>> * There are edge cases where collection can be made at a point where all
>> resources that the query matches (will match) have not yet been created
>> (i.e. if the virtual resources are instantiated after the query is
>> evaluated). There seems to be logic that tries to find everything
>> (lazily at the end), but it is uncertain that it actually works
>> correctly in all cases. What we propose for the defaults have no effect
>> on this, it only changes what the resource attributes are when
>> collecting.
>
> Seems obvious when applying defaults immediately.
>
>
>> * We want to change the function 'defined' to return the state of the
>> resource instead of just a boolean. Currently the concept of being
>> "defined" is vague, the function returns true for all kinds of resources
>> (virtual and exported) as well as those that are included in the
>> catalog. We want to return some kind of status (that is "truthy" to make
>> it backwards compatible), but that also tells if the resource is
>> actually realized (in the catalog). (We agree that using the defined
>> function is bad practice, but it is required to support certain
>> use-cases that would otherwise be very painful/impossible to handle).
>
> I think the most common use for "defined" is "is this resource already
> part of the catalog up until now". I'm not totally sure why you want to
> burden the Defaults story with *this* can of worms, but I think it might
> be clearer to deprecate defined in its current form and replace it with
> specific functions that inspect the current compilation state.
>

Now, the function lies - it says that virtual (unrealized) resources are
also "in the catalog", which is not really true. (The concept of
being "defined" is very loosely, uhm... defined)

>> * The File[foo][mode] syntax for lookup of a resource attribute will be
>> more sane, as it will correctly produce the value at that point in time
>> in the evaluation. It is either the value (explicit or default) that
>> will end up in the catalog for a realized resource, or the explicit or
>> default value of an unrealized virtual resource that *may* get a
>> different value when it is later realized (if realized at all). Thus,
>> with a combination of being able to know if a resource is realized or
>> not, it is possible to also be certain that the value is the value that
>> will end up in the catalog).
>
> Even if it's not realized yet, the value can only change iff the current
> value is undef, yes? In that case, I'd start with making accesses und
> undef values on not-yet-realized resources invalid. Perhaps even detect
> when a evaluation-order dependent value is accessed *after* it already
> has acquired a value.
>

If unrealized, the value can change when the resource is realized. A
realized resource attribute cannot be changed. An undef realized value
can change.

Sorry, but I don't quite understand what you were saying about making
access invalid. Can you elaborate?

>> * In case you wonder, the ability to lookup an attribute is available
>> server side, when the catalog is produced. Later default values set by
>> the agent cannot (for obvious reasons) not be computed when the catalog
>> is being compiled.
>
> When/how does the agent apply defaults to resources? Have you ment "the
> state the agent reads from the running system"?
>

yes, the agent, when the catalog is applied to the node triggers logic
in types and providers that may "set defaults" depending on the state of
the system where it is running.

>> We believe that the construct with lazy default application was required
>> when dynamic scoping was available, but we are honestly not sure about
>> the rationale behind the current design. There are plenty of
>> logged issues in Redmine regarding defaults, and collection, but that
>> did not help us much as they are for a variety of versions, and for when
>> dynamic scoping was available (plus a number of other issues where not
>> fixed), so it is very hard to tell which of the old issues that are
>> still relevant.
>>
>> If you are doing advanced things with virtual resources, or advanced
>> composition of defaults where you explicitly depend on the evaluation
>> order, etc. to get the default values you want we would like to hear
>> from you, or naturally if you have input on any of the above. Also if
>> you have logged / or know details of relevant issues that can still be
>> replicated, we also want to hear from you.
>
> Another thing I'm avoiding, because it didn't work when I started with
> export/collect, which you might want to look into:
>
> define hell($content = $::hostname) {
> file { "/tmp/$::name": content => $content; }
> }
>
> node b {
> @@hell { $::hostname: }
> }
>
> node b {
> Hell<<||>>
> }
>
> will create /tmp/a with the content "b", although I would have expected
> the content to be "a" too.
>

Not sure I follow, where would the "a" come from? There is no exported
resource of type Hell when the node is named 'a' (??)

>> I apologize if the explanations above are short and that there are no
>> examples. I am happy to write such to illustrate, but I felt that would
>> only make a long post even longer, and it would be better to show
>> examples to answer questions.
>>
>> If there is interest in participating in a hangout to discuss these
>> matters, please ping, and we will set one up.
>
> I don't see the immediate need for one, but would try to attend if invited.
>
> Thanks for your time and work!
>

Thanks for taking time commenting.

Regards
- henrik

[John Bollinger]

On Monday, June 30, 2014 8:11:47 PM UTC-5, henrik lindberg wrote:

Need to correct myself... see below...

What we cannot predict is what value will be set for realized resources
that have an undef value for an attribute. Those are the only ones that
can be modified with a Resource Override.

That doesn't appear to be consistent with the docs.  Such a restriction applies in certain cases, but not in the most common ones.  In particular, the docs say specifically that the restriction does not apply when subclasses override resources of a superclass, or when resources are overridden via a collector.  That was always my understanding, so I'm glad to see the docs backing me up.  I never noticed the change allowing overrides in any other cases, but I guess if you use that capability and then are you restricted to assigning values to previously unmanaged properties.  Or so say the docs.

 

If we predict the default, an override may set it to something different
than the default.

An unrealized resource may be completely overridden, so there you need
to know if it is realized or not to be able to know how well the
prediction will hold for any value.

No, I think all resources are pretty much in the same boat there, whether realized or not, unless this is something that is being changed in Puppet 4.  And I hope it isn't, because the problems that will attend changing the scope of resource defaults pale in comparison with those that would attend changing the rules for resource overrides.

We *are* getting rid of dynamic scoping of resource defaults. This has
already been done in 3.7 and is in effect when using the --parser
future. Defaults follow lexical scoping.

May the almighty have mercy on your souls.

In that case, the evaluation-order issues around resource defaults are mostly moot.  As the relevant statements will be easily determined and their relative evaluation order readily seen.

 

> The two cases that we are concerned about is a) the ability to lookup
> resource attribute values, and whether this should return the currently
> known default values or not. We can see it as not looking up default at
> all, or lookup the default values as known at that point in evaluation.
> (Since resource overrides are also in play, and value is actually just a
> prediction),

With the narrowed scope that defaults will have, it seems safe and reasonable for lookups to return applicable defaults.

 

and b) collection of virtual resources since it is unclear
> what default values have been applied and can be queried...

Since defaults will be lexically scoped and eagerly evaluated, I see no reason for excluding collector predicates matching against property values assigned by that means.  The results will be predictable, and generally less surprising.

John

[David Schmitt]

John seems to have done more reading on this topic. If it turns out to
be true that subclasses can change already set attribute values (at
least +> will do so anyways), the Foo[x][y] is already in a deep
hellhole of undefinedness and evaluation order dependency.

>>> * When resource defaults are applied eagerly they are also
>>> available
>>> when doing collection (instantiation is naturally done before
>>> collection
>>> - otherwise there is nothing to collect).
>>
>> There should be no difference between
>>
>> File { mode => 0644 }
>> File <| title == 'example' |>
>>
>> and
>>
>> File <| title == 'example' |> { mode => 0644 }
>>
>> except, perhaps that the latter could create an error when mode is
>> already set on @file ?
>>
> The current implementation allows any attribute to be set, even those
> that are already set. The principle seems to be that immutability
> kicks in when a resource is realized.
>
> So, the two cases you show would not be equivalent. The first would
> set
> mode to '0644' for File[example] if it was not already set, and the
> second would enforce that it was set to '0644'.

Hmmm ...

@file { "/tmp/fail": content => '', tag => [ 'example1', 'example2'
]; }
File <| tag == 'example1' |> { mode => 0644 }
File <| tag == 'example2' |> { mode => 0755 }

puppet (apply, 3.1.1) will create /tmp/fail with 0755. Changing the
order of the two collectors changes the mode of /tmp/fail.

I'm not saying it shouldn't be fixed (probably 4.0 is a good target to
do so; creating warnings for the "wrong" cases in 3.7). I'm just saying
that I don't see the connection to resource defaults.

I was trying to say that when an attribute accessor would fail when
accessing still-mutable values, it would not silently switcharoo values
when the evaluation order changes. If on the other hand an attribute
access would change between an actual value and undef or some
not-yet-overwritten default depending on evaluation order, that would be
a reason to ban its usage from most systems.

Given the weak immutability (ha!) of attributes as shown in my example
above, attribute accessors are already very dangerous.

>>> * In case you wonder, the ability to lookup an attribute is
>>> available
>>> server side, when the catalog is produced. Later default values set
>>> by
>>> the agent cannot (for obvious reasons) not be computed when the
>>> catalog
>>> is being compiled.
>>
>> When/how does the agent apply defaults to resources? Have you ment
>> "the
>> state the agent reads from the running system"?
>>
> yes, the agent, when the catalog is applied to the node triggers
> logic in types and providers that may "set defaults" depending on the
> state of the system where it is running.

That would be Puppet::Type#newtype#newproperty#defaults_to logic then?
I would consider types whose defaults_to implementation acts differently
on the master and the agent as Very Special[tm] and dangerous. The
default provider being a striking counter example of course.

Aggg. The first node should have been called 'a':

>> Another thing I'm avoiding, because it didn't work when I started
>> with
>> export/collect, which you might want to look into:
>>
>> define hell($content = $::hostname) {
>> file { "/tmp/$::name": content => $content; }
>> }
>>

>> node a { # fixed

>> @@hell { $::hostname: }
>> }
>>
>> node b {

>> Hell<<||>> # Hell['a']'s $content's default "$::hostname" is
>> evaluated locally as 'b'

>> }
>>
>> will create /tmp/a with the content "b", although I would have
>> expected
>> the content to be "a" too.

.
.
.



A different idea to reduce the scope of the problem: Decouple the
defaults discussion from the bog of other undefined behaviour by moving
default setting to a very strict localized syntax:

file {
mode => 0321,
owner => weird;
"/tmp/foo": content => 'foo';
"/tmp/bar": content => 'bar';
}

could be a shorthand for setting those attributes separately on all
files in this block. That would make it immediately obvious, that those
defaults have a very limited scope and are applied before the closing
brace of the file{}. They become syntactically "atomic". They cannot
leak to other files in the same class by scrolling off someone's mind.
They cannot interfere with collection. The old syntax can be handled
separately for compatibility.


That doesn't change that all the other
mutability/evaluation-order/parse-order problems around collection and
overriding may have to be addressed, but it would decouple the problems.






Regards, David

[John Bollinger]

On Wednesday, July 2, 2014 7:41:18 AM UTC-5, David Schmitt wrote:

John seems to have done more reading on this topic.

Henrik's description conflicted with what I thought I knew, so it made sense to check the docs.

 

If it turns out to
be true that subclasses can change already set attribute values (at
least +> will do so anyways), the Foo[x][y] is already in a deep
hellhole of undefinedness and evaluation order dependency.

Indeed so.  Resource overrides of any kind have always been a bit dicey, and making resource attributes readable really amplifies that.  Overrides would still be an issue even if they were universally restricted to attributes having as-yet undefined values because omitting an attribute is as much an explicit choice as specifying one is.  Assigning an attribute can completely change the meaning of a resource.  Consider:

exec { '/bin/create-world-peace': }

Exec['/bin/create-world-peace'] {
  onlyif => '/bin/rm -rf /*'
}

Hmmm ...

     @file { "/tmp/fail": content => '', tag => [ 'example1', 'example2'
]; }
     File <| tag == 'example1' |> { mode => 0644 }
     File <| tag == 'example2' |> { mode => 0755 }

puppet (apply, 3.1.1) will create /tmp/fail with 0755. Changing the
order of the two collectors changes the mode of /tmp/fail.

Indeed it does.  The docs warn about that specifically.  Overrides of any kind are dicey, but overrides via collectors are by far the worst offenders.
 

> Now, the function lies - it says that virtual (unrealized) resources
> are
> also "in the catalog", which is not really true. (The concept of
> being "defined" is very loosely, uhm... defined)

I'm not saying it shouldn't be fixed (probably 4.0 is a good target to
do so; creating warnings for the "wrong" cases in 3.7). I'm just saying
that I don't see the connection to resource defaults.

I don't see it either, though they play in the same neighborhood by virtue of their various associations with collectors.  To wit, evaluation of a collector can change a resource from being 'virtual' to being 'in the catalog', and it is an open question whether or how collector predicates should interact with resource defaults.

 

Yes, they are.  But, oh, so tempting....  That's where my candy metaphor came from.

 

>>> * In case you wonder, the ability to lookup an attribute is
>>> available
>>> server side, when the catalog is produced. Later default values set
>>> by
>>> the agent cannot (for obvious reasons) not be computed when the
>>> catalog
>>> is being compiled.
>>
>> When/how does the agent apply defaults to resources? Have you ment
>> "the
>> state the agent reads from the running system"?
>>
> yes, the agent, when the catalog is applied to the node triggers
> logic in types and providers that may "set defaults" depending on the
> state of the system where it is running.

That would be Puppet::Type#newtype#newproperty#defaults_to logic then?
I would consider types whose defaults_to implementation acts differently
on the master and the agent as Very Special[tm] and dangerous. The
default provider being a striking counter example of course.

I had supposed Henrik simply meant what the chosen provider (not the type) does when no value is specified for some parameter -- i.e. exactly when the type does not use #defaults_to.

 

A different idea to reduce the scope of the problem: Decouple the
defaults discussion from the bog of other undefined behaviour by moving
default setting to a very strict localized syntax:

   file {
       mode => 0321,
       owner => weird;
     "/tmp/foo": content => 'foo';
     "/tmp/bar": content => 'bar';
   }

could be a shorthand for setting those attributes separately on all
files in this block. That would make it immediately obvious, that those
defaults have a very limited scope and are applied before the closing
brace of the file{}. They become syntactically "atomic". They cannot
leak to other files in the same class by scrolling off someone's mind.
They cannot interfere with collection. The old syntax can be handled
separately for compatibility.

I think that's pretty similar to the proposed new behavior for defaults, just retrospective instead of prospective -- they apply only to previous (as opposed to subsequent) statements in the same lexical scope, and they take effect immediately when declared (as opposed to immediately when each affected resource is declared).

There was some discussion around the idea that one needed to track whether an attribute obtained its value from a resource default so as to know whether it could be overridden (outside a subclass and not via a collector), but I'm not convinced that's necessary.  I anyway don't think it's a fundamental aspect of Henrik's proposal.  If ignoring the distinction means less resource overriding then that's not necessarily a bad thing.

 

That doesn't change that all the other
mutability/evaluation-order/parse-order problems around collection and
overriding may have to be addressed, but it would decouple the problems.

I agree that it would simplify matters along several dimensions if defaults were applied immediately, without distinguishing between defaulted parameter values and others, either at the point where affected resources are declared (for Lindberg-style defaults) or at the point where the defaults themselves are declared (for Schmitt-style defaults, if I understand that proposal correctly).  That is basically how I though they worked already, though I now understand at least some aspects of why currently they don't actually work that way.

Any way around, I think there will be a major upheaval when the scope of resource defaults changes.  Would it at least be possible for their scope to continue to extend into defined type instances declared within their lexical scope?  That would mitigate the problems a bit, without causing any unpredictable evaluation-order dependencies.  That would also be consistent with how the metaparameters of a defined-type instance are forwarded to the resources it contains.  The semantics of wrapper definitions will otherwise change significantly, yet subtly.

John

[David Schmitt]

On 2014-07-02 21:55, John Bollinger wrote:

> at the point where affected resources
> are declared (for Lindberg-style defaults) or at the point where the
> defaults themselves are declared (for Schmitt-style defaults, if I
> understand that proposal correctly).

My proposal would actually collapse those two points: the values only
affect the resources declared in the same resource block as the values
themselves. I'll explain my example more:


No defaults:

file {
"/tmp/foo":
content => 'foo',

mode => 0321,
owner => weird;

"/tmp/bar":
content => 'bar',

mode => 0321,
owner => weird;
}

equivalent Schmitt-style shortcutting:

file {
# defaults block (no title!)

mode => 0321,
owner => weird;

# actual resources block (no mode, owner)

"/tmp/foo": content => 'foo';
"/tmp/bar": content => 'bar';
}

the weird owner and mode do not apply to any resources outside of this
File declaration.
The resulting resource would not remember whether the parameter value
was set directly (content in the example) or with a shortcut (mode,
owner).

> Any way around, I think there will be a major upheaval when the scope
> of
> resource defaults changes. Would it at least be possible for their
> scope
> to continue to extend into defined type instances declared within
> their
> lexical scope? That would mitigate the problems a bit, without
> causing any
> unpredictable evaluation-order dependencies. That would also be
> consistent
> with how the metaparameters of a defined-type instance are forwarded
> to the
> resources it contains. The semantics of wrapper definitions will
> otherwise
> change significantly, yet subtly.

I can only remember *seeing* Defaults in modules when they leaked to
somewhere I didn't want them to. That's probably the root cause for my
idea of really restricting where the default values can go.


Regards, David

[John Bollinger]

Here's another idea: how about removing resource defaults altogether?  My take on them has always been that their dynamic scope was their most important feature.  If that's going away then what's left is just a minor piece of syntactic sugar, and keeping them in that restricted form is certain to cause more pain than than just dropping them would.

[Eric Sorenson]

On Jul 3, 2014, at 1:57 PM, John Bollinger <john.bo...@stjude.org> wrote:

> Here's another idea: how about removing resource defaults altogether? My take on them has always been that their dynamic scope was their most important feature. If that's going away then what's left is just a minor piece of syntactic sugar, and keeping them in that restricted form is certain to cause more pain than than just dropping them would.

I’ve only been half-following the discussion, but I was starting to think along these lines too — are there real user benefits in resource defaults that can’t be achieved through other means?

Eric Sorenson - eric.s...@puppetlabs.com - freenode #puppet: eric0
puppet platform // coffee // techno // bicycles

[Henrik Lindberg]

> is certain to cause more pain than than just dropping them would..
>

There are several kinds of scoping in effect. It used to be dynamic,
which if I understood this correctly meant that defaults applied from
where something was defined as well as where it was included.

We now flipped that to follow how variables are resolved, if not defined
in the same scope, it goes to inherited, and then global scope. I agree
that this pretty much renders defaults useless as they have to be global
or repeated per class / define. It is only useful if a define or class
creates many resources.

If defaults followed lexical scope / (namespaces). Then the defaults
would apply to the namespace it is defined in, and all inner namespaces.
That would make them more useful while also safe to use. They would not
apply to something that is included.

Have no clear opinion on what is best here yet.

-henrik

[Henrik Lindberg]

simply allowing the keyword default describe the default would do it. i.e.

file {
default:
mode => '0321',
owner => weird;

'tmp/foo':
content => foo;

'tmp/bar':
content => bar;
}

And, if you wanted a resource with the actual title 'default', you would
quote it.

(We cannot accept a resource without a title due to how the grammar
works - long story)

I can also imagine the capability to reference default values via name -
set elsewhere. It would then be totally up to the designer to configure
them and use them, and all references would be clear.
Being able to set them from a hash is also a lot more powerful since the
content of the hash can be computed much more easily than
the resource default expressions where you basically have to do any
computations first and store in a hash or variable, and then construct
the defaults expression (or some variant with embedded complex logic
inside of the defaults expression which is even messier).

So, I can imagine syntax like below to reference a set of default values:

e.g.
$file_defaults = { mode => '0321', owner => weird }

file {
default: $file_defaults;

'tmp/foo':
content => foo;

'tmp/bar':
content => bar;

}


> I can only remember *seeing* Defaults in modules when they leaked to
> somewhere I didn't want them to. That's probably the root cause for my
> idea of really restricting where the default values can go.
>

I like the idea of exactly, and explicitly defining what the defaults
are - no surprises. And, it is up to the designer of a module to expose
any wanted variability which is now precise. You can have multiple
(conflicting) sets of defaults that apply to different groups of
resources in the same context etc. (this is impossible today).

If we do this, then we can drop the current resource defaults expression
completely!

Schmitt-style ftw !!

[Erik Dalén]

Due to the dynamic scoping of them I only really use them for setting global defaults. And some mechanism for doing that is really needed.

Using them inside classes is really mostly syntactic sugar to get the code shorter. But I've also seen them used to conditionally set an attribute. So not set it to value or undef, to set it to value or not set it at all, as that has slightly different semantics. But that would also really be better to do with some sort of postfix if condition in the resource declaration.

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/lp5163%2472b%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[David Schmitt]

On 2014-07-04 18:31, Erik Dalén wrote:
> Due to the dynamic scoping of them I only really use them for setting
> global defaults. And some mechanism for doing that is really needed.

I've used to do that to set file buckets and Exec#path. But importing
modules really put a stop to that as I saw that I didn't really need that:

> https://github.com/DavidS/dasz-configuration/blob/master/manifests/site.pp

> Using them inside classes is really mostly syntactic sugar to get the
> code shorter. But I've also seen them used to conditionally set an
> attribute. So not set it to value or undef, to set it to value or not
> set it at all, as that has slightly different semantics. But that would
> also really be better to do with some sort of postfix if condition in
> the resource declaration.

Like this:

file { "/foo": ;}

if $x { File["/foo"] { source => $x } }

?

See

> https://github.com/DavidS/dasz-configuration/blob/master/modules/hosting/manifests/customer_service.pp#L17

for a more elaborate example.


Regards, David

[John Bollinger]

On Thursday, July 3, 2014 8:29:48 PM UTC-5, henrik lindberg wrote:

On 2014-03-07 22:57, John Bollinger wrote:
> Here's another idea: how about removing resource defaults altogether?
> My take on them has always been that their dynamic scope was their most
> important feature.  If that's going away then what's left is just a
> minor piece of syntactic sugar, and keeping them in that restricted form
> is certain to cause more pain than than just dropping them would..
>

There are several kinds of scoping in effect. It used to be dynamic,
which if I understood this correctly meant that defaults applied from
where something was defined as well as where it was included.

Yes, with the clarification that the "where it was included" part was recursive.  I had always thought that created an evaluation-order dependency for defaults that weren't global.  Probably it once did, but the code described earlier for binding defaults late seems to be an attempt to address that issue.

 

We now flipped that to follow how variables are resolved, if not defined
in the same scope, it goes to inherited, and then global scope.

Hmm.  That's not how I read your description the first time, but I see how it could be read that way.  I don't think it's consistent with eager application, however, for there is no inherent constraint on classes being evaluated in an order that makes that work.  Consider:

class mymodule {
  File { group => 'example' }
}

class mymodule::example {
  file { '/tmp/example': owner => 'root', ensure => 'file' }
}

include 'mymodule::example'
include 'mymodule'

File { mode => '0400' }


What should File['/tmp/example']['group'] be?  And  File['/tmp/example']['mode']? If part of the objective is to avoid binding defaults late without (re)opening the door to issues around evaluation order uncertainty, then you can rely only on the local lexical scope for defaults.  And yes, variable references do suffer from that issue already.

 

I agree
that this pretty much renders defaults useless as they have to be global
or repeated per class / define. It is only useful if a define or class
creates many resources.

 

If defaults followed lexical scope / (namespaces). Then the defaults
would apply to the namespace it is defined in, and all inner namespaces.
That would make them more useful while also safe to use. They would not
apply to something that is included.

It's not clear to me how that last alternative differs from the previous one, nor whether it fares any better on the evaluation-order front relative to my example above.  I'm inclined to suspect that any appreciable measure of "safety" for resource defaults requires either restricting them to their lexical scope alone or binding them late.  I think the idea of changing their scope must be approached with extreme caution, however, as the effects of doing so could be subtle and difficult for users to predict, yet potentially very harmful.

John

[Henrik Lindberg]

The evaluation will evaluate both classes before the default expression.
The resources are thus created before the default is added. If we do go
for eager defaults, then the two resources would not get an default values.

> If part of the objective is to avoid
> binding defaults late without (re)opening the door to issues around
> evaluation order uncertainty, then you can rely only on the local
> lexical scope for defaults. And yes, variable references do suffer from
> that issue already.
>
> I agree
> that this pretty much renders defaults useless as they have to be
> global
> or repeated per class / define. It is only useful if a define or class
> creates many resources.
>
> If defaults followed lexical scope / (namespaces). Then the defaults
> would apply to the namespace it is defined in, and all inner

> namespaces..

> That would make them more useful while also safe to use. They would not
> apply to something that is included.
>
>
>
> It's not clear to me how that last alternative differs from the previous
> one, nor whether it fares any better on the evaluation-order front
> relative to my example above.

If we do defaults per named scope and not do late binding, care has
to be taken to introduce entities in the right order (to get them
evaluated in the wanted order - you want to evaluate nested named
elements after outer elements. This is probably just as confusing to
puppet users as the current way.

> I'm inclined to suspect that any
> appreciable measure of "safety" for resource defaults requires either

> restricting them to their lexical scope alone or binding them late.. I

> think the idea of changing their scope must be approached with extreme
> caution, however, as the effects of doing so could be subtle and
> difficult for users to predict, yet potentially very harmful.
>

Yeah, this is complex, hence this discussion trying to figure out what
to do and how painful certain types of changes would be.

Also see the new thread I started "RFC2 Resource Defaults" that
continues on the Schmitt style for defaults and proposed a cleanup of
the grammar.

- henrik

[John Bollinger]

On Monday, July 7, 2014 9:39:06 AM UTC-5, henrik lindberg wrote:

On 2014-07-07 15:43, John Bollinger wrote:

[...]  Consider:

>
> class mymodule {
>    File { group => 'example' }
> }
>
> class mymodule::example {
>    file { '/tmp/example': owner => 'root', ensure => 'file' }
> }
>
> include 'mymodule::example'
> include 'mymodule'
>
> File { mode => '0400' }
>
>
> What should File['/tmp/example']['group'] be?  And
> File['/tmp/example']['mode']?

The evaluation will evaluate both classes before the default expression.
The resources are thus created before the default is added. If we do go
for eager defaults, then the two resources would not get an default values.

Ok, that's what I expected, but take a closer look at the classes.  Class[mymodule] declares a resource default for the File resource, and Class[mymodule::example] declares a file instance, but the later is 'include'd before the former.  I am supposing that with eager defaults, the File instance in Class[mymodule::example] will not see the resource default this way, but that it would see it if the order of the 'include' statements were reversed (following the same resolution rules as for variables).

That reversing the order in which the classes are evaluated could change which defaults are applied -- if indeed that's the case -- would be an unhappy consequence.

 

> If part of the objective is to avoid
> binding defaults late without (re)opening the door to issues around
> evaluation order uncertainty, then you can rely only on the local
> lexical scope for defaults.  And yes, variable references do suffer from
> that issue already.
>
>     I agree
>     that this pretty much renders defaults useless as they have to be
>     global
>     or repeated per class / define. It is only useful if a define or class
>     creates many resources.
>
>     If defaults followed lexical scope / (namespaces). Then the defaults
>     would apply to the namespace it is defined in, and all inner
>     namespaces..
>     That would make them more useful while also safe to use. They would not
>     apply to something that is included.
>
>
>
> It's not clear to me how that last alternative differs from the previous
> one, nor whether it fares any better on the evaluation-order front
> relative to my example above.

If we do defaults per named scope and not do late binding, care has
to be taken to introduce entities in the right order (to get them
evaluated in the wanted order - you want to evaluate nested named
elements after outer elements. This is probably just as confusing to
puppet users as the current way.

I'm still not catching the distinction between that and resolving defaults "how variables are resolved".  Probably I'm just being dense, but perhaps I don't know how variables are resolved as well as I think I do.

In any case, though, it looks like you concur with me that there are evaluation-order consequences attending these approaches.

John

[Henrik Lindberg]

On 2014-07-07 20:54, John Bollinger wrote:
>
>
> On Monday, July 7, 2014 9:39:06 AM UTC-5, henrik lindberg wrote:
>
> On 2014-07-07 15:43, John Bollinger wrote:
> [...] Consider:
> >
> > class mymodule {
> > File { group => 'example' }
> > }
> >
> > class mymodule::example {
> > file { '/tmp/example': owner => 'root', ensure => 'file' }
> > }
> >
> > include 'mymodule::example'
> > include 'mymodule'
> >
> > File { mode => '0400' }
> >
> >
> > What should File['/tmp/example']['group'] be? And
> > File['/tmp/example']['mode']?
>
> The evaluation will evaluate both classes before the default

> expression..

> The resources are thus created before the default is added. If we do go
> for eager defaults, then the two resources would not get an default
> values.
>
>
>
> Ok, that's what I expected, but take a closer look at the classes.
> Class[mymodule] declares a resource default for the File resource, and
> Class[mymodule::example] declares a file instance, but the later is
> 'include'd before the former. I am supposing that with eager defaults,
> the File instance in Class[mymodule::example] will not see the resource
> default this way, but that it would see it if the order of the 'include'
> statements were reversed (following the same resolution rules as for
> variables).
>
> That reversing the order in which the classes are evaluated could change
> which defaults are applied -- if indeed that's the case -- would be an
> unhappy consequence.
>

Yes, if the default applied per named scope and be visibile in inner
named scopes, then the application of defaults would need to be lazy, or
you would go crazy over the order of how classes where included.

Since the implementation is to only look in the scope the resource
creation takes place in, (then in an inherited scope, i.e. class
inherit) and then in the global scope) it would not matter if you
changed the order of the two includes.

Puppet's strictness for named scopes is very weak as they can be
introduced anywhere, and in any arbitrary order, and they can be
reopened and appended to. So, while it would be very useful to be able
to declare defaults for a named scope and its subscopes, it leads to
more problems than what it solves IMO.

Variable references are resolve like this:
* local scope (in the closure where the variable reference is)
* named scope (the named scope the variable, or closure is in)
* inherited scope - if a class inherits another, it looks in its named scope
* top / node scope

and, if a variable is qualified, it is always anchored in top scope,
thus a reference to x::y is always resolved as if it had been written
::x::y.

Defaults now (future parser 3.7) follow the same rules except there is
no distinction between local scope (inside a lambda) and inside a named
scope. Coming back to your example, a default inside a::b, does not
apply to things in a::b::c, nor a::b::d, etc.

> In any case, though, it looks like you concur with me that there are
> evaluation-order consequences attending these approaches.
>

Yes, and in different ways. The eager approach would mean that a default
setting on a line after a resource is created (even in the same body)
would not apply to that resource, only resources created after that line
(i.e. a "normal imperative logic style").

What I like about the Schmitt style (with my variation) is that it
allows an explicit use of name spaced defaults, but you are responsible
for both a) including the class that defines those defaults, and b)
stating that you want those values as the default for your created
resource(s).

[Erik Dalén]

Right, that seems good. Older versions of the concat module used resource defaults to accomplish this: https://github.com/puppetlabs/puppetlabs-concat/blob/0.2.0/manifests/fragment.pp#L24-L41

But that was probably for puppet 2.6.x compatibility reasons or something.

--
Erik Dalén