The 4x scope

[Henrik Lindberg]

Hi,
we are just started to get more concrete on how to implement things for
4x and breaking it down into actionable items. If you have looked in
Jira, there are currently 5 big issues in the epic "Biff the Catalog
Builder" [1] - which is the goal (a new, better performing catalog
builder (what is currently known as the "compiler") where we can fix
many known issues that today are just to hard to implement.

This time, I want to talk about the implementation of Scope, which is
part of "(PUP-1832) Implement the Puppet 4.0 Runtime" [2].

Currently scope has many responsibilities (too many):

* it is classic computer language scope (what is visible "here")
* for a class it also represents one aspect of "an instance of a class"
(the attributes of the class are variables in that scope).
* Inheritance is achieved by looking up and continuing the search for a
variable in another "scope".

Coming up with a new implementation is important to make scope perform
well. Thus it is important to know:

- write vs read ratio
- unqualified vs. qualified lookup (i.e. reading $a:.b::x from within
$a::b vs from other scopes)
- typical nesting levels of named scopes

We also have to decide if any of the relative name-space functionality
should remain (i.e. reference to x::y is relative to potentially a series of
other name spaces ("dynamic scoping"), or if it is always a global
reference when it is qualified.

The implementation idea we have in mind is that there is one global
scope where all "qualified variables" are found/can be resolved, and
that all other variables are in local scopes that nest. (Local scopes
include ephemeral scopes for match variables).

Given the numbers from measuring the read ratio, we (sort of already
know, but still need to measure) need a fast route from any scope to the
global - we know that a qualified variable is never resolved by any
local scope so we can go straight to the global scope. (This way
we do not have to traverse the chain up to the "parent most" scope (the
global one). Local scopes are always local, there is no way to address
the local variables from some other non-nested scope - essentially how
the regular CPU stack works, or how variables in a language like C work).

i.e. we have something like this in Scope

Scope
attr_reader :global_scope
attr_reader :parent_scope
# ...
end


The global scope keeps an index designed to be as fast as possible to
resolve a qualified name to a value. The design of this index depends on
the frequency of different types of lookup. If all qualified lookups are
absolute it would simply be a hash of all absolute names to values (it
really cannot be faster than that).

The logic for lookup then becomes:
- for un-qualified name, search up the parent chain (this chain does not
reach the global scope), if still unresolved, look in global scope.
- for qualified name, look up in global scope directly

If we need to also consider relative namespaces (i.e. x::y could mean
z::x::y, or a::b::c::x::y etc. we can then either probe in turn with
each name (which is fine if the number of things to probe is low), or
provide a reverse index where y is first looked up to get the next level
of names, etc. (the idea being that this requires fewer operations to
find the right one).

IF we can completely remove the notion of relative namespacing we gain
performance!

The global scope, in addition to having the qualified names also needs
to separate the names by "kind" since we can have the same name for
different "kinds". We can now keep keep all named things in the global
scope - functions, types, variables, etc. Global scope and loading are
associated (more about loading in a later post) but it is worth noting
that it may be of value to be able to record that there has already been
an attempt of loading a particular name, and that there was nothing
there to load...

We are going to need the following kinds of scopes:

* Global Scope - holding map from kind, to fully qualified name to value
* Local Scope - holding variables that shadow parent scope
* Ephemeral / Match Scope (read only) - when a match is made
* Class Scope - the topmost scope for a class - needed because variable
lookup in it, and its nested scope needs to lookup all class attributes
(and defined them) via reading/setting variables.
* Resource Scope - the topmost scope for a user defined resource type -
needed because its parameters are available as read only variables.

The resource scope simply makes the resource parameters available. It
behaves as a local scope otherwise.

The class scope looks up unqualified variables in the class itself, if
not found there, it continues up the parent chain of scopes. If the
class inherits from another, then, the parent scope is one that
represents its super class.

In class scope, setting a variable also means that it is set in global
scope with the fully qualified name. This is where the logic around
class private variables comes in. If it is private, it cannot be
accessed from the outside (i.e. with a qualified name), and thus it
is only set in the class / class-scope. This in turn brings up the issue
of also supporting "protected" variables; only visible from within the
class logic, and the logic in sub classes, and if subclasses should see
private inherited variables or not (probably not).

The above could probably do with some picture :-)

Now, some questions...

- Are there any particular performance concerns you think we need to be
aware of?
- Do you have concerns about things we missed? Something important scope
needs to do?
- Do you have metrics from your environment? (number of lookups of
various kinds, etc)
- What is your reaction to getting rid of dynamic/relative name
resolution? (Breakage vs. sanity...)

Regards
- henrik

Links
---

[1]: https://tickets.puppetlabs.com/browse/PUP-1789
[2]: https://tickets.puppetlabs.com/browse/PUP-1832

[Trevor Vaughan]

Henrik,

All of this looks great to me. However, I was asked by someone recently if the language had the concept of a private class scope.

We're seeing more patterns in the wild where people are creating classes that are only meant to be used internally to the class and not exposed to the rest of the world.

Is there some way that the new scoping system could account for private classes?

The best we could come up with right now is the idea of having a 'private' directory just to make it clear that they are not meant for public consumption but a 'private' keyword would be great so that the language itself could enforce the restriction.

Thanks, and looking forward to the performance gains (but a bit worried about my custom types that use cross-resource variables).

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/lfthtr%24vnh%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Eric Sorenson]

Hi Trevor, this is tracked under PUP-523, please feel free to chime in on that ticket if you have additional use cases/requirements it doesn’t address.

https://tickets.puppetlabs.com/browse/PUP-523


—eric0

> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/lfthtr%24vnh%241%40ger.gmane.org.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699
> tvau...@onyxpoint.com
>
> -- This account not approved for unencrypted proprietary information --
>

> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoXKr6-okx%2B_VoyLEpf_-MkxdVwJ4vw7XJRPzV860V3ueQ%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.

Eric Sorenson - eric.s...@puppetlabs.com - freenode #puppet: eric0
puppet platform // coffee // techno // bicycles

[Trevor Vaughan]

Got it. Just thought of it while reading the e-mail. I'll head over there tomorrow to get it in.

Thanks,

Trevor

On Thu, Mar 13, 2014 at 10:01 PM, Eric Sorenson <eric.s...@puppetlabs.com> wrote:

Hi Trevor, this is tracked under PUP-523, please feel free to chime in on that ticket if you have additional use cases/requirements it doesn't address.

https://tickets.puppetlabs.com/browse/PUP-523

--eric0

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/81045344-9198-4167-B6D6-F1E712A35886%40puppetlabs.com.

For more options, visit https://groups.google.com/d/optout.

[Henrik Lindberg]

On 2014-14-03 1:59, Trevor Vaughan wrote:
> Henrik,
>
> All of this looks great to me. However, I was asked by someone recently
> if the language had the concept of a private class scope.
>

Not yet.

> We're seeing more patterns in the wild where people are creating classes
> that are only meant to be used internally to the class and not exposed
> to the rest of the world.
>

The idea is to allow classes and defines in 4x to be private to a
module. That is, while they are visible in certain ways, it will not be
possible to instantiate them (i.e. create/include them in the catalog)
from outside of the module. (There may be other aspects of private that
we may want to add such as also not being able to read parameters of
such instances once they are placed in the catalog).

Further, the idea is to also support private variables in classes - this
means that these variables can only be read from within the same class.
We may also add protected variables (only visible inside the class and a
class that inherits from it) - subject to debate and design, we are
trying to keep it as simple as possible...

> Is there some way that the new scoping system could account for private
> classes?
>

Yes, that is the idea. (Technically being "private" also has an effect
on other parts of the system; modules, loaders).

> The best we could come up with right now is the idea of having a
> 'private' directory just to make it clear that they are not meant for
> public consumption but a 'private' keyword would be great so that the
> language itself could enforce the restriction.
>

There will be a private keyword that can be used for classes, user
defined types, and functions as well as for class variables (other
variables are by definition already private).

Likewise, we may also add the notion of private for resource types (i.e.
plugins in Ruby) and their attributes.

We have some design work left to do on that (after all an agent needs to
be able to operate on them...

Meanwhile, I think you can check calling module in your logic and call
fail if it is not from the same module. Not sure how well that works though.

> Thanks, and looking forward to the performance gains (but a bit worried
> about my custom types that use cross-resource variables).
>

The idea is that qualified names (anything with :: in them) are always
"external references", and they can only see what is visible to general
code. Intersted to hear more about our "cross-resource variables" and
how you think that may break - do you rely on relative name resolution?
(if so, prepare by always using global references that start with ::
when you are referring to something that is neither in the local scope,
nor inherited)

Regards
- henrik

[Trevor Vaughan]

So, it sounds like we're getting everything we wanted! That's great to hear.

So, I suppose that I should have been clearer. We are starting to label all variables and class calls with $::name to ensure that the full removal of dynamic scoping works properly. While more verbose, I find it to be much more clear and easier to maintain overall.

What I'm actually doing is to use class variables in my providers so that I can bundle up all of my resource applications until the last one has passed through the catalog. This explains it with working examples https://www.onyxpoint.com/storing-puppet-provider-metadata-for-single-instance-application/.

It appears to be one of the more popular (of the limited) posts on my site right now and others that I've spoken with have expressed interest in doing this type of thing so anything that makes it easier/cleaner would be most welcome.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/lftopi%24h9m%241%40ger.gmane.org.

For more options, visit https://groups.google.com/d/optout.

[Henrik Lindberg]

On 2014-14-03 3:25, Trevor Vaughan wrote:
> So, it sounds like we're getting everything we wanted! That's great to hear.
>
> So, I suppose that I should have been clearer. We are starting to label
> all variables and class calls with $::name to ensure that the full
> removal of dynamic scoping works properly. While more verbose, I find it
> to be much more clear and easier to maintain overall.
>

Glad to hear that.

> What I'm actually doing is to use class variables in my providers so
> that I can bundle up all of my resource applications until the last one
> has passed through the catalog. This explains it with working examples
> https://www.onyxpoint.com/storing-puppet-provider-metadata-for-single-instance-application/.
>

This works fine as long as there is nothing else that depends on the
result (since it only appears at the very end).

> It appears to be one of the more popular (of the limited) posts on my
> site right now and others that I've spoken with have expressed interest
> in doing this type of thing so anything that makes it easier/cleaner
> would be most welcome.
>
> Thanks,
>
> Trevor
>

Interesting, it is an orchestration of sorts of all the resources
managed by the type / provider. The orchestration itself is simple;
simply do something with all of them at the end (on flush).

While we have ideas for this kind of fine grained orchestration, this is
not slated for inclusion in puppet until later in the 4x series. I don't
yet have anything written up - there is simply a ton of other things to
do first...

However, jumping ahead - the idea is roughly that providers have access
to a state machine and may queue up things to do for later, and that it
may be doing things in parallel, have shared steps (like in your example
to read the file), etc. Will come back to this topic later...

Meanwhile, thanks for sharing the example - it illustrates the need for
fine grained orchestration well - a Resource is too heavy, and there is
no supported way to break it up into smaller Resource-ettes/tasks. While
you could write out each line to the file system (and use it to maintain
state), and then later have another Resource type pick them up and apply
them to the target file, that quickly snowballs into a mess of resource
dependencies, overhead in logging and what not...

Regards
- henrik

[Erik Dalén]

Not terribly important, but are there any thoughts on making it so you can order variables any way you like in the manifest? They are immutable anyway, so these two examples should be equivalent:

$a=1

$b=$a+2

or

$b=$a+2

$a=1

You can put resources in any order you like, so it would make some sense to be able to do the same with variables. I guess it could pose some difficulties with functions reading the scope though, but after moving to epp templates there shouldn't be many of those kinds of functions.

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/lfthtr%24vnh%241%40ger.gmane.org.

For more options, visit https://groups.google.com/d/optout.

--
Erik Dalén

[Andy Parker]

On Thu, Mar 13, 2014 at 5:59 PM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

Henrik,

All of this looks great to me. However, I was asked by someone recently if the language had the concept of a private class scope.

Ask and ye shall find: https://tickets.puppetlabs.com/browse/PUP-523

We are planning on reserving the private keyword in puppet 4 and implementing private variables and classes in the puppet 4 series.

 

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoXKr6-okx%2B_VoyLEpf_-MkxdVwJ4vw7XJRPzV860V3ueQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

Join us at PuppetConf 2014, September 23-24 in San Francisco - http://bit.ly/pupconf14

[Henrik Lindberg]

On 2014-14-03 11:48, Erik Dalén wrote:
> Not terribly important, but are there any thoughts on making it so you
> can order variables any way you like in the manifest? They are immutable
> anyway, so these two examples should be equivalent:
>
> $a=1
> $b=$a+2
>
> or
>
> $b=$a+2
> $a=1
>
> You can put resources in any order you like, so it would make some sense
> to be able to do the same with variables. I guess it could pose some
> difficulties with functions reading the scope though, but after moving
> to epp templates there shouldn't be many of those kinds of functions.
>

Yes, there has been thoughts about this. It is kind of difficult though...

$a = [1,2,3]
$b = $a.map |$x| { $c + $x }
$c = $b[1]

and the Journey to Prolog begins... (or to a Spreadsheet like language...)

While it is true that resources can be placed in any order, it is
important to remember that there are two distinct phases - one that
computes the catalog, and one that applies it. The order among resources
matter if you want to be able to get an attribute set in one resource
while defining another (currently difficult without Ruby function), but
this is available in the future evaluator - e.g.

$mode_of_foo_file = File['foo'][mode]

Currently, we have no guarantees that a function call is side effect
free - it may create new classes, resources and variables.

I am also not sure if making the language completely lazy makes it any
easier for users with the different kind of corner cases (when to stop
iterating if there are loops in the logic, how to understand what is
wrong in those cases, etc.)

What I think we are opening up for in the 4x series is a far better
Catalog model (the end result) thus opening up for interesting language
experiments - say a Clojure or Prolog (on JVM) language that builds a
catalog. If such an implementation turns out to be fantastic and easy to
use / understand, it may take over the world...

Regards
- henrik

[Andy Parker]

On Thu, Mar 13, 2014 at 5:19 PM, Henrik Lindberg <henrik....@cloudsmith.com> wrote:

Hi,
we are just started to get more concrete on how to implement things for 4x and breaking it down into actionable items. If you have looked in Jira, there are currently 5 big issues in the epic "Biff the Catalog Builder" [1] - which is the goal (a new, better performing catalog builder (what is currently known as the "compiler") where we can fix many known issues that today are just to hard to implement.

This time, I want to talk about the implementation of Scope, which is
part of "(PUP-1832) Implement the Puppet 4.0 Runtime" [2].

Currently scope has many responsibilities (too many):

* it is classic computer language scope (what is visible "here")
* for a class it also represents one aspect of "an instance of a class"
  (the attributes of the class are variables in that scope).
* Inheritance is achieved by looking up and continuing the search for a
  variable in another "scope".

Coming up with a new implementation is important to make scope perform well. Thus it is important to know:

- write vs read ratio
- unqualified vs. qualified lookup (i.e. reading $a:.b::x from within $a::b vs from other scopes)
- typical nesting levels of named scopes

We also have to decide if any of the relative name-space functionality should remain (i.e. reference to x::y is relative to potentially a series of
other name spaces ("dynamic scoping"), or if it is always a global reference when it is qualified.

Other than using the search() function (which I'm not sure it even works), do you have an example of this happening? I can't seem to cause it to occur. If it doesn't really exist, then I'm not too concerned about dropping it :). I know that the relative name lookup happens when finding classes (include(b) can refer to a::b), but haven't been able to get it for variables yet.

 

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/lfthtr%24vnh%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

--

[Andy Parker]

On Fri, Mar 14, 2014 at 9:24 AM, Andy Parker <an...@puppetlabs.com> wrote:

On Thu, Mar 13, 2014 at 5:19 PM, Henrik Lindberg <henrik....@cloudsmith.com> wrote:

Hi,
we are just started to get more concrete on how to implement things for 4x and breaking it down into actionable items. If you have looked in Jira, there are currently 5 big issues in the epic "Biff the Catalog Builder" [1] - which is the goal (a new, better performing catalog builder (what is currently known as the "compiler") where we can fix many known issues that today are just to hard to implement.

This time, I want to talk about the implementation of Scope, which is
part of "(PUP-1832) Implement the Puppet 4.0 Runtime" [2].

Currently scope has many responsibilities (too many):

* it is classic computer language scope (what is visible "here")
* for a class it also represents one aspect of "an instance of a class"
  (the attributes of the class are variables in that scope).
* Inheritance is achieved by looking up and continuing the search for a
  variable in another "scope".

Coming up with a new implementation is important to make scope perform well. Thus it is important to know:

- write vs read ratio
- unqualified vs. qualified lookup (i.e. reading $a:.b::x from within $a::b vs from other scopes)
- typical nesting levels of named scopes

We also have to decide if any of the relative name-space functionality should remain (i.e. reference to x::y is relative to potentially a series of
other name spaces ("dynamic scoping"), or if it is always a global reference when it is qualified.

Other than using the search() function (which I'm not sure it even works), do you have an example of this happening? I can't seem to cause it to occur. If it doesn't really exist, then I'm not too concerned about dropping it :). I know that the relative name lookup happens when finding classes (include(b) can refer to a::b), but haven't been able to get it for variables yet.

Figured out a case of this happening:

  1 class a {

  2   include b

  3 }

  4

  5 class a::b {

  6   include c

  7   notice($c::x)

  8 }

  9

 10 class a::b::c {

 11   $x = 1

 12 }

 13

 14 include a

Line 7 ends up looking up the class c, which is looked up relative to a::b and so it finds a::b::c and then it performs the $x lookup in that class.

A consequence of this, which I think we should get rid of, is that anything visible from a::b::c is able to be looked up in this fashion. So another way of getting the kernel fact from a::b is to use $c::kernel (or in the new fact hash that is introduced in 3.5.0 $c::facts["kernel"]).

[David Schmitt]

On 2014-03-14 01:19, Henrik Lindberg wrote:
> - What is your reaction to getting rid of dynamic/relative name
> resolution? (Breakage vs. sanity...)

I totally support that $x can only be a local variable and $foo::x can
only be $::foo::x.

I've been bitten a few times by having a "local" class name shadow a
more global one. E.g: Given there is a "bar" class from the module with
the same name. When adding a local modificator class "foo::bar" in the
"foo" module, the name "bar" suddenly has change meaning throughout
"foo" without any way to notice, except for super-humanly code review
and weird breakage.


I sometimes use very local defines like this:

class foo (...) {
define helper(...) {
# ...
}

helper { $ary: foo => bar }
}

I guess those could still fall under "local" references as long as they
are defined within the braces of the surrounding class. Perhaps external
reference to those should be forbidden anyways ;-)



Regards, David

[David Schmitt]

On 2014-03-14 17:00, Henrik Lindberg wrote:
> On 2014-14-03 11:48, Erik Dalén wrote:
>> Not terribly important, but are there any thoughts on making it so you
>> can order variables any way you like in the manifest? They are immutable
>> anyway, so these two examples should be equivalent:

> I am also not sure if making the language completely lazy makes it any
> easier for users with the different kind of corner cases (when to stop
> iterating if there are loops in the logic, how to understand what is
> wrong in those cases, etc.)

I can remember chatting with Markus in Ghent about the Futures parser
where the compiler would have become completely lazy. My feeling is that
to be useful no cycles would have to be allowed, thus making evaluation
straightforward (computationally).


Regards, David

[David Schmitt]

Yeah, that's what I was talking about in my other mail. Under certain
patterns this can become quite common and quite painful, both in
debugging and how the resolution looks like.


Regards, David

[John Bollinger]

On Thursday, March 13, 2014 7:19:22 PM UTC-5, henrik lindberg wrote:

We also have to decide if any of the relative name-space functionality
should remain (i.e. reference to x::y is relative to potentially a series of
other name spaces ("dynamic scoping"), or if it is always a global
reference when it is qualified.

Can you choose a different term than "dynamic scoping" for what you're describing there?  It's not consistent with other uses of that term with which I am familiar, and historically "dynamic scoping" has meant something different in Puppet.

 

The implementation idea we have in mind is that there is one global
scope where all "qualified variables" are found/can be resolved, and
that all other variables are in local scopes that nest. (Local scopes
include ephemeral scopes for match variables).

Given the numbers from measuring the read ratio, we (sort of already
know, but still need to measure) need a fast route from any scope to the
global - we know that a qualified variable is never resolved by any
local scope so we can go straight to the global scope. (This way
we do not have to traverse the chain up to the "parent most" scope (the
global one).

I think that's fine, as long as it's consistent, but it has the potential to present oddities.  For example, in the body of class m, can one declare class ::m::a via its unqualified name (e.g. "include 'a'")?  If so, then should one not also from the same scope be able to refer to the variables of ::m::a via relative names ($a::foo)?

I see two main reasonable alternatives:

• class names are always treated as absolute.  Class ::m can declare class ::m::a only via its qualified name, and $a::foo is always equivalent to $::a::foo.
• class names can be expressed absolutely or relative to the innermost enclosing class scope (~ the current namespace), only, both for class declaration and for variable lookup.  Class ::m can declare class ::m::a via its unqualified name, and can refer to the variables of ::m::a via relative names ($a::var).
  

Either approach provides consistency in that any way it is permissible to refer to a class itself, it is also permissible to refer to that class's variables by appending '::varname'.  Note that the latter does not require traversing the chain of enclosing scopes, nor looking up names directly in any local scope; rather, it could be implemented as maximum two lookups against the global scope.

I expect that we will retain the ability to refer to variables via their unqualified names within some nest of scopes related to where they are declared (e.g. up to the innermost named (class or resource) scope).  Given, then, that that form of relative name lookup will be supported, I think generalizing that to classes and resources as well (second alternative) bears serious consideration.

On the other hand, those who have commented in the past seem to agree that Puppet's historic behavior of traversing the full chain of nested scopes, trying to resolve relative names with respect to each, is more surprising than useful.  I'm on board with that; I'm just suggesting that there may be both room and use for a more limited form of relative naming.

Local scopes are always local, there is no way to address
the local variables from some other non-nested scope - essentially how
the regular CPU stack works, or how variables in a language like C work).

i.e. we have something like this in Scope

Scope
   attr_reader :global_scope
   attr_reader :parent_scope
   # ...
end


The global scope keeps an index designed to be as fast as possible to
resolve a qualified name to a value. The design of this index depends on
the frequency of different types of lookup. If all qualified lookups are
absolute it would simply be a hash of all absolute names to values (it
really cannot be faster than that).

The logic for lookup then becomes:
- for un-qualified name, search up the parent chain (this chain does not
reach the global scope), if still unresolved, look in global scope.

From the description alone, I'm not sure how it can be asserted that the chain of local scopes does not reach global scope, unless by the the trivial fact that the global scope is not itself a local scope.  What I would hope to see, and perhaps what is meant, is that the lookup stops at local scopes that correspond to classes and resources.  In particular, I think it is essential that unqualified class name lookups not be resolved against parent namespaces.

That is, in class ::m::a::b, "include 'foo'" must not refer to ::m::a::foo, and certainly not to ::m::foo, but I'd be ok if it could refer to ::m::a::b::foo.  As a special (but important) case, in ::m::a::b, "include 'b'" must not refer to ::m::a::b itself, and "include 'a'" should not refer to ::m::a.

I'm going to try to digest some more of this over the weekend.  Perhaps I'll have more to say on Monday.  Such as about scoping function names so that different environments can bind different implementations to the same name, maybe.


John

[Henrik Lindberg]

I have a long and rambling response written in several installments
between meetings - so I apologize if it is not completely consistent...

To summarize the proposal for 4x (starting out with the cleanest most
strict to see what this means in practice):

1. unqualified variable references is "what can be seen in this
evaluation scope, outer evaluation scopes, and then global"
2. qualified variable references are absolute
3. definition of classes and defines with relative names are named
relative to the namespace it is in
4. resolution of qualified names used as references (for include etc)
are absolute

The rationale for 2, 3, 4 is that this is faster, most user seem to not
trust relative name resolution and throw in :: everywhere (for both
sanity and speed). For those that actually understand how it currently
works and actually wants relative name resolution it does mean a bit
more characters to type with a small loss in ease of refactoring (moving
a class means it can resolve against other classes than before - both a
blessing and a curse).

We are contemplating having an alias feature.

I then ramble on about scope and try to answer questions...

Regards
- henrik

On 2014-14-03 23:07, John Bollinger wrote:
>
>
> On Thursday, March 13, 2014 7:19:22 PM UTC-5, henrik lindberg wrote:
>
>
> We also have to decide if any of the relative name-space functionality
> should remain (i.e. reference to x::y is relative to potentially a
> series of
> other name spaces ("dynamic scoping"), or if it is always a global
> reference when it is qualified.
>
>
>
> Can you choose a different term than "dynamic scoping" for what you're
> describing there? It's not consistent with other uses of that term with
> which I am familiar, and historically "dynamic scoping" has meant
> something different in Puppet.
>

Not quite sure what the correct term is in current Puppet - I mean "the
various ways current puppet resolves a name such as x::y".

> The implementation idea we have in mind is that there is one global
> scope where all "qualified variables" are found/can be resolved, and
> that all other variables are in local scopes that nest. (Local scopes
> include ephemeral scopes for match variables).
>
> Given the numbers from measuring the read ratio, we (sort of already
> know, but still need to measure) need a fast route from any scope to
> the
> global - we know that a qualified variable is never resolved by any
> local scope so we can go straight to the global scope. (This way
> we do not have to traverse the chain up to the "parent most" scope (the
> global one).
>
>
>
> I think that's fine, as long as it's consistent, but it has the
> potential to present oddities. For example, in the body of class m, can
> one declare class ::m::a via its unqualified name (e.g. "include 'a'")?
> If so, then should one not also from the same scope be able to refer to
> the variables of ::m::a via relative names ($a::foo)?
>

There are several concepts at play:
* the name given to a class or user defined resource type
* the loading of it
* referencing a variable

Class naming

Currently a class or define gets a name in the name space where it is
defined - its (possibly qualified name) is appended to the name where it
is defined. Thus:

class a {
class b {
}
class x::y {
}
}

Creates the three classes ::a, ::a::b, and ::a::x::y

This construct is exactly the same as if they were defined like this:

class a {
}
class a::b {
}
class a::x::y {
}

The fact that a class is defined inside of another does not give it any
special privileges (reading private content inside the class it is
defined in, etc.). This is a naming operation only.

Likewise, when a class is included, this inclusion is in some arbitrary
namespace and it currently searches for the relative name. The
suggestion is to not do this, and instead require a fully qualified
(absolute) name.

include a
include b::a
include a::x::y


Sidebar:
| (I use the term "define" to mean "defining what a named entity is" as
| opposed to the term "declare" which is a term that denotes a
| definition of the existence of something (typically having some given
| type) - e.g. "int c" in the C language, which declares c, but does
| not define it.
|
| (Just saying since the use of define / declare may confuse someone)

> I see two main reasonable alternatives:
>

> * class names are always treated as absolute. Class ::m can declare

> class ::m::a only via its qualified name, and $a::foo is always
> equivalent to $::a::foo.

If you mean that class ::m::a can be defined inside of class ::m, we do
that either by:

a) as today, class gets its (relative) name concatenated on to the
containing class' name, otherwise its absolute name.
b) the name is always absolute, a class a {} inside a class m {} gets
the fully qualified name ::a
c) enforce that they are always named with a starting :: to be able to
flag down all relative names.
d) forbid that a nested class is given an absolute name

Of these I prefer a) since it causes the least breakage and surprise.

(Side note, the idea that nesting classes should not be allowed has
been raised as well - to further break the illusion that they have
some privileged relation to each other - they are not "inner classes" as
in Java or anything like that, they are not protected/private in any way
- the are just named after where they are defined).

> * class names can be expressed absolutely or relative to the innermost

> enclosing class scope (~ the current namespace), only, both for
> class declaration and for variable lookup. Class ::m can declare
> class ::m::a via its unqualified name, and can refer to the
> variables of ::m::a via relative names ($a::var).
>

This is like a) from naming the class, but keeps relative resolution of
references. Maybe it is a really bad idea to remove this ability - but
it is what opens up the can of worms... is it also relative to the name
space of the super class? is it relative to any outer name space? to any
outer name space of an inherited class?

> Either approach provides consistency in that any way it is permissible
> to refer to a class itself, it is also permissible to refer to that
> class's variables by appending '::varname'. Note that the latter does
> not require traversing the chain of enclosing scopes, nor looking up
> names directly in any local scope; rather, it could be implemented as
> maximum two lookups against the global scope.
>

Well, it is not really possible to refer to a class with a variable
it does not evaluate to an instance of class, it may evaluate to a
variable in another namespace though... (this is also confusing)

What the proposed (strict) rules means:

class a {
class b {
$x = 1
}
class c inherits b {
$y = $x + 10
}
}

The resolution of $x will lookup the $x in local scope representing the
class a::c, fail, and then in its parent scope representing a::b, and
there find x.

If instead

class c inherits b {
$y = $b::x + 10
}

was used, it would immediately go to the global scope and resolve b::x
and find the value 10.

now, if we instead treats b::x as a relative reference to the name space
it is used in - then it may be a reference to:
* ::a::c::x
* ::a::b::x
* ::b::x

What if b also inherits? What if the namespaces are more deeply nested?

$x = 0
class aa {
$x = 1
class a {
$x = 2
class b {
}
class c inherits b {
$y = $x
$z = $b::x
}
}
}
class b {
$x = 4
}

What is $aa::a::c::y and $aa::a::c::z ?

In the proposal, the $y evaluates to 0 (there is no x in c, nor in b,
they do not see into aa, and can not see into aa::b). And $z evaluates
to :undef, since there is no x in b.

In 3.x the value of $aa::a::c::z becomes 0, since when it reaches class
aa::a::b and it does not have an x, then x resolves the global x. (Jīng!
<- Chinese Surprise).

With relative naming the search is done in this order

* aa::a::c::b::x
* aa::a::b::x
* aa::b::x
* b::x

(if we remove the 3x surprising behavior to resolve to global x when
there is no x in b by setting $x in b) and move the b class around
between the various namespaces it is possible to verify that it searches
in the order above.

We can do that in 4x as well (sans the Jīng!) if we come to the
conclusion that that would be the best. (i.e. worst case 4 hash lookups
for a 3 level nesting of names). We cannot really optimize this - the
names have to be tried in that given order.

Making it strict means that there is only one lookup, but the c class
would have to be written like this:

class c inherits b {
$y = $x
$z = $aa::a::b::x
}

if we insist on making a qualified reference to the x in b (a $x gets
the same result).

We could make the inherited class have special status - and thus resolve
against it - but not sure if it is worth doing this.

> I expect that we will retain the ability to refer to variables via their
> unqualified names within some nest of scopes related to where they are
> declared (e.g. up to the innermost named (class or resource) scope).
> Given, then, that that form of relative name lookup will be supported, I
> think generalizing that to classes and resources as well (second
> alternative) bears serious consideration.
>

There is also the ability to reference a class and access its attributes
via the Class type. This way, it is totally clear what the resolution
is, and what the names are relative to. e.g.

$b = Class[some::class::somewhere]
$b[x]

and if this is done in a class, and you don't want the $b to be visible

private $b = Class[...]

This way there is no guessing what a relative name may mean. (In essence
relative names are only (optionally) used when defining classes and user
defined resource types.

> On the other hand, those who have commented in the past seem to agree
> that Puppet's historic behavior of traversing the full chain of nested
> scopes, trying to resolve relative names with respect to each, is more
> surprising than useful. I'm on board with that; I'm just suggesting
> that there may be both room and use for a more limited form of relative
> naming.
>

I am struggling with the balance of being useful, not having to type too
much, and ease of refactoring with sanity and performance... this
discussion is very valuable.

I like the simplicity of "an unquailified variable = what I see here",
and "a qualified variable = an absolute reference".

Nested ("local") scopes only contains unqualified names, and an inner
scope shadows an outer scope (there are a few additional rules for
restricted names such as $trusted, and $facts which may not be shadowed
in any scope). Qualified names (for variables) can only be created in
classes and these are only the public attributes of those classes. No
local (shadowing) scope places this "global scope" as an outer scope.

$x = 10
class a {
$x = 20
$y = $x
$z = $::x
}

Here the variables $a::x == 20, $a::y == 20, and $z == 10

The $::x is not found in an outer scope of the scope used to evaluate
the logic inside of class a.

The local scopes dies when evaluation using that scope - eh. goes out of
scope. The persisted values are kept in the global-scope index (and in
the instantiated classes and created resources).

> That is, in class ::m::a::b, "include 'foo'" must not refer to
> ::m::a::foo, and certainly not to ::m::foo, but I'd be ok if it could
> refer to ::m::a::b::foo. As a special (but important) case, in
> ::m::a::b, "include 'b'" must not refer to ::m::a::b itself, and
> "include 'a'" should not refer to ::m::a.
>

I think (but is not 100% sure) that it would be best to have to qualify
the name - i.e.

include foo # is include ::foo
include x::y # is include ::x::y

Other languages have solved the same issue in different ways:

* Ruby is obviously very flexible in how it searches (it also makes it
slow), and sometimes (just like in Puppet) it is mysterious why it
works or not in some cases.
* Java uses an import to import a name which can then be used in short
form, nested classes can be relatively referenced.
* Some Java like (new) languages use an import/alias mechanism

If we go down that path, these name imports would appear at the start of
the file and apply to the content of that file - i.e. it is a help to
the *parser* to construct the correct code (there is no searching at
runtime). Now sadly, import is a function that is just deprecated in the
Puppet Programming Language and reintroducing it with a different
meaning would just be a cruel joke... if instead we want to be able to
alias names maybe we could use "alias"

alias apache = mystuff::better_apache::apache

To support an alias like that, the only reasonable thing a parser could
do is to replace every "apache" in every qualified name with the alias -
i.e. apache::foo becomes mystuff::better_apache::apache::foo

A powerful mechanism to reduce typing - but that are also tricky if we
support more than a first pass of alias replacements, multi segement
aliases etc. (A sane impl. could perhaps only perform the replacement of
the first segment, and that an alias cannot be qualified itself.
(An alias could also be set to ::)

I am not sure I want to see aliases like these in the language.
Sometimes a bit more typing is good for (esp. the future) you.

We have a problem with referencing a class directly with a variable
since we can do this

class a {
$b = { x = jing }
class b {
$x = 10
}
notice $b::x
notice $b
}

$b is not a reference to the class, but in $b::x it is (this is kind of
confusing).

Super

Yet another way of handling resolutions is to add a super (reserved)
namespace word, that resolves the superclass. It would function as an
(absolute) reference to the superclass and mean give me a variable as
the superclass sees it (given class is allowed to see it). e.g.

class c inherits b {
$z = $super::x
}

But I am not sure that throwing yet another object oriented term into
the non object oriented puppet casserole makes it any sweeter...

> I'm going to try to digest some more of this over the weekend. Perhaps
> I'll have more to say on Monday.

I can imagine having a hangout on this topic as well...

> Such as about scoping function names
> so that different environments can bind different implementations to the
> same name, maybe.
>

There will be support for scoping function names. i.e. you can call

mymodule::foo(x)

All such references are currently (albeit still at the idea state)
absolute names - no shadowing, and no "local functions" are planned.

Aliasing is being contemplated, which means it is possible to alias
certain functions.

alias foo = mymodule::foo

Which would make all calls to foo() go to mymodule::foo() in the
.pp file having that alias at the top.

[Daniele Sluijters]

Hi,

I like just about everything in the proposal but I have my doubts about this:

> The class scope looks up unqualified variables in the class itself, if 
> not found there, it continues up the parent chain of scopes. If the 
> class inherits from another, then, the parent scope is one that 
> represents its super class.

'Dynamic' scoping has always tripped up people in Puppet and caused interesting side-effects. Even though your implementation's behaviour is well defined I think it'll only add to the confusion because the implementation is different from what currently exists. I'm afraid that people will expect lookups to work one way but depending on the Puppet version that behaviour changes introducing the possibility for all kinds of subtle bugs. This might get very troublesome for community module maintainers too.

Personally I'm of the inclination that if a variable is not defined within the current scope you need to be explicit about where you want to get it from and just bring it in through the fully qualified name. If that sometimes means you need to explicitly pass a variable down the chain all the better because it immediately becomes obvious where it's coming from.

Sure experienced programmers or people who are more familiar with how the scoping model in Puppet works will be able to use this to their advantage but the code that's produced and potentially released might not be easy to understand for others.

[Henrik Lindberg]

On 2014-15-03 16:54, Daniele Sluijters wrote:
> Hi,
>
> I like just about everything in the proposal but I have my doubts about
> this:
>
> > The class scope looks up unqualified variables in the class itself, if
> > not found there, it continues up the parent chain of scopes. If the
> > class inherits from another, then, the parent scope is one that
> > represents its super class.
>
> 'Dynamic' scoping has always tripped up people in Puppet and caused
> interesting side-effects. Even though your implementation's behaviour is
> well defined I think it'll only add to the confusion because the
> implementation is different from what currently exists. I'm afraid that
> people will expect lookups to work one way but depending on the Puppet
> version that behaviour changes introducing the possibility for all kinds
> of subtle bugs. This might get very troublesome for community module
> maintainers too.
>

The proposal is not really "dynamic scoping", but yes it is different
than 3x - so maybe best to make rules even simpler (as you propose below).

> Personally I'm of the inclination that if a variable is not defined
> within the current scope you need to be explicit about where you want to
> get it from and just bring it in through the fully qualified name. If
> that sometimes means you need to explicitly pass a variable down the
> chain all the better because it immediately becomes obvious where it's
> coming from.
>

That makes it totally clear where things come from. The only thing that
would need to be different (from what is proposed) is if we want to add
protected attributes to classes (only visible to itself and subclasses).
That would complicate the index in the global scope somewhat, but can
probably be implemented to not have a performance impact (i.e. extra
index, only checked last if caller should have access).

> Sure experienced programmers or people who are more familiar with how
> the scoping model in Puppet works will be able to use this to their
> advantage but the code that's produced and potentially released might
> not be easy to understand for others.
>

Yeah, and how it works today is a mystery to most Puppet Users, even
experts are tripped up by some of the quirks.

It will interesting to see how the different ideas works in practice. We
have a goal to "not break every module" in the initial release of 4x,
and having the new simpler/stricter lookup rules may be too much
breakage to tolerate and we have to introduce a "legacy name resolution"
option or something like that to help migration. Whatever we do we need
to make sure that a well formed 4x (not using any other new 4x features)
also work on 3x.

- henrik

[David Schmitt]

On 2014-03-16 01:36, Henrik Lindberg wrote:
>> Sure experienced programmers or people who are more familiar with how
>> the scoping model in Puppet works will be able to use this to their
>> advantage but the code that's produced and potentially released might
>> not be easy to understand for others.
>>
>
> Yeah, and how it works today is a mystery to most Puppet Users, even
> experts are tripped up by some of the quirks.

Yes.

> It will interesting to see how the different ideas works in practice. We
> have a goal to "not break every module" in the initial release of 4x,
> and having the new simpler/stricter lookup rules may be too much
> breakage to tolerate and we have to introduce a "legacy name resolution"
> option or something like that to help migration. Whatever we do we need
> to make sure that a well formed 4x (not using any other new 4x features)
> also work on 3x.

In the worst case, you could add a special "option strict"[1] sigil, to
request strict lookup on a file-by-file basis for as long as "legacy"
lookup has to be supported.

Would probably be a nightmare to support by the implementation though.



[1]http://stackoverflow.com/q/2454552

[Erik Dalén]

Fair enough.

On a bit similar note though, will the new scope implementation be capable of closures (if you use something like https://github.com/dalen/puppet-defn to store lambdas in variables)?

 

--
Erik Dalén

[John Bollinger]

On Friday, March 14, 2014 10:48:31 PM UTC-5, henrik lindberg wrote:

 [I wrote:]

> Such as about scoping function names
> so that different environments can bind different implementations to the
> same name, maybe.
>

There will be support for scoping function names. i.e. you can call

mymodule::foo(x)

All such references are currently (albeit still at the idea state)
absolute names - no shadowing, and no "local functions" are planned.

Aliasing is being contemplated, which means it is possible to alias
certain functions.

alias foo = mymodule::foo

Which would make all calls to foo() go to mymodule::foo() in the
.pp file having that alias at the top.

I love the idea of qualified function name, but that's not what I was talking about.  Puppet currently suffers from a different kind of scoping problem in that different versions of the same function, declared with the same name (e.g. by different versions of the same module) may exist in different environments, but Puppet can only use one of them -- whichever happens to get loaded first, IIRC.  Qualified function names do not solve this problem.


John

[James Sweeny]

On Mon, Mar 17, 2014 at 9:19 AM, John Bollinger <john.bo...@stjude.org> wrote:

I love the idea of qualified function name, but that's not what I was talking about.  Puppet currently suffers from a different kind of scoping problem in that different versions of the same function, declared with the same name (e.g. by different versions of the same module) may exist in different environments, but Puppet can only use one of them -- whichever happens to get loaded first, IIRC.  Qualified function names do not solve this problem.


John

https://jira.puppetlabs.com/browse/PUP-731 ?

--

James Sweeny
Professional Services

http://puppetlabs.com/

[John Bollinger]

On Friday, March 14, 2014 10:48:31 PM UTC-5, henrik lindberg wrote:

I have a long and rambling response written in several installments
between meetings - so I apologize if it is not completely consistent...

To summarize the proposal for 4x (starting out with the cleanest most
strict to see what this means in practice):

1. unqualified variable references is "what can be seen in this
evaluation scope, outer evaluation scopes, and then global"
2. qualified variable references are absolute
3. definition of classes and defines with relative names are named
relative to the namespace it is in
4. resolution of qualified names used as references (for include etc)
are absolute

As far as I can tell, (1) and (3) are the same behavior as 3x.  I don't think there is any argument about those.  Certainly I'm focused on (2) and (4).

 

The rationale for 2, 3, 4 is that this is faster, most user seem to not
trust relative name resolution and throw in :: everywhere (for both
sanity and speed). For those that actually understand how it currently
works and actually wants relative name resolution it does mean a bit
more characters to type with a small loss in ease of refactoring (moving
a class means it can resolve against other classes than before - both a
blessing and a curse).

We are contemplating having an alias feature.

I then ramble on about scope and try to answer questions...

Regards
- henrik

On 2014-14-03 23:07, John Bollinger wrote:
>
>
> On Thursday, March 13, 2014 7:19:22 PM UTC-5, henrik lindberg wrote:
>
>
>     We also have to decide if any of the relative name-space functionality
>     should remain (i.e. reference to x::y is relative to potentially a
>     series of
>     other name spaces ("dynamic scoping"), or if it is always a global
>     reference when it is qualified.
>
>
>
> Can you choose a different term than "dynamic scoping" for what you're
> describing there?  It's not consistent with other uses of that term with
> which I am familiar, and historically "dynamic scoping" has meant
> something different in Puppet.
>
Not quite sure what the correct term is in current Puppet - I mean "the
various ways current puppet resolves a name such as x::y".

I'm not aware of an existing term that applies, so maybe we can just call it "classic" scoping or "3x" scoping?

 > I think that's fine, as long as it's consistent, but it has the

> potential to present oddities.  For example, in the body of class m, can
> one declare class ::m::a via its unqualified name (e.g. "include 'a'")?
> If so, then should one not also from the same scope be able to refer to
> the variables of ::m::a via relative names ($a::foo)?
>
There are several concepts at play:
* the name given to a class or user defined resource type
* the loading of it
* referencing a variable

Class naming

Currently a class or define gets a name in the name space where it is
defined - its (possibly qualified name) is appended to the name where it
is defined. Thus:

class a {
   class b {
   }
   class x::y {
   }
}

Creates the three classes ::a, ::a::b, and ::a::x::y

This construct is exactly the same as if they were defined like this:

class a {
}
class a::b {
}
class a::x::y {
}

The fact that a class is defined inside of another does not give it any
special privileges (reading private content inside the class it is
defined in, etc.). This is a naming operation only.

Yes, I've got that already.

 

Likewise, when a class is included, this inclusion is in some arbitrary
namespace and it currently searches for the relative name. The
suggestion is to not do this, and instead require a fully qualified
(absolute) name.

include a
include b::a
include a::x::y

Yes, I'm tracking with you there.  I would argue for more precise wording, however: for the purpose of declaring classes, relative class names (all of the above being examples) are resolved only relative to the global namespace.

 

Sidebar:
| (I use the term "define" to mean "defining what a named entity is" as
| opposed to the term "declare" which is a term that denotes a
| definition of the existence of something (typically having some given
| type) - e.g. "int c" in the C language, which declares c, but does
| not define it.
|
| (Just saying since the use of define / declare may confuse someone)

We are speaking the same language.

 

> I see two main reasonable alternatives:
>
>   * class names are always treated as absolute.  Class ::m can declare
>     class ::m::a only via its qualified name, and $a::foo is always
>     equivalent to $::a::foo.
If you mean that class ::m::a can be defined inside of class ::m

No, I don't.  I mean the same thing by "declare" that you do.

 

(Side note, the idea that nesting classes should not be allowed has
been raised as well - to further break the illusion that they have
some privileged relation to each other - they are not "inner classes" as
in Java or anything like that, they are not protected/private in any way
- the are just named after where they are defined).

I would be fine with that.  If scoping will change, then that's an especial reason to also remove nested class and defined type definition, as those muddy the waters not only on OO-confusion grounds but also simply on lexical scoping grounds.

 

>   * class names can be expressed absolutely or relative to the innermost
>     enclosing class scope (~ the current namespace), only, both for
>     class declaration and for variable lookup.  Class ::m can declare
>     class ::m::a via its unqualified name, and can refer to the
>     variables of ::m::a via relative names ($a::var).
>
This is like a) from naming the class, but keeps relative resolution of
references. Maybe it is a really bad idea to remove this ability - but
it is what opens up the can of worms... is it also relative to the name
space of the super class? is it relative to any outer name space? to any
outer name space of an inherited class?

I return here to my thesis: if in some evaluation scope I can declare a given class via name N -- whether relative or absolute -- then in that same scope I should be able to refer to that class's public variables via the form $N::varname.

If relative names are resolved only vs the global namespace, then there is no problem.  I think that there is also no problem if relative names are resolved against the current namespace if possible and otherwise against the global namespace, but not against any intermediate namespaces.

It may also be that different rules need to apply to variable names than to class names.  In fact, it might be easiest to maintain the consistency I am requesting by characterizing variable lookups as composite operations consisting of (1) looking up the class name, and then (2) looking up the variable relative to that class.  Whether the variable lookup part of that should be able to pick up superclass variables is an open question, which might most easily be answered "no".

 

> Either approach provides consistency in that any way it is permissible
> to refer to a class itself, it is also permissible to refer to that
> class's variables by appending '::varname'.  Note that the latter does
> not require traversing the chain of enclosing scopes, nor looking up
> names directly in any local scope; rather, it could be implemented as
> maximum two lookups against the global scope.
>

Well, it is not really possible to refer to a class with a variable

I'm not talking about storing class references in variables, I'm talking about literal references in DSL code:

  include 'foo::bar'
  class { 'baz': }
  $v_bar = $foo::bar::v
  $v_baz = $baz::v

What the proposed (strict) rules means:

class a {
   class b {
     $x = 1
   }
   class c inherits b {
     $y = $x + 10
   }
}

The resolution of $x will lookup the $x in local scope representing the
class a::c, fail, and then in its parent scope representing a::b, and
there find x.

If instead

   class c inherits b {
     $y = $b::x + 10
   }

was used, it would immediately go to the global scope and resolve b::x
and find the value 10.

I was about to assert that both of those are how it works in classic scoping, too, but I see that that might not be exactly the case.  I do assert that classic scoping yields the same result, but I'm not sure whether it would do if there were a $a::x.  Under my proposal, the existence of an $a::x would not be relevant.

 

now, if we instead treats b::x as a relative reference to the name space
it is used in - then it may be a reference to:
* ::a::c::x
* ::a::b::x
* ::b::x

No, I don't quite see that.  If $b::x appears in the definition of class ::a::c (inherits ::a::b), then it can refer to $::b::x or it can refer to $::a::c::b::x, only.  Or it least that's my suggestion.

 

What if b also inherits? What if the namespaces are more deeply nested?

Namespace nesting depth is not relevant because my proposal is to consider only the current namespace and the global one.

Inheritance is a complicating factor, but a separate one, I think.  The question there is whether "$::b::x" means "the variable 'x' declared by class ::b" or whether it means "the variable to which '$x' refers in the evaluation scope of class ::b".  I think either one could work, but at the moment I'm favoring the former.  That question arises regardless of the question of support for any version relative class names.

 

$x = 0
class aa {
   $x = 1
   class a {
     $x = 2
     class b {
     }
     class c inherits b {
       $y = $x
       $z = $b::x
     }
   }
}
class b {
     $x = 4
}

What is $aa::a::c::y and $aa::a::c::z ?

In the proposal, the $y evaluates to 0 (there is no x in c, nor in b,
they do not see into aa, and can not see into aa::b).

Unless my eyes deceive me, you did not write a class ::aa::b, but rather a class ::b.  I will take it as your comments indicate you intended, rather than as written.  Since we agree that nesting class definitions is and should be significant to name resolution only inasmuch as it affects the qualified names of declared classes, it would probably be easier on everyone to avoid nesting in future examples.

In my proposal, $aa::a::c::y and $aa::a::c::z take the same values, for essentially the same reasons.  It is central to my proposal that intermediate namespaces between the global one and the current one not be consulted for relative names, unless as a result of class inheritance.  The difference would arise if there were also this class:

class aa::a::c::b {
  $x = 42
}

In that case, $aa::a::c::y would still take the value 0, but $aa::a::c::z would take the value 42 because within the definition of class aa::a::c variable name 'b::x' is looked up first as $::aa::a::c::b::x.  If that first lookup fails, then the only other one considered is $::b::x.

In 3.x the value of $aa::a::c::z becomes 0, since when it reaches class
aa::a::b and it does not have an x, then x resolves the global x. (Jīng!
<- Chinese Surprise).

With relative naming the search is done in this order

* aa::a::c::b::x
* aa::a::b::x
* aa::b::x
* b::x

(if we remove the 3x surprising behavior to resolve to global x when
there is no x in b by setting $x in b) and move the b class around
between the various namespaces it is possible to verify that it searches
in the order above.

Yes, I understand the 3x way, and I agree that it has surprising and undesirable results.  I am not advocating for it.

 

We can do that in 4x as well (sans the Jīng!) if we come to the
conclusion that that would be the best. (i.e. worst case 4 hash lookups
for a 3 level nesting of names). We cannot really optimize this - the
names have to be tried in that given order.

That's not quite what I am suggesting.  Mine is less surprising and more performant -- maximum 2 + (# of superclasses) lookups regardless of degree of namespace nesting.

 

Making it strict means that there is only one lookup, but the c class
would have to be written like this:

     class c inherits b {
       $y = $x
       $z = $aa::a::b::x
     }

if we insist on making a qualified reference to the x in b (a $x gets
the same result).

We could make the inherited class have special status - and thus resolve
against it - but not sure if it is worth doing this.

That's a significant question of what class inheritance means (or should mean).  I think a fine approach that wouldn't break too hard with 3x is that class inheritance adds ancestor classes' namespaces to those that are searched for unqualified variable names, between the current namespace and the global one.  Referring back to my partitioning of variable lookups into class lookups and local variable lookups, that amounts to using ancestor classes only for the variable lookup part, not for the class lookup part.

 

> I expect that we will retain the ability to refer to variables via their
> unqualified names within some nest of scopes related to where they are
> declared (e.g. up to the innermost named (class or resource) scope).
> Given, then, that that form of relative name lookup will be supported, I
> think generalizing that to classes and resources as well (second
> alternative) bears serious consideration.
>
There is also the ability to reference a class and access its attributes
via the Class type. This way, it is totally clear what the resolution
is, and what the names are relative to. e.g.

  $b = Class[some::class::somewhere]
  $b[x]

and if this is done in a class, and you don't want the $b to be visible

  private $b = Class[...]

This way there is no guessing what a relative name may mean. (In essence
relative names are only (optionally) used when defining classes and user
defined resource types.

Whereas that's pretty cool, it doesn't really address the problem of avoiding breaking modules as much as possible.  Relative naming is pervasive -- especially so if you consider that even most qualified names appearing in current Puppet manifests are still relative ("a::b" rather than "::a::b").

 

> On the other hand, those who have commented in the past seem to agree
> that Puppet's historic behavior of traversing the full chain of nested
> scopes, trying to resolve relative names with respect to each, is more
> surprising than useful.  I'm on board with that; I'm just suggesting
> that there may be both room and use for a more limited form of relative
> naming.
>
I am struggling with the balance of being useful, not having to type too
much, and ease of refactoring with sanity and performance... this
discussion is very valuable.

I like the simplicity of "an unquailified variable = what I see here",
and "a qualified variable = an absolute reference".

That does have a profound simplicity, but it's not very friendly to refactoring.  I am suggesting starting from a different axiom: the class name I can use here for one purpose, I can also use here for any other purpose.

Our two simplicities do not inherently conflict, but they clash if any form of relative class naming is recognized, whether my proposed narrow form or the broader 3x form.  Yet my sense is that relative class naming is fairly common in current code, mostly in forms that would be served by my proposal, and it is a great facilitator of refactoring (at least for people who undertake that task by hand, instead of via an automated tool).

>  From the description alone, I'm not sure how it can be asserted that

> the chain of local scopes does not reach global scope, unless by the the
> trivial fact that the global scope is not itself a local scope.  What I
> would hope to see, and perhaps what is meant, is that the lookup stops
> at local scopes that correspond to classes and resources.  In
> particular, I think it is essential that unqualified class name lookups
> not be resolved against parent namespaces.
>

Nested ("local") scopes only contains unqualified names, and an inner
scope shadows an outer scope (there are a few additional rules for
restricted names such as $trusted, and $facts which may not be shadowed
in any scope). Qualified names (for variables) can only be created in
classes and these are only the public attributes of those classes. No
local (shadowing) scope places this "global scope" as an outer scope.

Ok, got it.  Thanks for clearing that up.

 

> That is, in class ::m::a::b, "include 'foo'" must not refer to
> ::m::a::foo, and certainly not to ::m::foo, but I'd be ok if it could
> refer to ::m::a::b::foo.  As a special (but important) case, in
> ::m::a::b, "include 'b'" must not refer to ::m::a::b itself, and
> "include 'a'" should not refer to ::m::a.
>
I think (but is not 100% sure) that it would be best to have to qualify
the name - i.e.

   include foo    # is include ::foo
   include x::y   # is include ::x::y

Having to qualify the name is the current situation, but it is a relatively common error for people to fail to do so, writing, for example,

class site::ntp {
  # intended to refer to class ::ntp
  include 'ntp'
}

That error is resolved equally well, however, either by resolving all class name lookups only against the global namespace or by resolving them first against the current namespace and then against the global one (skipping intermediate ones, and possibly inherited ones).  In my estimation, the latter would break less code and would allow coding style that is more conducive to refactoring.

 

Other languages have solved the same issue in different ways:

* Ruby is obviously very flexible in how it searches (it also makes it
   slow), and sometimes (just like in Puppet) it is mysterious why it
   works or not in some cases.
* Java uses an import to import a name which can then be used in short
   form, nested classes can be relatively referenced.
* Some Java like (new) languages use an import/alias mechanism

I'm not seeing that the problem addressed by those mechanisms is necessarily one that has been raised here.  At least, I think those approaches have much broader scope than we have heretofore been discussing, and it is not immediately evident to me that Puppet would benefit much from such a mechanism relative to the simpler, implicit (but much narrower) mechanism I proposed.

If we go down that path, these name imports would appear at the start of
the file and apply to the content of that file - i.e. it is a help to
the *parser* to construct the correct code (there is no searching at
runtime). Now sadly, import is a function that is just deprecated in the
Puppet Programming Language and reintroducing it with a different
meaning would just be a cruel joke... if instead we want to be able to
alias names maybe we could use "alias"

alias apache = mystuff::better_apache::apache

To support an alias like that, the only reasonable thing a parser could
do is to replace every "apache" in every qualified name with the alias -
i.e. apache::foo becomes mystuff::better_apache::apache::foo

Well, I'd say replace every initial segment exactly matching "apache", but that's my fetish for precision talking.  Or is that in fact different from what you meant?

 

A powerful mechanism to reduce typing - but that are also tricky if we
support more than a first pass of alias replacements, multi segement
aliases etc. (A sane impl. could perhaps only perform the replacement of
the first segment, and that an alias cannot be qualified itself.
(An alias could also be set to ::)

I am not sure I want to see aliases like these in the language.
Sometimes a bit more typing is good for (esp. the future) you.

I am not sure I want to see such a thing, either.  Let's shelf that idea for now.

 

We have a problem with referencing a class directly with a variable
since we can do this

   class a {
     $b = { x = jing }
     class b {
       $x = 10
     }
     notice $b::x
     notice $b
   }

$b is not a reference to the class, but in $b::x it is (this is kind of
confusing).

You are right, that can be a bit confusing, but it's not related to scoping: the same confusion arises if there is a class ::b.  The rule for users to learn is simply that you look at the last segment of a variable reference for the actual variable name, and I don't think that's either hard or unintuitive.

 

Super

Yet another way of handling resolutions is to add a super (reserved)
namespace word, that resolves the superclass. It would function as an
(absolute) reference to the superclass and mean give me a variable as
the superclass sees it (given class is allowed to see it). e.g.

   class c inherits b {
     $z = $super::x
   }

But I am not sure that throwing yet another object oriented term into
the non object oriented puppet casserole makes it any sweeter...

Nor am I, especially since I'm having trouble thinking of many appropriate uses for such a construct in Puppet code.

John

[John Bollinger]

On Monday, March 17, 2014 8:36:25 AM UTC-5, James Sweeny wrote:

On Mon, Mar 17, 2014 at 9:19 AM, John Bollinger <john.bo...@stjude.org> wrote:

I love the idea of qualified function name, but that's not what I was talking about.  Puppet currently suffers from a different kind of scoping problem in that different versions of the same function, declared with the same name (e.g. by different versions of the same module) may exist in different environments, but Puppet can only use one of them -- whichever happens to get loaded first, IIRC.  Qualified function names do not solve this problem.


John

https://jira.puppetlabs.com/browse/PUP-731 ?

Yes, though the issue description seems to focus on types rather than functions.  The title mentions functions, and the issue is basically the same for them.


John

[Felix Frank]

On 03/17/2014 05:25 PM, John Bollinger wrote:
> https://jira.puppetlabs.com/browse/PUP-731

> <https://jira.puppetlabs.com/browse/PUP-731> ?
>
>
>
> Yes, though the issue description seems to focus on types rather than
> functions. The title mentions functions, and the issue is basically the
> same for them.

Yes. Note that the original RedMine ticket was referred to what is now
https://tickets.puppetlabs.com/browse/PUP-1033 and pertains to functions
only.

Regards,
Felix

[Jeff McCune]

  

 

  

 VM   

    

   

 

  

 

    

     

-Jeff

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/lg0ii1%24uo0%241%40ger.gmane.org.

[Henrik Lindberg]

On 2014-17-03 10:07, Erik Dalén wrote:
>
>
>
> On 14 March 2014 17:00, Henrik Lindberg <henrik....@cloudsmith.com

Yes, the new design makes that possible, but there are corner cases that
requires thought - what should happen if a closure is created in a class
and creates more resources after a class is evaluated etc.

We do not have a ticket logged to add support for closures, not yet at
least. I see this as something that can be defined during the 4x series
of releases as it is not fundamental for 4.0 itself (enough other things
to deal with first).

- henrik

[Henrik Lindberg]

The idea to solve that is to limit the visibility of modules that a
module does not depend on. This way, the same function (or different
versions of it) may be used from different modules.

And naturally, one environment may be different from other environments,
only sharing what is identical.

Currently, the implementation of Puppet::Parser::Function prohibits this
since it infects Scope. The new Function API and new loaders that we
plan to implement for 4.0 will fix this.

- henrik

[Henrik Lindberg]

Thanks John for taking the time to read and comment in detail -
I think I get what you are proposing, and I think that may work.

Have to read the entire thing through a couple of time more before
responding.

- henrik

> names (/all/ of the above being examples) are resolved only relative to

> consider only the current namespace and the global one..

> */not/* be consulted for relative names, unless as a result of class

> are searched for /unqualified/ /variable/ names, between the current

> Our two simplicities do not /inherently/ conflict, but they clash if any

> form of relative class naming is recognized, whether my proposed narrow
> form or the broader 3x form. Yet my sense is that relative class naming
> is fairly common in current code, mostly in forms that would be served
> by my proposal, and it is a great facilitator of refactoring (at least
> for people who undertake that task by hand, instead of via an automated
> tool).
>
>
> > From the description alone, I'm not sure how it can be asserted that
>
> > the chain of local scopes does not reach global scope, unless by
> the the
> > trivial fact that the global scope is not itself a local scope.
> What I
> > would hope to see, and perhaps what is meant, is that the lookup
> stops
> > at local scopes that correspond to classes and resources. In
> > particular, I think it is essential that unqualified class name
> lookups
> > not be resolved against parent namespaces.
> >
>
> Nested ("local") scopes only contains unqualified names, and an inner
> scope shadows an outer scope (there are a few additional rules for
> restricted names such as $trusted, and $facts which may not be shadowed
> in any scope). Qualified names (for variables) can only be created in
> classes and these are only the public attributes of those classes. No
> local (shadowing) scope places this "global scope" as an outer scope.
>
>
>

> Ok, got it. Thanks for clearing that up..

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-dev+...@googlegroups.com

> <mailto:puppet-dev+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/9ba450af-8181-4ad9-a118-5dde719137c8%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-dev/9ba450af-8181-4ad9-a118-5dde719137c8%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Erik Dalén]

On 14 March 2014 01:19, Henrik Lindberg <henrik....@cloudsmith.com> wrote:

Hi,
we are just started to get more concrete on how to implement things for 4x and breaking it down into actionable items. If you have looked in Jira, there are currently 5 big issues in the epic "Biff the Catalog Builder" [1] - which is the goal (a new, better performing catalog builder (what is currently known as the "compiler") where we can fix many known issues that today are just to hard to implement.

This time, I want to talk about the implementation of Scope, which is
part of "(PUP-1832) Implement the Puppet 4.0 Runtime" [2].

Currently scope has many responsibilities (too many):

* it is classic computer language scope (what is visible "here")
* for a class it also represents one aspect of "an instance of a class"
  (the attributes of the class are variables in that scope).
* Inheritance is achieved by looking up and continuing the search for a
  variable in another "scope".

Coming up with a new implementation is important to make scope perform well. Thus it is important to know:

- write vs read ratio
- unqualified vs. qualified lookup (i.e. reading $a:.b::x from within $a::b vs from other scopes)
- typical nesting levels of named scopes

We also have to decide if any of the relative name-space functionality should remain (i.e. reference to x::y is relative to potentially a series of
other name spaces ("dynamic scoping"), or if it is always a global reference when it is qualified.

The implementation idea we have in mind is that there is one global scope where all "qualified variables" are found/can be resolved, and that all other variables are in local scopes that nest. (Local scopes include ephemeral scopes for match variables).

Given the numbers from measuring the read ratio, we (sort of already know, but still need to measure) need a fast route from any scope to the global - we know that a qualified variable is never resolved by any local scope so we can go straight to the global scope. (This way

we do not have to traverse the chain up to the "parent most" scope (the global one). Local scopes are always local, there is no way to address the local variables from some other non-nested scope - essentially how the regular CPU stack works, or how variables in a language like C work).

i.e. we have something like this in Scope

Scope
  attr_reader :global_scope
  attr_reader :parent_scope
  # ...
end


The global scope keeps an index designed to be as fast as possible to resolve a qualified name to a value. The design of this index depends on the frequency of different types of lookup. If all qualified lookups are absolute it would simply be a hash of all absolute names to values (it really cannot be faster than that).

The logic for lookup then becomes:
- for un-qualified name, search up the parent chain (this chain does not reach the global scope), if still unresolved, look in global scope.

- for qualified name, look up in global scope directly

If we need to also consider relative namespaces (i.e. x::y could mean z::x::y, or a::b::c::x::y etc. we can then either probe in turn with each name (which is fine if the number of things to probe is low), or provide a reverse index where y is first looked up to get the next level of names, etc. (the idea being that this requires fewer operations to find the right one).

IF we can completely remove the notion of relative namespacing we gain performance!

The global scope, in addition to having the qualified names also needs to separate the names by "kind" since we can have the same name for different "kinds". We can now keep keep all named things in the global scope - functions, types, variables, etc. Global scope and loading are
associated (more about loading in a later post) but it is worth noting that it may be of value to be able to record that there has already been an attempt of loading a particular name, and that there was nothing there to load...

We are going to need the following kinds of scopes:

* Global Scope - holding map from kind, to fully qualified name to value
* Local Scope - holding variables that shadow parent scope
* Ephemeral / Match Scope (read only) - when a match is made
* Class Scope - the topmost scope for a class - needed because variable lookup in it, and its nested scope needs to lookup all class attributes (and defined them) via reading/setting variables.
* Resource Scope - the topmost scope for a user defined resource type - needed because its parameters are available as read only variables.

The resource scope simply makes the resource parameters available. It behaves as a local scope otherwise.

The class scope looks up unqualified variables in the class itself, if not found there, it continues up the parent chain of scopes. If the class inherits from another, then, the parent scope is one that represents its super class.

In class scope, setting a variable also means that it is set in global scope with the fully qualified name. This is where the logic around class private variables comes in. If it is private, it cannot be accessed from the outside (i.e. with a qualified name), and thus it
is only set in the class / class-scope. This in turn brings up the issue of also supporting "protected" variables; only visible from within the class logic, and the logic in sub classes, and if subclasses should see private inherited variables or not (probably not).

The above could probably do with some picture :-)

Now, some questions...

- Are there any particular performance concerns you think we need to be aware of?
- Do you have concerns about things we missed? Something important scope needs to do?

Also IMO it would be really good if support for doing the stuff in https://tickets.puppetlabs.com/browse/PUP-1985 could be added. Basing default values of parameters on the values of earlier parameters.

Atm you are forced to do the ugly kind of stuff that the style guide recommends with $real_foo variables (puppet <2.6.2 style):

http://puppet-lint.com/checks/class_parameter_defaults/

--
Erik Dalén

[Henrik Lindberg]

On 2014-19-03 17:26, Erik Dalén wrote:
>
> Also IMO it would be really good if support for doing the stuff in
> https://tickets.puppetlabs.com/browse/PUP-1985 could be added. Basing
> default values of parameters on the values of earlier parameters.
>
> Atm you are forced to do the ugly kind of stuff that the style guide
> recommends with $real_foo variables (puppet <2.6.2 style):
> http://puppet-lint.com/checks/class_parameter_defaults/
>

This is somewhat tricky, but we could possibly achieve this by
processing parameters from left to right even if the values given to it
are given in a Hash (which potentially is unordered).

We should also only accept variable references to parameters to the left
of where it is used, or using global variables, anything else is a
grey-zone (are they evaluated in the caller's scope? or some other scope?).

I updated PUP-1985 and brought it into the 3.6 work.

Regards
- henrik

[Erik Dalén]

Those seem like very reasonable limitations (unless we allow variables in any order everywhere ofc).
Any absolute references to variables in classes that have already been parsed should also be allowed.

> I updated PUP-1985 and brought it into the 3.6 work.

Great.
>
> Regards
> - henrik

>
>
>
> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/lgcj31%2422j%241%40ger.gmane.org.

[Henrik Lindberg]

On 2014-19-03 18:20, Erik Dalén wrote:
>
> On 19 Mar 2014 18:11, "Henrik Lindberg" <henrik....@cloudsmith.com

> <mailto:henrik....@cloudsmith.com>> wrote:
> >
> > On 2014-19-03 17:26, Erik Dalén wrote:
> >>
> >>
> >> Also IMO it would be really good if support for doing the stuff in
> >> https://tickets.puppetlabs.com/browse/PUP-1985 could be added. Basing
> >> default values of parameters on the values of earlier parameters.
> >>
> >> Atm you are forced to do the ugly kind of stuff that the style guide
> >> recommends with $real_foo variables (puppet <2.6.2 style):
> >> http://puppet-lint.com/checks/class_parameter_defaults/
> >>
> >
> > This is somewhat tricky, but we could possibly achieve this by
> processing parameters from left to right even if the values given to it
> are given in a Hash (which potentially is unordered).
> >
> > We should also only accept variable references to parameters to the
> left of where it is used, or using global variables, anything else is a
> grey-zone (are they evaluated in the caller's scope? or some other scope?).
> >
>
> Those seem like very reasonable limitations (unless we allow variables
> in any order everywhere ofc).
> Any absolute references to variables in classes that have already been
> parsed should also be allowed.
>

Yes, anything that is externally referenced. (i.e. evaluated and
available via a fully qualified name).

- henrik