Namespaces, part 2
Dan Sugalski
namespace entries get a post-pended null character) it's time for the
ops to handle them, as well as some extended semantics.
We already have find_global and store_global. These work fine, we're
going to keep them, though I think we're going to pare down the
semantics a bit.
What I want to keep is:
$Px = find_global 'name'
$Px = find_global [key; key; key], 'name'
store_global 'name', $Px
store_global [key; key; key], $Px
These load and store from the current namespace and the specified
namespace respectively. The case where the namespace is specified by
string I think should go away. (And this'll impact me quite a bit,
but it's still the right thing to do)
Next I want to add in the op variants:
$Px = find_global [key; key]
$Px = find_global $Px, [key; key]
$Px = find_global $Py, 'name'
to fetch out namespace PMCs and work with fetched namespace PMCs.
That is, if I have the namespace Foo, Foo::Bar, and Foo::Bar::Baz,
and then have a thing named xyzzy in Foo::Bar::Baz, I can get it by
doing:
$P1 = find_global ['Foo'; 'Bar'; 'Baz'], 'xyzzy'
or
$P0 = find_global ['Foo']
$P1 = find_global $P0, ['Bar'; 'Baz']
$P2 = find_global $P1, 'xyzzy'
or
$P0 = find_global ['Foo'; 'Bar'; 'Baz']
$P1 = find_global $P0, 'xyzzy'
That is, if we don't specify the name of the thing in the namespace
we get the namespace PMC itself, which we can then go look things up
in. This is handy since it means code can cache the namespace PMC so
if there are a dozen variables to go fetch out, we don't have to do
the multi-level hash lookup each time. (That'd be nasty)
Making one section of a namespace an alias for another part is just a
matter of getting the PMC for the section of the namespace tree you
want to alias and sticking it into the namespace it should hang off
of. Right *now* I'm not inclined to have a store_global variant to do
this, but if we want to completely hide all namespace operations
behind ops I'm OK with that.
Now, with that out of the way, let's talk about overlaid namespaces.
The namespace specification has always allowed for multiple
overlapping namespaces. What this is supposed to allow you do to is
have two or more full namespace trees available at once, with lookups
and stores automatically running through each of the namespaces until
the appropriate thing is found. (And the fact that we want to do
this, and do it sorta-kinda lexically, is a reason for the namespace
ops)
For this, we're going to add the ops:
overlay_namespace $Px
overlay_namespace $Px, $Iy
remove_namespace $Px
remove_namespace $Ix
The first two add the namespace $Px to the current search list -- the
first version makes it the first space that will be searched, the
second form puts it at $Ix in the list of namespaces to search.
The remove ops removes a namespace from the list. The first version
removes the namespace $Px (the same PMC you used to add in with
overlay_namespace) from the search list wherever it is, the second
form removes the namespace at offset $Ix.
Right now (and I expect this will change, but we're starting simple)
store_global will *always* store into the first namespace in the
list, while load_global will look through the namespaces in the list
until it finds one that has the thing being looked for, doing the
error pitching only if the thing isn't in any of the namespaces in
the list
So, basically, a sub doesn't have a single namespace tree -- it has a
list of trees, and we search that list for each lookup, and we store
into the first tree in the list on store.
Comments?
--
Dan
--------------------------------------it's like this-------------------
Dan Sugalski even samurai
d...@sidhe.org have teddy bears and even
teddy bears get drunk
Tim Bunce
> Okay, since we've got the *basic* semantics down (unified namespace,
> namespace entries get a post-pended null character) it's time for the
> ops to handle them, as well as some extended semantics.
I agree with Larry when he said "But I really don't care so much
what the flag is, as long as it's visible on printout. I think
even / would be better than \0."
But if you _really_ want to go with a null, Larry also had a point
when he said "Well, I'd prepend the null just to reduce confusion
(or rather, to force the confusion earlier)".
> What I want to keep is:
>
> $Px = find_global 'name'
> $Px = find_global [key; key; key], 'name'
> These load [...] from the current namespace and the specified
> namespace respectively.
How do you define "current"? Compile-time, load-time, run-time,
run-time for each sub? From what you say later I'm guessing
you mean "run-time for each sub". So "current namespace" is an
attribute of the sub?
If so, then how is that represented within the interpreter struct
when it enters an anonymous subroutine via a continuation?
I ask partly because I'm (still) not paying enough attention,
and because once you add...
> Next I want to add in the op variants:
>
> $Px = find_global [key; key]
> $Px = find_global $Px, [key; key]
> $Px = find_global $Py, 'name'
Then "$Px = find_global 'name'" could be changed by the compiler
(or byteloader?) into "$Px = find_global $Py, 'name'" where $Py is
the register, or "pseudo register", holding the "current" namespace.
> The case where the namespace is specified by
> string I think should go away. (And this'll impact me quite a bit,
> but it's still the right thing to do)
I agree.
> to fetch out namespace PMCs and work with fetched namespace PMCs.
> $P0 = find_global ['Foo']
> $P1 = find_global $P0, ['Bar'; 'Baz']
> $P2 = find_global $P1, 'xyzzy'
> That is, if we don't specify the name of the thing in the namespace
> we get the namespace PMC itself, which we can then go look things up
> in. This is handy since it means code can cache the namespace PMC so
> if there are a dozen variables to go fetch out, we don't have to do
> the multi-level hash lookup each time. (That'd be nasty)
Yeap.
> Right *now* I'm not inclined to have a store_global variant to do
> this, but if we want to completely hide all namespace operations
> behind ops I'm OK with that.
The issue here is the postpending of a null to indicate a namespace?
If the application does it then it's breaking the encapsulation of
that concept? Yeap, that's bad and should be avoided.
By the way, you said:
> store_global 'name', $Px
> store_global [key; key; key], $Px
but shouldn't that be:?
store_global 'name', $Px
store_global [key; key; key], 'name', $Px # 'name' was missing
Then a store_global variant without the name param would be how you'd
store a namespace:
store_global [key1; key2; key3], $Px
I think that's worth adding up front.
Other random thoughts:
I presume that store_global will auto-vivify namespace in the list.
You'll may need to define a way to explicitly get the root namespace.
I think some minimal top-level namespace policy needs to be set now
to limit the risk of confusion and/or polution later. I'd suggest:
* Every Language gets a top-level namespace named after the language.
So the Perl compiler adds 'Perl'; to all the namespace lookups
(or something equivalent to that in effect).
* Every Language namespace uses "ParrotRoot" as the way to access
the root namespace (so find_global ['Perl'; 'ParrotRoot'] returns
the root namespace). That way any number of languages can coexist
within Parrot at runtime with no risk of clashing. The only
"polution" is the single "ParrotRoot" name. The Python language
root namespace, for example, would be visible to all other languages
as ParrotRoot.Python.
> Now, with that out of the way, let's talk about overlaid namespaces.
I don't think I ever read a description of what the purpose of this was.
I get the "what" but not the "why". Without the "why" it's hard to
critique the "how".
> The namespace specification has always allowed for multiple
> overlapping namespaces. What this is supposed to allow you do to is
> have two or more full namespace trees available at once, with lookups
> and stores automatically running through each of the namespaces until
> the appropriate thing is found.
I understand that, but...
> (And the fact that we want to do this, and do it sorta-kinda lexically,
> is a reason for the namespace ops)
... not that. But let's ignore that for now. I'll come back to it below.
> For this, we're going to add the ops:
>
> overlay_namespace $Px
> overlay_namespace $Px, $Iy
>
> remove_namespace $Px
> remove_namespace $Ix
>
> The first two add the namespace $Px to the current search list
There's that "current" word again. Same confusion in my mind.
Unless "current" can be explained *really simply* I'd suggest it's
not used anywhere. All namespace ops that assume a "current"
namespace could be removed and the "currrent" namespace would then
become explicit: it would just be another register. Unless I'm
missing something I think the gain in clarity would be worth it.
> the first version makes it the first space that will be searched, the
> second form puts it at $Ix in the list of namespaces to search.
"puts" or "inserts"?
So a "namespace is like a hash" is now "a namespace is like an
ordered list of hashes"? Okay.
But if I do:
$P0 = find_global ['Foo']
overlay_namespace $P0
and the namespace in $P0 had already been overlayed, would we then
have a "list of lists of hashes"?
Or, are you saying that the overlay "magic" is related to the
vague (to me) concept of "current" and that's where the "sorta-kinda
lexically" comes in?
All seems rather complicated to me (but I could easily be missing
the point). I'm also concerned that while most of Parrot is pushing
logic into PMCs in order that it's encapsulated *and overridable*,
namespace behaviour is being hard-wired into ops.
Here's what I'd suggest:
A. Let all operations on namespaces be explicit about the namespace
they're operating on. Define a pseudo-register for current namespace.
B. Define a vtable interface for a Namespace PMC. Namespace PMC
should search themselves and insert entries into themselves.
C. Define a NamespaceList PMC that has the same interface as a
Namespace PMC but delegates to a list of one or more Namespace PMCs.
D. Instead of using an overlay_namespace op a sub would create/manipulate
a NamespaceList PMC. That NamespaceList PMC could be passed around
and act just like a Namespace PMC.
E. [extra thought] Namespace PMCs would auto-vivify namespaces below
themselves to be of the same type of PMC. Languages could install
language-specific PMCs at their root (which would then auto-vivify
down the tree) to support any language-specific behaviour that might
be useful - though still supporting Parrot mandated behaviour.
Feel free to shoot me down ([1] especially as I don't know what I'm
talking about when it comes to parrot internals like vtables).
> Right now (and I expect this will change, but we're starting simple)
> store_global will *always* store into the first namespace in the list
By using a NamespaceList PMC subs could implement whatever behaviour
they want.
Tim.
Brent 'Dax' Royal-Gordon
> > Now, with that out of the way, let's talk about overlaid namespaces.
>
> I don't think I ever read a description of what the purpose of this was.
> I get the "what" but not the "why". Without the "why" it's hard to
> critique the "how".
Clearly I'm not Dan, but I think the idea here is that, for example,
in the following code:
module Foo::Bar {
class Baz { $quux }
}
You can have "the current namespace" actually be [ ::Foo::Bar::Baz,
::Foo::Bar, ::* ] (or, for the last one, whatever the namespace that
@*ARGS and friends are in is called), so that the search for $quux can
be done very easily.
--
Brent 'Dax' Royal-Gordon <br...@brentdax.com>
Perl and Parrot hacker
There is no cabal.
[I currently have a couple Gmail invites--contact me if you're interested.]
Leopold Toetsch
> Next I want to add in the op variants:
> $Px = find_global [key; key]
> $Px = find_global $Px, [key; key]
> $Px = find_global $Py, 'name'
I've already proposed some time ago that these variants of namespace
manipulation aren't really necessary. I think that we would just need
one more opcode:
$Px = get_namespace
All other variants of accessing items, setting, overlaying or whatever
can be done then with keyed opcodes on that namespace PMC.
And given that we want to tie the namepspace, we need a distinct
namespace PMC type.
> So, basically, a sub doesn't have a single namespace tree -- it has a
> list of trees, and we search that list for each lookup, and we store
> into the first tree in the list on store.
Is that "list of trees" only the top-level node or can any node be a
list of trees?
leo
Jeff Clites
> Okay, since we've got the *basic* semantics down (unified namespace,
> namespace entries get a post-pended null character)
I'll ask again, what about subs? Do they get name-mangled too?
> $Px = find_global [key; key; key], 'name'
As Leo pointed out in a thread of the same name last year, this is a
new syntax--keyed access on nothing. I assume you mean for [key; key;
key] to serve as a sort of literal syntax for an array of strings? If
so, we should make the syntax for that sort of thing explicit. Or, we
can do what Leo suggested (in a thread of the same name last year, and
more recently), and write this as keyed access on a particular
namespace (typically, the root namespace):
$P0 = root_namespace
$P1 = find_global $P0['Foo'; 'Bar'; 'Baz'], 'xyzzy'
> The case where the namespace is specified by string I think should go
> away.
I assume by "specified by string", you mean a string such as
"Foo::Bar"? If so, I agree, and I'll note that it would be possible to
write per-language utility functions to parse these apart--compilers
will need to be able to process these, in some form, and so might
programmers in general, but it doesn't need to be an op.
> That is, if I have the namespace Foo, Foo::Bar, and Foo::Bar::Baz, and
> then have a thing named xyzzy in Foo::Bar::Baz, I can get it by doing:
>
> $P1 = find_global ['Foo'; 'Bar'; 'Baz'], 'xyzzy'
>
> or
>
> $P0 = find_global ['Foo']
> $P1 = find_global $P0, ['Bar'; 'Baz']
> $P2 = find_global $P1, 'xyzzy'
...
> That is, if we don't specify the name of the thing in the namespace we
> get the namespace PMC itself, which we can then go look things up in.
> This is handy since it means code can cache the namespace PMC so if
> there are a dozen variables to go fetch out, we don't have to do the
> multi-level hash lookup each time. (That'd be nasty)
Several things here:
1) By your name mangling scheme, it seems we don't need a special
syntax for looking up a namespace--it's just something else in the
namespace above it, so looking up "Foo::Bar" would be:
$P0 = root_namespace
$P1 = find_global $P0['Foo'], 'Bar'
But if everything's lumped together and name-mangled in a non-segmented
namespace, then you don't need find_global, you just need this (as Leo
suggested):
$P0 = root_namespace
$P1 = $P0['Foo'; 'Bar'] # P1 holds Foo::Bar
$P2 = $P1['Baz'; 'xyzzy'] # P2 holds xyzzy from Foo::Bar::Baz
$P2 = $P0['Foo'; 'Bar'; 'Baz'; 'xyzzy'] # same thing
I was formerly a proponent of keeping the final "thing" in a separate
parameter, but with your current mangling-plus-flat-namespace proposal,
I don't see a reason for it.
2) What's putting in the trailing null bytes (HLL programmer, compiler,
or Parrot's implementation of the lookup)? I'd assume it would be the
HLL compiler, so shouldn't the above be:
$P2 = $P0['Foo\0'; 'Bar\0'; 'Baz\0'; '$xyzzy'] #or at least, shouldn't
the PASM variant look like this
(In particular, I noticed you said, "a thing named xyzzy", but didn't
include a sigil?)
3) Canonical decomposition: Consider the following:
$P0 = root_namespace
$P2 = $P0['Foo\0'; 'Bar\0'; 'Baz\0'; '$xyzzy']
v.
$P0 = root_namespace
$P0 = $P0['Foo\0']
$P0 = $P0['Bar\0']
$P0 = $P0['Baz\0']
$P2 = $P0['$xyzzy']
(The example's the same even with your other syntax.)
These appear equivalent, _but_ with the former, the root namespace
could take into consideration to whole list keys, and decide what to
return based on that information; with the latter, it only sees
'Foo\0'. So in the general case (in light of namespace tie-ing), they
could do different things. The upshot of this is that we need to make
explicit the algorithm followed by a multi-keyed lookup--basically,
iterative v. recursive, if you think it through. Recursive is more
flexible and powerful (because any namespace along the way can "short
circuit" the rest of the lookup), but means that HLL compilers must
emit the first option above (with an iterative approach, where Parrot
controls the algorithm and namespaces have less control, a compiler
would have a choice).
4) Similar to (3), you can't do much caching of a namespace PMC, in
light of tying and such. Consider:
$a = $Foo::Bar::bar;
somesub();
$b = $Foo::Bar::zoo;
A compiler can't safely optimize this to cache the lookup of the
Foo::Bar namespace, because somesub() might perform tying of one of the
relevant namespaces, or otherwise rearrange the hierarchy, and you'd
get the wrong $zoo. Even without somesub(), in light of namespace tying
it's possible that the lookup of $bar actually caused a namespace
rearrangement, so you can't even cache across adjacent lookups (or a
lookup and a store, such as $Foo::Bar::bar = $Foo::Bar::zoo).
5) Python. Language crossing issues aside, there are issues with Python
itself. Python does not know, at compile-time, what's a namespace and
what's some other sort of object. Consider:
x = a.b.c.d
Here a, a.b, and a.b.c might all be modules, or they might just be
attributes of non-module objects.
That has some consequences: (a) The Python HLL compiler can't
name-mangle namespace/module names with a null byte, since it doesn't
know, nor can really the lookup process do this behind the scenes,
since it doesn't know either--namespaces are just objects with lots of
attributes. (b) Lookup will have to be "one level at a time"--the
compiler can't emit find_global ['a'; 'b'; 'c']..., since it can't know
it's doing a namespace lookup (though the keyed-access approach might
save us). (c) If we use keyed access, a Python compiler could
potentially emit $P0['a'; 'b'; 'c'; 'd'], but you wouldn't get this
from Python-to-Parrot bytecode translation, so in light of (3), with
namespace tying you could get different behavior for the same Python
code depending on whether you bytecode translated or compiled
direct-to-Parrot, so you'd probably want to stick to
one-level-at-a-time even for direct-to-Parrot compilers. (d) In light
of this, Python and Perl might "respond" very differently to namespace
tying, since the lookup algorithms might be different.
6) I don't think you've covered how Python code would access Perl
variables in your proposed scheme.
> Making one section of a namespace an alias for another part is just a
> matter of getting the PMC for the section of the namespace tree you
> want to alias and sticking it into the namespace it should hang off
> of. Right *now* I'm not inclined to have a store_global variant to do
> this, but if we want to completely hide all namespace operations
> behind ops I'm OK with that.
>
> Now, with that out of the way, let's talk about overlaid namespaces.
I don't think we need special ops for namespace overlaying. If we have
API powerful enough to allow one to create namespaces with custom
behavior, then "overlaying" is just creating a custom namespace which
delegates lookups into other namespaces, and then just storing this
into the appropriate slot. I think we'd get overlaying for free. For
instance, this is trivial to do in Python today:
>>> class wrapper:
... def __init__(self, modulename):
... self.wrapped = __import__(modulename)
... def __getattr__(self, attrname):
... print "Getting " + attrname
... return getattr(self.wrapped, attrname)
...
>>> os = wrapper("os")
>>> os.O_APPEND
Getting O_APPEND
8
>>> os.stat("/dev/null")
Getting stat
(8630, 27649924L, 26798244L, 1, 0, 0, 0L, 1095909600, 1096963882,
1096963882)
Here, I created something which looks like the os module, but actually
logs all lookups before delegating to the "real" os module.
It would be similarly simple to create something which searches a list
of namespaces:
mynamespace = multiwrapper("os", "sys", "xml")
mynamespace.foo() # searches os, then sys, then xml
If we can do this, there's no need for dedicated ops.
JEff
Jeff Clites
> Tim Bunce <tim....@pobox.com> wrote:
>>> Now, with that out of the way, let's talk about overlaid namespaces.
>>
>> I don't think I ever read a description of what the purpose of this
>> was.
>> I get the "what" but not the "why". Without the "why" it's hard to
>> critique the "how".
>
> Clearly I'm not Dan, but I think the idea here is that, for example,
> in the following code:
>
> module Foo::Bar {
> class Baz { $quux }
> }
>
> You can have "the current namespace" actually be [ ::Foo::Bar::Baz,
> ::Foo::Bar, ::* ] (or, for the last one, whatever the namespace that
> @*ARGS and friends are in is called), so that the search for $quux can
> be done very easily.
This may have changed for Perl6, but at least for Perl5, non-lexicals
are only ever looked for in the current package, the top-level (main)
namespace, plus the pre-defined variables (don't know if these are
handled at compile-time), according to the Camel. So having an
arbitrary search list might not be used in the Perl case.
JEff
Brent 'Dax' Royal-Gordon
> On Oct 4, 2004, at 9:58 PM, Brent 'Dax' Royal-Gordon wrote:
> > You can have "the current namespace" actually be [ ::Foo::Bar::Baz,
> > ::Foo::Bar, ::* ] (or, for the last one, whatever the namespace that
> > @*ARGS and friends are in is called), so that the search for $quux can
> > be done very easily.
>
> This may have changed for Perl6,
It has, thus the reason I suggested this.
> but at least for Perl5, non-lexicals
> are only ever looked for in the current package, the top-level (main)
> namespace, plus the pre-defined variables (don't know if these are
> handled at compile-time), according to the Camel. So having an
> arbitrary search list might not be used in the Perl case.
--