[perl #40443] Separate vtable functions from methods (using :vtable)

[Matt Diephouse]

# New Ticket Created by Matt Diephouse
# Please include the string: [perl #40443]
# in the subject line of all future correspondence about this issue.
# <URL: http://rt.perl.org/rt3/Ticket/Display.html?id=40443 >

At the OSCON 2006 Hackathon, it was decided that we should separate
vtables from methods and add a new :vtable label for PIR subs to mark
them as vtable functions. This gets rid of the current namespace
pollution caused by vtables and allows us to abandon the leading
underscores for PIR vtable functions.

.sub get_string :vtable
.return("end namespace pollution!")
.end

It appears that this never made it into RT.

--
Matt Diephouse

[jnthn@jnthn.net via RT]

On Sun Oct 01 16:22:10 2006, mdiep wrote:
> At the OSCON 2006 Hackathon, it was decided that we should separate
> vtables from methods and add a new :vtable label for PIR subs to mark
> them as vtable functions. This gets rid of the current namespace
> pollution caused by vtables and allows us to abandon the leading
> underscores for PIR vtable functions.
>
> .sub get_string :vtable
> .return("end namespace pollution!")
> .end
>

Just to check, that this is still meant to happen? Anyone feel it should
be put off until the objects/namespaces stuff is sorted out, or shall I
just dive right in?

Thanks,

Jonathan

[Chromatic]

On Wednesday 25 October 2006 16:38, jn...@jnthn.net via RT wrote:

> Just to check, that this is still meant to happen?

Unless Allison objects, it seems like the cleanest way to support overloading
semantics from various languages (Perl 5: FETCH, Python:
__some_really_ugly_method_name__) without actually making namespace
collisions.

> Anyone feel it should be put off until the objects/namespaces stuff is
> sorted out, or shall I just dive right in?

As long as the interface works (and it seems sensible to me), we can fix the
implementation in plenty of ways.

-- c

[Allison Randal]

This is the main thing Chip and I talked about in our last face-to-face
meeting. We came up with 3 basic parameters: whether a method is a
vtable method, whether it has a vtable name distinct from the method
name, and whether it has a method name at all (or is anonymous, i.e.
only a vtable method). The interface I scrawled out over coffee is:

# method name is the same as vtable name
.sub get_string :method :vtable

# accessible as either $obj.stringify() or vtable
.sub stringify :method :vtable('get_string')

# accessible only as vtable
.sub get_string :method :anon :vtable
.sub stringify :method :anon :vtable('get_string')

Which reuses the existing concept of:

# method has an entry in the namespace
.sub stringify :method

# method has no entry in the namespace
.sub stringify :method :anon

(There's not much point in using the 4th option where you specify a name
for the method, make it anonymous, and then specify a separate name for
the vtable entry, but we'll support it for the sake of orthogonality.)

The reasoning for making :vtable additive rather than exclusive with
:method is that it's common to want to add behavior that's both a method
on the object and a vtable entry.

As a side effect of this solution, the leading double underscore goes
away for vtable names. So it's now just 'init' instead of '__init',
'get_bool' instead of '__get_bool', etc. The underscores were just a
hack to alleviate namespace pollution anyway.

Even though "pure" vtable methods aren't stored in the addressable part
of the namespace, we came to the conclusion that they should be attached
to the namespace, because an anonymous class can be associated with a
namespace at runtime, and when it is, it should get not only the methods
of that namespace, but also the overloaded vtable entries.

(I can see an argument of brevity for an alternate solution where the
3rd and 4th options have only :vtable and not :method and :anon:

# accessible only as vtable
.sub get_string :vtable
.sub stringify :vtable('get_string')

but I don't like the fact that this would make :vtable mean different
things in different contexts. Together with :method, :vtable would mean
"also make an entry in the vtable overload list for this namespace", but
without :method it would mean "don't make an entry for this subroutine
in the namespace, and treat this subroutine as a method, and make an
entry in the vtable list". I prefer to keep each of those separated into
their own adverbs, especially since we already have the adverbs.)

Allison

[Patrick R. Michaud]

On Wed, Oct 25, 2006 at 11:02:59PM -0700, Allison Randal wrote:
> jn...@jnthn.net via RT wrote:
> >On Sun Oct 01 16:22:10 2006, mdiep wrote:
> >>At the OSCON 2006 Hackathon, it was decided that we should separate
> >>vtables from methods and add a new :vtable label for PIR subs to mark
> >>them as vtable functions.
> >

> >Just to check, that this is still meant to happen? Anyone feel it should
> >be put off until the objects/namespaces stuff is sorted out, or shall I
> >just dive right in?
>
> This is the main thing Chip and I talked about in our last face-to-face
> meeting. We came up with 3 basic parameters: whether a method is a
> vtable method, whether it has a vtable name distinct from the method
> name, and whether it has a method name at all (or is anonymous, i.e.
> only a vtable method). The interface I scrawled out over coffee is:
>
> # method name is the same as vtable name
> .sub get_string :method :vtable
>
> # accessible as either $obj.stringify() or vtable
> .sub stringify :method :vtable('get_string')
>
> # accessible only as vtable
> .sub get_string :method :anon :vtable
> .sub stringify :method :anon :vtable('get_string')

> ...

+1

Pm

[jnthn@jnthn.net via RT]

Hi,

Thanks Allison for clarifications and @other for agreement; I've taken
this ticket and will get cracking on implementing this.

Jonathan

[Jonathan Worthington]

Hi,

As of r15039, :vtable and :vtable("...") are now both implemented. See
example code at the end of this email.

As of now, please use this new syntax. I have left in support for the
old "__"-prefix lookup as plenty of old code is using it; please note
that if you have a :vtable("xxx") and a "__xxx" then the first will be
called, not the second.

I'll leave it to Chip and/or Allison to decide how long the deprecation
cycle is for this.

Implementation wise, at the moment I'm just storing these entries in
"\0VTABLE\0" sub-namespace within the class namespace, however I'm not
sure I like this (feels like we got rid of one bit of namespace
pollution by adding another, albeit a less noticeable one). I don't see
another way to attach the VTABLE namespace to the main one, however. But
that's an implementation detail - even if we change that any
:vtable-using code will go on working fine.

Tests and some extra error checking code to come.

Jonathan

.sub main :main
$P0 = newclass [ "Test" ]
$P1 = new [ "Test" ]
$I1 = $P1[11]
print $I1
print "\n"
$S1 = $P1[11]
print $S1
print "\n"
.end

.namespace [ "Test" ]

.sub get_integer_keyed_int :method :vtable
.param int key
.return(42)
.end

.sub monkey :method :vtable("get_string_keyed_int")
.param int key
.return("monkey")
.end

[Jonathan Worthington]

Jonathan Worthington wrote:
> Tests and some extra error checking code to come.

Also now done. If you write one of:

.sub not_a_vtable_method :method :vtable
.sub badger :method :vtable("not_a_vtable_method")

It's a compiler error.

I'll leave this ticket open a few more days for comments, then if there
are none close it up.

Thanks,

Jonathan

[Allison Randal]

Jonathan Worthington wrote:
> Hi,
>
> As of r15039, :vtable and :vtable("...") are now both implemented.
> See example code at the end of this email.

Awesome!

> As of now, please use this new syntax. I have left in support for the
> old "__"-prefix lookup as plenty of old code is using it; please
> note that if you have a :vtable("xxx") and a "__xxx" then the first
> will be called, not the second.

Updating Punie went smoothly, but when I updated TGE::Rule:

-.sub __init :method
+.sub 'init' :method :anon :vtable

I got this error:

../../parrot -o TGE/Rule.pbc --output-pbc TGE/Rule.pir
error:imcc:'init' is not a v-table method, but was used with :vtable.

in file 'TGE/Rule.pir' line 49

> I'll leave it to Chip and/or Allison to decide how long the
> deprecation cycle is for this.

Since it is so widely used, let's deprecate it two releases from now, so
we can have one release with both options, and then deprecate it in the
following release.

> Implementation wise, at the moment I'm just storing these entries in
> "\0VTABLE\0" sub-namespace within the class namespace, however I'm
> not sure I like this (feels like we got rid of one bit of namespace
> pollution by adding another, albeit a less noticeable one). I don't
> see another way to attach the VTABLE namespace to the main one,
> however. But that's an implementation detail - even if we change that
> any :vtable-using code will go on working fine.

This is a good place to start, but I agree it's not what we want
ultimately. Chip and I talked about attaching a data structure of vtable
entries as a property on the namespace object.

Allison

[Jonathan Worthington]

Allison Randal wrote:
> Updating Punie went smoothly, but when I updated TGE::Rule:
>
> -.sub __init :method
> +.sub 'init' :method :anon :vtable
>
> I got this error:
>
> ../../parrot -o TGE/Rule.pbc --output-pbc TGE/Rule.pir
> error:imcc:'init' is not a v-table method, but was used with :vtable.
>
> in file 'TGE/Rule.pir' line 49

leo++ fixed half of this when he cleaned up the validation code, but
there was another problem too. Fixed that in 15048, so this should work now.

> Since it is so widely used, let's deprecate it two releases from now,
> so we can have one release with both options, and then deprecate it in
> the following release.

Sounds good.

> This is a good place to start, but I agree it's not what we want
> ultimately. Chip and I talked about attaching a data structure of
> vtable entries as a property on the namespace object.

I implemented this to test it out and it kinda worked. However,
properties don't get serialized so if you compile the program to a PBC
and run that it doesn't work. Additionally, properties don't get inherited.

It'd be nice to actually attach these methods to a v-table itself
somehow. However, unless we make classes a compile-time construct rather
than "turning a namespace into a class" using newclass as we do now, I
don't see how this can happen (at least, not cleanly). Maybe something
to be addressed in the objects PDD.

Thanks,

Jonathan

[Chromatic]

On Sunday 29 October 2006 08:25, Jonathan Worthington wrote:

> It'd be nice to actually attach these methods to a v-table itself
> somehow. However, unless we make classes a compile-time construct rather
> than "turning a namespace into a class" using newclass as we do now, I
> don't see how this can happen (at least, not cleanly). Maybe something
> to be addressed in the objects PDD.

I think that was part of the motivation for :init.

-- c

[Jonathan Worthington]

But by the time you're running anything inside :init it's runtime.

Jonathan

[Allison Randal]

Jonathan Worthington wrote:
>
>> This is a good place to start, but I agree it's not what we want
>> ultimately. Chip and I talked about attaching a data structure of
>> vtable entries as a property on the namespace object.
> I implemented this to test it out and it kinda worked. However,
> properties don't get serialized so if you compile the program to a PBC
> and run that it doesn't work.

Why don't they get serialized?

> Additionally, properties don't get inherited.

In the sense that they're only attached to individual object instances
of the class, not to the class itself. And even if they were attached to
the class, it would be to the class PMC, and wouldn't be picked up by
subclass PMCs.

Although, in this case what we're talking about is a property attached
to an object instance of the namespace class. Inheritance is determined
by the classes associated with the namespace object, not the namespace
object itself. The storage of vtable methods in the namespace is really
only temporary until they can be loaded as vtable overloads on the
class. (It's also there for the sake of later use by other classes if
they become associated with that namespace.) The vtable overloads only
enter into inheritance once they are loaded into the vtable.

This is why properties seemed like a good fit: the vtable entries in
namespaces are optional, out-of-band information, only used in loading.
But, they could just as well be implemented as a separate attribute in
the namespace class.

> It'd be nice to actually attach these methods to a v-table itself
> somehow. However, unless we make classes a compile-time construct rather
> than "turning a namespace into a class" using newclass as we do now, I
> don't see how this can happen (at least, not cleanly).

It also wouldn't allow for runtime association of an anonymous class
with a namespace, or runtime switching of the namespace a class is
associated with.

> Maybe something to be addressed in the objects PDD.

Yes, good entry for the questions section.

Allison

[Jonathan Worthington]

Allison Randal wrote:

> Jonathan Worthington wrote:
>> I implemented this to test it out and it kinda worked. However,
>> properties don't get serialized so if you compile the program to a
>> PBC and run that it doesn't work.
> Why don't they get serialized?

I guess, that's just not implemented. But I don't know if this is a
quirk of the implementation or a design decision that properties are
just runtime things. If we want it to always happen, it needs to be
implemented as part of the standard PMC packing code though, I think.
(Note that default.pmc implements properties, so any PMC can have them
set, it appears; serializing them as standard would thus make sense).

>> Additionally, properties don't get inherited.
> In the sense that they're only attached to individual object instances
> of the class, not to the class itself. And even if they were attached
> to the class, it would be to the class PMC, and wouldn't be picked up
> by subclass PMCs.
>
> Although, in this case what we're talking about is a property attached
> to an object instance of the namespace class. Inheritance is
> determined by the classes associated with the namespace object, not
> the namespace object itself. The storage of vtable methods in the
> namespace is really only temporary until they can be loaded as vtable
> overloads on the class. (It's also there for the sake of later use by
> other classes if they become associated with that namespace.) The
> vtable overloads only enter into inheritance once they are loaded into
> the vtable.
>
> This is why properties seemed like a good fit: the vtable entries in
> namespaces are optional, out-of-band information, only used in
> loading. But, they could just as well be implemented as a separate
> attribute in the namespace class.

Agree; looking now at what I had to do (walk the MRO) to get inheritance
of :vtable methods to work already, doing it with the data in properties
is no harder. That leaves the serialization issue as the main one. A
ruling on that, and I can implement :vtable using a property. (leo
pointed out the double indirection, as you're looking up the vtable hash
in the property hash, is a performance issue - we can perhaps optimize
this later...I care more about getting the correct behavior right now).

>> It'd be nice to actually attach these methods to a v-table itself
>> somehow. However, unless we make classes a compile-time construct
>> rather than "turning a namespace into a class" using newclass as we
>> do now, I don't see how this can happen (at least, not cleanly).
> It also wouldn't allow for runtime association of an anonymous class
> with a namespace, or runtime switching of the namespace a class is
> associated with.

Must...get...head...out...of....static-language-world. :-)

Thanks,

Jonathan

[Chromatic]

On Monday 30 October 2006 13:53, Jonathan Worthington wrote:

> Agree; looking now at what I had to do (walk the MRO) to get inheritance
> of :vtable methods to work already, doing it with the data in properties
> is no harder.

This is perhaps a sign that it's time for a super or next opcode.

-- c

[Allison Randal]

Jonathan Worthington wrote:
>> error:imcc:'init' is not a v-table method, but was used with :vtable.
>> in file 'TGE/Rule.pir' line 49
> leo++ fixed half of this when he cleaned up the validation code, but
> there was another problem too. Fixed that in 15048, so this should work
> now.

This is fixed now (many thanks!)

I still have failing tests. It looks like the 'init' methods aren't
being called. (In TGE::Compiler the tree transformation rules are never
added to tree grammar. In TGE::Tree the AddrRegistry PMC in the 'agid'
attribute is never created, and in TGE::Grammar the ResizablePMCArray in
the 'rules' attribute is never created, so they cause 'Null PMC
access...' errors.)

Is there a lingering piece of Parrot that depends on finding the init
vtable methods prepended with "__"?

Allison

[Allison Randal]

Jonathan Worthington wrote:
>>
>> This is why properties seemed like a good fit: the vtable entries in
>> namespaces are optional, out-of-band information, only used in
>> loading. But, they could just as well be implemented as a separate
>> attribute in the namespace class.
> Agree; looking now at what I had to do (walk the MRO) to get inheritance
> of :vtable methods to work already, doing it with the data in properties
> is no harder. That leaves the serialization issue as the main one. A
> ruling on that, and I can implement :vtable using a property.

Can anyone think of a reason we shouldn't serialize properties? If not,
let's try it. If we don't like the results, we can back it out.

I suspect that originally it was expected that properties would only be
attached at runtime. But, since we have the option of flagging subs to
execute at various stages of compilation, it seems safe to assume that
properties can be attached at various stages of compilation too.

> (leo
> pointed out the double indirection, as you're looking up the vtable hash
> in the property hash, is a performance issue - we can perhaps optimize
> this later...I care more about getting the correct behavior right now).

He has a good point, but agreed, let's get it working and optimize later.

Allison

[Leopold Toetsch]

Am Montag, 30. Oktober 2006 22:53 schrieb Jonathan Worthington:
> Allison Randal wrote:
> > Jonathan Worthington wrote:
> >> I implemented this to test it out and it kinda worked. However,
> >> properties don't get serialized so if you compile the program to a
> >> PBC and run that it doesn't work.
> >
> > Why don't they get serialized?
>
> I guess, that's just not implemented.

It's at least partially implemented. But I think going via props in that case
isn't the best way to implement it.

We have basically the problem that classes are only constructed at runtime,
which has 2 negative impacts: associating the vtable info with the class
needs some helper storage like the props and namespaces can only be vaguely
attached to the correct class at runtime too.

To get around that the following came to my mind:

- we create a class when needed during compile time
- it get's a flag set (one of private0..7) not_yet_finished_FLAG or such
- newclass, when checking if a class exists, examines that flag and continues
adding the runtime information
- the classes constructed in that way are frozen into the PBC

leo

[Larry Wall]

On Mon, Oct 30, 2006 at 11:48:14PM -0800, Allison Randal wrote:
: I suspect that originally it was expected that properties would only be

: attached at runtime. But, since we have the option of flagging subs to
: execute at various stages of compilation, it seems safe to assume that
: properties can be attached at various stages of compilation too.

Indeed, that's part of why Perl 6 eventually threw out properties
entirely, on the supposition that properties are simply attributes
that are mixed in at run-time rather than compiled in.

Larry

[Allison Randal]

Larry Wall wrote:
>
> Indeed, that's part of why Perl 6 eventually threw out properties
> entirely, on the supposition that properties are simply attributes
> that are mixed in at run-time rather than compiled in.

From an HLL perspective, I agree. But there's always more than one way
to satisfy an HLL abstraction.

From the Parrot perspective, attributes can only be added to classes,
while properties can be added to instantiated objects without modifying
the class (essentially creating a very lightweight singleton).
Properties can also be added to classes without appearing in the
instantiated objects of that class. It's a useful distinction.

I can see an argument for having an interface that can retrieve and
store both attributes and properties, but that's orthogonal to the
question of where the two kinds of data members are stored in the
object/class structure.

Allison

[Allison Randal]

Leopold Toetsch wrote:
>
> We have basically the problem that classes are only constructed at runtime,
> which has 2 negative impacts: associating the vtable info with the class
> needs some helper storage like the props and namespaces can only be vaguely
> attached to the correct class at runtime too.

As a virtual machine for dynamic languages, one of the core requirements
is that it has to be able to construct a class completely at runtime. We
may add compile-time construction of classes at some point as an
optimization, but we'll never eliminate runtime construction. That
"vague attachment" is an intentional distinction between namespaces and
the classes associated with those namespaces, allowing for runtime
association of a class with a namespace (even an anonymous class).

> To get around that the following came to my mind:
>
> - we create a class when needed during compile time
> - it get's a flag set (one of private0..7) not_yet_finished_FLAG or such
> - newclass, when checking if a class exists, examines that flag and continues
> adding the runtime information
> - the classes constructed in that way are frozen into the PBC

I can see the appeal, but unfortunately it won't give us all the
features we need. This is certainly an option to consider if/when we do
add compile-time construction of classes, with the understanding that
only a subset of cases would be able to use it (as is true of most
optimizations).

Allison

[Nicholas Clark]

On Tue, Oct 31, 2006 at 10:43:11PM -0800, Allison Randal wrote:
> Leopold Toetsch wrote:
> >
> >We have basically the problem that classes are only constructed at
> >runtime, which has 2 negative impacts: associating the vtable info with
> >the class needs some helper storage like the props and namespaces can only
> >be vaguely attached to the correct class at runtime too.
>
> As a virtual machine for dynamic languages, one of the core requirements
> is that it has to be able to construct a class completely at runtime. We
> may add compile-time construction of classes at some point as an
> optimization, but we'll never eliminate runtime construction. That

When serialising data structures, is parrot going to need to serialise any
anonymous or otherwise run time generated classes?

If so, isn't this mechanism going to be capable of achieving "compile time"
class generation?

Nicholas Clark

[Leopold Toetsch]

Am Mittwoch, 1. November 2006 07:43 schrieb Allison Randal:
> We
> may add compile-time construction of classes at some point as an
> optimization, but we'll never eliminate runtime construction.

Of course not. I never said that.

> > To get around that the following came to my mind:
> >
> > - we create a class when needed during compile time

> I can see the appeal, but unfortunately it won't give us all the

> features we need. This is certainly an option to consider if/when we do
> add compile-time construction of classes, with the understanding that
> only a subset of cases would be able to use it (as is true of most
> optimizations).

It's actually not an optimization IMHO. It could be one, if we add support for
adding attributes at compile-time too [1]. But for now, I'm just speaking of
consolidating class construction code.

We now have:

compile-time run-time
-----------------------------------------------
.namespace ['Foo']
.sub bar :method
.sub baz :method :vtable(..)
cl = newclass 'Foo'
addattribute cl, 'x'
addmethod cl, 'blurb', meth

The namespace directive in combination with :method is defining parts of a
class namely all the statically compiled methods and the namespace. The
problem is how to forward the compile-time part of a class properly to the
run-time.

Actually the current implementation is partly wrong to deal with too much
usage of:

.namespace ['Foo']
.sub _init :load
cl = newclass ['Foo'] # would be ['Foo';'Foo']

and similar. Above should be

.sub _init :load
cl = newclass ['Foo'] # ['Foo']
...
.namespace ['Foo']

Having a class available at compile-time would elminate this ambuigity.

> Allison

leo

[1] in fact, we can do that already - see my answer to Nicholas.

[Leopold Toetsch]

Am Mittwoch, 1. November 2006 10:57 schrieb Nicholas Clark:
> When serialising data structures, is parrot going to need to serialise any
> anonymous or otherwise run time generated classes?

Yep, and it does so already.

> If so, isn't this mechanism going to be capable of achieving "compile time"
> class generation?

Indeed. There's a hackish implementation[1] for achieving compile-time class
(and object) creation:

$ cat t/pmc/sub_42.pir
.sub make_obj :immediate, :anon
.local pmc cl, o

cl = newclass "Foo"
addattribute cl, 'x'

o = new 'Foo'
$P0 = new String
$P0 = "ok 1\n"
setattribute o, 'x', $P0
.return (o)
.end

.sub main :main
.const .Sub o = "make_obj"
$P0 = getattribute o, 'x'
print $P0
.end

$ ./parrot -o s42.pbc t/pmc/sub_42.pir
$ ./parrot -t s42.pbc
23 set P1, PC8 ...
26 getattribute P0, P1, "x" ... P1=Object(Foo)= ...
30 print P0 P0=String=PMC(0xb61440 Str:"ok 1\n")
ok 1
32 end

(trace edited for clarity)

But attaching the namespace to the class still suffers from the mentioned
problem, only earlier so.

> Nicholas Clark

leo

[1] the return result of the :immediate (if any) replaces the subroutine
object.

[Allison Randal via RT]

The :vtable feature has been implemented. Any remaining bugs in the
implementation should be submitted as separate tickets.