Objects, classes, metaclasses, and other things that go bump in the night
Dan Sugalski
time to turn our attention to objects and all the stuff that goes
with them.
I'd originally thought that the bits we'd put in place would be
sufficient to do everyone's object system (well, all the languages
that we explicitly care about at least) but, well... that turns out
not to be the case.
Here's a quick recap on the architecture bits that already exist.
Each PMC can:
*) Return a PMC which represents that PMC's class
*) Look up and return a PMC representing a named method
*) Look up and return a named property for the PMC
I think that we can accommodate most everyone's needs with this,
assuming we set up some protocols and give a really big kick to the
code in objects.c.
The rest of this should be prefaced with the caveat that I'm finding
that the more I'm learning about how all the different languages do
objects the less I actually know what's going on, so this all might
be wrong, or at least really sub-optimal.
Anyway, with the vtable slots as they stand, we can make a method
call on an object. There's a single op for this -- it calls the
object's find_method function and then invokes the resulting PMC. The
object is in complete control over what invokable PMC gets returned,
and the invoked PMC has full control over what it actually does.
Python has this interesting concept where properties and methods are
sort of mixed -- that is if you do:
a = foo.some_thing
a will be either the property some_thing or the method object you'd
get if you invoked some_thing on foo, bound up such that you can
later do:
a()
and it's the same as if you'd done foo.some_thing(). I think. Pretty
sure, at least, though I'm not sure of the order. Anyway, for this
the core functionality is fine, though it assumes that the python
compiler will emit a "fetch prop/test/fetch method/bind method"
sequence for the a=foo.bar statement. I think I'm OK with that,
though I'm more than willing to take arguments to the contrary.
("It's not easy" isn't a good argument -- we're down at the machine
level. Nothing's easy. "It breaks because of X and therefore has to
be a fundamental thing to guarantee it works" is a good argument)
This assumes we have binding functionality that takes some parameters
and produces an invokable PMC that uses those, and any other
provided, parameters later on. Given that Perl 6 is doing
higher-order functions (or curried functions, or whatever) where
there's going to be a fair amount of binding going on I'm OK with
core support for this, where core may be "we have a bind PMC type
with some methods on it" or something like that. Dunno, we can work
that out later.
PMCs also implement 'is', 'does', and 'can'. These are three vtable
methods that allow code to query a PMC to see if it it 'is' an
instance of a named class, if it 'does' a particular interface, and
whether it 'can' invoke the named method. PMCs are supposed to answer
truthfully, however no assumptions are made as to how the pmc
actually is, does, or can do something. It's perfectly acceptable for
a PMC that has an AUTOLOAD (or its non-perl equivalent) function to
answer true to any can query, and how an interface is implemented in
a PMC's parent classes (if indeed it even has any parent classes) is
also generally irrelevant, so long as the answers are either correct
or egregiously wrong.
Anyway, so much for the 'outside world' view of objects as black box
things that have properties and methods. That part's easy, and I
think we're fine there.
The tricky part is all the bits that back the objects. Or, rather,
arranging so that everyone can reasonably have their own custom
backing bits while still allowing mix-n-match inheritance between the
bits, and providing enough information to do things properly, or to
cleanly fail. We're not going to *require* that classes of different
sorts be able to inherit from one another, just require that if they
do allow that then they behave properly.
Almost everything we do here is going to be with method calls.
There's very little that I can see that requires any faster access
than that, so as long as we have a proper protocol that everyone can
conform to, we should be OK.
The one big exception to the "we use methods" thing is with composite
objects. The only sane way I can think of to handle cross-system
inheritance is through delegation and composition -- that is, if you
have class A which inherits from Foo and A_Class, each of which is
from a unique object system (like, say, CLOS, perl 5's 'object
system', and new-style python classes), there's no way a single class
system can reasonably express the differing semantics, so the only
sane thing to do is have a composite object. The 'main' object is
class A's (however that class instantiates objects) and hanging off
it are two sub-objects, one for Foo and one for A_Class. These
sub-objects will have a flag set marking them as a sub-object, and
will have the master composing object as a property. We can get into
how this is handled later. The important thing here is that we are
going to have a flag bit that says "I'm only part of an object" and
whenever a method call is made on a sub object parrot will
automatically look for the 'master' object and make the method call
on it instead.
Parrot's object system is going to be generally class-based -- that
is, each object has a class that's responsible for managing the
object. Classes, being objects, themselves have classes, but past
that we don't look too closely or we'll end up getting bitten by the
snake. (No, not that snake, the other one) This does mean that
prototype-based object systems are going to be kinda annoying, but if
you do things right then the object you're using as a prototype is
your class anyway, so it ought to work out OK. I think. Hopefully, at
least, since prototype-based object systems aren't our main interest.
A class is responsible for instantiating objects, providing basic
information, providing subclasses of itself, merging in with another
class for multiple inheritance, and managing methods. Basically we
access the class object when we want to subclass a class, add a class
into another class' inheritance hierarchy, make new objects, and
mange a namespace. (I think. we might go with classes managing their
own methods, or may just declare that classes are going to find their
methods in a namepsace of the same name so had darned well better go
look there. I'm as yet somewhat undecided -- selector namespaces are
awfully tempting, and there's MMD to consider)
So. What does a class need to be able to do? I'm thinking the
following basic methods (the names are up in the air)
subclass - To create a subclass of a class object
add_parent - To add a parent to the class this is invoked on
become_parent - Called on the class passed as a parameter to add_parent
class_type - returns some unique ID or other so all classes in one
class family have the same ID
instantiate - Called to create an object of the class
add_method - called to add a method to the class
remove_method - called to remove a method from a class
namespace_name - returns the name of this class' namespace
All objects also must be able to perform the method:
get_anonymous_subclass - to put the object into a singleton anonymous
subclass
I can see adding some information-fetching methods for introspection
to this list, so I'm up for those too.
Some fallout you might not have thought about: Objects are ultimately
responsible for handling method finding, which means that each object
controls the search order of its parent classes when looking for
methods, so if you want to fiddle with that, it's fine. It also means
you can mess around with the class hierarchy as it appears to
external code (fibbing about who you are, or aren't) and suchlike
things.
Now, there is one big gotcha here -- multimethod dispatch. I'm not
entirely sure that we can do this properly and still give classes
full control over how methods are looked for and where they go. I
*think* it can be done, but I'm not feeling clever enough to see how
without having a relatively costly double lookup on every method call
(first to see if there's a registered MMD method for the named
method, then the regular dispatch if there's not) so I'm not sure we
will. Efficient MMD wins, if we can make it look like
perl/python/ruby/tcl's method lookup rules are in force, even if they
really aren't under the hood.
--
Dan
--------------------------------------it's like this-------------------
Dan Sugalski even samurai
d...@sidhe.org have teddy bears and even
teddy bears get drunk
Sam Ruby
> Right, so with at least a basic rework of the string stuff in, it's time
> to turn our attention to objects and all the stuff that goes with them.
>
> I'd originally thought that the bits we'd put in place would be
> sufficient to do everyone's object system (well, all the languages that
> we explicitly care about at least) but, well... that turns out not to be
> the case.
>
> Here's a quick recap on the architecture bits that already exist.
>
> Each PMC can:
>
> *) Return a PMC which represents that PMC's class
> *) Look up and return a PMC representing a named method
> *) Look up and return a named property for the PMC
>
> I think that we can accommodate most everyone's needs with this,
> assuming we set up some protocols and give a really big kick to the code
> in objects.c.
>
> The rest of this should be prefaced with the caveat that I'm finding
> that the more I'm learning about how all the different languages do
> objects the less I actually know what's going on, so this all might be
> wrong, or at least really sub-optimal.
>
> Anyway, with the vtable slots as they stand, we can make a method call
> on an object. There's a single op for this -- it calls the object's
> find_method function and then invokes the resulting PMC. The object is
> in complete control over what invokable PMC gets returned, and the
> invoked PMC has full control over what it actually does.
That works fine today. And get_attr_str nearly does so too, though
SUPER(idx) returns "Can't set non-core object attribs yet".
These feel like the right building blocks. Provide a reasonable default
behavior, and allow PMCs to assume complete control when they need to.
> Python has this interesting concept where properties and methods are
> sort of mixed -- that is if you do:
>
> a = foo.some_thing
>
> a will be either the property some_thing or the method object you'd get
> if you invoked some_thing on foo, bound up such that you can later do:
>
> a()
>
> and it's the same as if you'd done foo.some_thing(). I think. Pretty
> sure, at least, though I'm not sure of the order. Anyway, for this the
> core functionality is fine, though it assumes that the python compiler
> will emit a "fetch prop/test/fetch method/bind method" sequence for the
> a=foo.bar statement. I think I'm OK with that, though I'm more than
> willing to take arguments to the contrary. ("It's not easy" isn't a good
> argument -- we're down at the machine level. Nothing's easy. "It breaks
> because of X and therefore has to be a fundamental thing to guarantee it
> works" is a good argument)
The Pirate compiler emits the following for "a=foo.bar":
find_lex $P1, 'foo'
getattribute $P0, $P1, 'bar'
store_lex -1, 'a', $P0
This works its way into Parrot_PyClass_get_attr_str, which does the
right thing.
> This assumes we have binding functionality that takes some parameters
> and produces an invokable PMC that uses those, and any other provided,
> parameters later on. Given that Perl 6 is doing higher-order functions
> (or curried functions, or whatever) where there's going to be a fair
> amount of binding going on I'm OK with core support for this, where core
> may be "we have a bind PMC type with some methods on it" or something
> like that. Dunno, we can work that out later.
I don't think this needs to be in the core. Objects should be free to
decide if they support a given attribute or not. If any given Perl
object wants to provide an invokable PMC in such circumstances, it is
free to do so. Or not. Python objects don't have that freedom.
> PMCs also implement 'is', 'does', and 'can'. These are three vtable
> methods that allow code to query a PMC to see if it it 'is' an instance
> of a named class, if it 'does' a particular interface, and whether it
> 'can' invoke the named method. PMCs are supposed to answer truthfully,
> however no assumptions are made as to how the pmc actually is, does, or
> can do something. It's perfectly acceptable for a PMC that has an
> AUTOLOAD (or its non-perl equivalent) function to answer true to any can
> query, and how an interface is implemented in a PMC's parent classes (if
> indeed it even has any parent classes) is also generally irrelevant, so
> long as the answers are either correct or egregiously wrong.
This is a messy discussion that we probably should come back to. What
does "does hash" really mean? Some languages put some restrictions on
keys. How hash misses are handled by any other language is generally
going to be a surprise.
> Anyway, so much for the 'outside world' view of objects as black box
> things that have properties and methods. That part's easy, and I think
> we're fine there.
Properties and methods? Or attributes and methods? Parrot provides all
three. Ultimately, guidance needs to be provided on what a "well
behaved" Parrot object provides as its view to the outside world. The
CLS document that was discussed before will need to cover this.
At the moment, Pirate assumes that the outside view is attributes and
methods. And, internally, both are stored as properties.
> The tricky part is all the bits that back the objects. Or, rather,
> arranging so that everyone can reasonably have their own custom backing
> bits while still allowing mix-n-match inheritance between the bits, and
> providing enough information to do things properly, or to cleanly fail.
> We're not going to *require* that classes of different sorts be able to
> inherit from one another, just require that if they do allow that then
> they behave properly.
Cool.
> Almost everything we do here is going to be with method calls. There's
> very little that I can see that requires any faster access than that, so
> as long as we have a proper protocol that everyone can conform to, we
> should be OK.
Agreed. In fact, "as long as we have a proper protocol that everyone
can conform to, we should be OK." is now officially my mantra of the day.
> The one big exception to the "we use methods" thing is with composite
> objects. The only sane way I can think of to handle cross-system
> inheritance is through delegation and composition -- that is, if you
> have class A which inherits from Foo and A_Class, each of which is from
> a unique object system (like, say, CLOS, perl 5's 'object system', and
> new-style python classes), there's no way a single class system can
> reasonably express the differing semantics, so the only sane thing to do
> is have a composite object. The 'main' object is class A's (however that
> class instantiates objects) and hanging off it are two sub-objects, one
> for Foo and one for A_Class. These sub-objects will have a flag set
> marking them as a sub-object, and will have the master composing object
> as a property. We can get into how this is handled later. The important
> thing here is that we are going to have a flag bit that says "I'm only
> part of an object" and whenever a method call is made on a sub object
> parrot will automatically look for the 'master' object and make the
> method call on it instead.
Parrot_CompositeObject_find_method should do that. object.c should not.
"as long as we have a proper protocol that everyone can conform to, we
should be OK."
> Parrot's object system is going to be generally class-based -- that is,
> each object has a class that's responsible for managing the object.
> Classes, being objects, themselves have classes, but past that we don't
> look too closely or we'll end up getting bitten by the snake. (No, not
> that snake, the other one) This does mean that prototype-based object
> systems are going to be kinda annoying, but if you do things right then
> the object you're using as a prototype is your class anyway, so it ought
> to work out OK. I think. Hopefully, at least, since prototype-based
> object systems aren't our main interest.
ECMAScript is prototype-based. And, of course, "as long as we have a
proper protocol that everyone can conform to, we should be OK."
"does Object" should mean that C<find_method> and C<get_attr_str> and a
few other interfaces are provided.
"does Class" should mean that C<instantiate> and a few other interfaces
are provided.
Nothing more. Nothing less.
> A class is responsible for instantiating objects, providing basic
> information, providing subclasses of itself, merging in with another
> class for multiple inheritance, and managing methods. Basically we
> access the class object when we want to subclass a class, add a class
> into another class' inheritance hierarchy, make new objects, and mange
> a namespace. (I think. we might go with classes managing their own
> methods, or may just declare that classes are going to find their
> methods in a namepsace of the same name so had darned well better go
> look there. I'm as yet somewhat undecided -- selector namespaces are
> awfully tempting, and there's MMD to consider)
Now for my first major pushback. What the heck is a "namespace"? I
realize that in Perl that namespaces and classes are hopelessly
intertwined, but in Python, ECMAScript, and other languages, all classes
are essentially anonymous, and generally only known by the lexical
variables that they happen to be bound to at any given time.
> So. What does a class need to be able to do? I'm thinking the following
> basic methods (the names are up in the air)
>
> subclass - To create a subclass of a class object
> add_parent - To add a parent to the class this is invoked on
> become_parent - Called on the class passed as a parameter to add_parent
> class_type - returns some unique ID or other so all classes in one
> class family have the same ID
> instantiate - Called to create an object of the class
> add_method - called to add a method to the class
> remove_method - called to remove a method from a class
> namespace_name - returns the name of this class' namespace
Now for my second major pushback. At the moment, INTVAL base_type
serves as class_type. Monotonically increasing integers work fine for a
closed set of core classes, possibly augmented by a few dynamically
loaded extensions. But the paradigm breaks down when classes may be
created on the fly and ultimately garbage collected.
All classes are objects, so class_type should return a PMC* which "does"
C<instantiate>.
And, yes, the PMC's for all the "core" and "dynclass" are indexed for
convenient access. But, in any case, there should be a "new_p_p" opcode
which should do a $1=$2->vtable->instantiate(interpreter);
> All objects also must be able to perform the method:
>
> get_anonymous_subclass - to put the object into a singleton anonymous
> subclass
Sorry, you lost me here.
> I can see adding some information-fetching methods for introspection to
> this list, so I'm up for those too.
Cool.
> Some fallout you might not have thought about: Objects are ultimately
> responsible for handling method finding, which means that each object
> controls the search order of its parent classes when looking for
> methods, so if you want to fiddle with that, it's fine. It also means
> you can mess around with the class hierarchy as it appears to external
> code (fibbing about who you are, or aren't) and suchlike things.
Yes.
> Now, there is one big gotcha here -- multimethod dispatch. I'm not
> entirely sure that we can do this properly and still give classes full
> control over how methods are looked for and where they go. I *think* it
> can be done, but I'm not feeling clever enough to see how without having
> a relatively costly double lookup on every method call (first to see if
> there's a registered MMD method for the named method, then the regular
> dispatch if there's not) so I'm not sure we will. Efficient MMD wins, if
> we can make it look like perl/python/ruby/tcl's method lookup rules are
> in force, even if they really aren't under the hood.
Yes, MMD is the big elephant in the room. A commit later today will
create a PyComplex and PyLong that (for the moment) is a wholesale clone
of Complex and BigInt respectively simply because I wanted to override a
few methods and couldn't figure out how to get MMD to behave the way I
wanted. Obviously, this will need to be refactored later.
- Sam Ruby
Leopold Toetsch
> Dan Sugalski wrote:
[ method lookup ]
> Parrot_CompositeObject_find_method should do that. object.c should not.
I think, when going down the class hierarchy, we just have to call
class->vtable->find_method() again, istead of the Parrot_find_global.
> And, yes, the PMC's for all the "core" and "dynclass" are indexed for
> convenient access. But, in any case, there should be a "new_p_p" opcode
> which should do a $1=$2->vtable->instantiate(interpreter);
That is exactly what is currently PObj = Pclass->vtable->new_extended().
>> All objects also must be able to perform the method:
>>
>> get_anonymous_subclass - to put the object into a singleton anonymous
>> subclass
> Sorry, you lost me here.
A singleton object is the sole instantiation of a class. You'll always
get that one object.
> Yes, MMD is the big elephant in the room. A commit later today will
> create a PyComplex and PyLong that (for the moment) is a wholesale clone
> of Complex and BigInt respectively simply because I wanted to override a
> few methods and couldn't figure out how to get MMD to behave the way I
> wanted. Obviously, this will need to be refactored later.
We need multiple inheritance for PMCs. A PyComplex isa(PyObject,
Complex) PMC. There is minimal support for that - used in the
OrderedHash PMC. But the PMC compiler should be more clever:
void add(...) {
MMD_PyComplex: {
Complex.SELF.add(...);
}
should install the MMD subroutine for the complex PMC
Parrot_Complex_add_Complex()
> - Sam Ruby
leo
Sam Ruby
> Sam Ruby <ru...@intertwingly.net> wrote:
>
>>Dan Sugalski wrote:
>
> [ method lookup ]
>
>>Parrot_CompositeObject_find_method should do that. object.c should not.
>
> I think, when going down the class hierarchy, we just have to call
> class->vtable->find_method() again, istead of the Parrot_find_global.
If "we" is Parrot_CompositeObject_find_method, then I'm quite OK with
that. If "we" is Parrot_default_find_method, then I'm fine too.
However, if "we" is src/object.c, then let's talk some more.
>>And, yes, the PMC's for all the "core" and "dynclass" are indexed for
>>convenient access. But, in any case, there should be a "new_p_p" opcode
>>which should do a $1=$2->vtable->instantiate(interpreter);
>
> That is exactly what is currently PObj = Pclass->vtable->new_extended().
Cool!
>>>All objects also must be able to perform the method:
>>>
>>> get_anonymous_subclass - to put the object into a singleton anonymous
>>> subclass
>
>>Sorry, you lost me here.
>
> A singleton object is the sole instantiation of a class. You'll always
> get that one object.
I know what a singleton is. I don't know what it means to "put the
object into a singleton anonymous subclass", or why you would want to do
such a thing.
>>Yes, MMD is the big elephant in the room. A commit later today will
>>create a PyComplex and PyLong that (for the moment) is a wholesale clone
>>of Complex and BigInt respectively simply because I wanted to override a
>>few methods and couldn't figure out how to get MMD to behave the way I
>>wanted. Obviously, this will need to be refactored later.
>
> We need multiple inheritance for PMCs. A PyComplex isa(PyObject,
> Complex) PMC. There is minimal support for that - used in the
> OrderedHash PMC. But the PMC compiler should be more clever:
>
> void add(...) {
> MMD_PyComplex: {
> Complex.SELF.add(...);
> }
>
> should install the MMD subroutine for the complex PMC
> Parrot_Complex_add_Complex()
Even if that were done, that would only handle cases where PyComplex
were the first argument. In general, it would be nice if there were a
way to express "for purposes of MMD this class is a Complex".
- Sam Ruby
Mark Sparshatt
>>>> All objects also must be able to perform the method:
>>>>
>>>> get_anonymous_subclass - to put the object into a singleton
>>>> anonymous
>>>> subclass
>>
>>
>>> Sorry, you lost me here.
>>
>>
>> A singleton object is the sole instantiation of a class. You'll always
>> get that one object.
>
>
> I know what a singleton is. I don't know what it means to "put the
> object into a singleton anonymous subclass", or why you would want to do
> such a thing.
>
I think that this is in order to supports Ruby's feature of being able
add methods to individual objects.
In the following code
class A
end
a = A.new
def a.method()
end
this creates an anomynous subclass of A and makes a an instance of the
new class.
--
Mark Sparshatt
Leopold Toetsch
> Leopold Toetsch wrote:
>>
>> I think, when going down the class hierarchy, we just have to call
>> class->vtable->find_method() again, istead of the Parrot_find_global.
> If "we" is Parrot_CompositeObject_find_method, then I'm quite OK with
> that. If "we" is Parrot_default_find_method, then I'm fine too.
> However, if "we" is src/object.c, then let's talk some more.
The object's find_method vtable can do the method lookup on its own of
course. The implementation in object.c should provide a reasonable
default MRO. Composite objects that are ignoring the presence of
foreign classes are not compliant to interoperbility.
When the default method lookup calls again the find_method vtable of
parents and this method continues method lookup in a compliant way, all
should work.
> Even if that were done, that would only handle cases where PyComplex
> were the first argument. In general, it would be nice if there were a
> way to express "for purposes of MMD this class is a Complex".
Yep.
> - Sam Ruby
leo
Leopold Toetsch
> Anyway, so much for the 'outside world' view of objects as black box
> things that have properties and methods.
[ ... ]
> Almost everything we do here is going to be with method calls.
> There's very little that I can see that requires any faster access
> than that, so as long as we have a proper protocol that everyone can
> conform to, we should be OK.
The distinction object vs class is still in the way a bit, when it comes
to method calls.
$ python
...
>>> i = 4
>>> i.__sub__(1)
3
>>> help(int)
...
| __sub__(...)
| x.__sub__(y) <==> x-y
Given that a PyInt is just a PMC it currently needs a lot of ugly hacks
to implement the full set of operator methods. dynclasses/py*.pmc is
mostly just duplicating existing code from standard PMCs.
Iff the assembler just emits a method call for the opcode:
sub P0, P1, P2
we have that equivalence too, with proper inheritance and static and dynamic
operator overloading.
I've duplicated FixedPMCArray.get_pmc_keyed_int() as a method, starting
with this line:
METHOD PMC* __getitem__(INTVAL key) {
and measured 5 M array lookups:
$ parrot -C vt-bench.pasm
Vtable get_pmc_keyed_int 0.32 s
__getitem__ 0.36 s
The actual code in the loop is:
lp2:
callmethodcc "__getitem__"
inc I16
lt I16, 5000000, lp2
Our internal name for this method is "__get_pmc_keyed_int" but that
doesn't really matter. Either the Python translator emits a proper name
or the method gets aliased at runtime.
The timing from above is again done with the CGP core and the method is
cached in the PIC (polymorphic inline cache).
This is the relevant part of the executed opcode:
typedef PMC* (*func_pi)(Interp*, PMC*, INTVAL);
func_pi f;
pic = (Parrot_PIC *) cur_opcode[1];
idx = REG_INT(5);
self = REG_PMC(2);
if (self->vtable->base_type == pic->lru[0].lr_type) {
f = (func_pi)pic->lru[0].f.real_function;
REG_PMC(5) = (f)(interpreter, self, idx);
goto *((void*)*(cur_opcode += 2));
}
The PIC has three cache slots. Literature states that the fast path is
hit for 94% of the cases. If the object of that method call at that
bytecode location changes, another 5% is executed in a second C<if>
block. If the type match fails or for the first time VTABLE_find_method
is called to get the function pointer.
When the find_method returns a subroutine object a different opcode is
executed that calls the overloaded method. Overloaded methods are
*not* executed in a secondary runloop anymore. This improves performance
by about 50%.
I've already shown that these scheme also nicely fits for function-like
opcodes (math.sin() and such). I think it's worth some more
consideration.
leo
Sam Ruby
Point of order: discussions would be a lot easier if they didnt *start
out* with perjorative terms like "ugly hacks". One of my goals is to
eliminate the need for if (!Interp_flags_TEST(INTERP,
PARROT_PYTHON_MODE)) sprinkled throughout the various "standard" (i.e.
Perl) PMCs.
Yes, this all needs to be radically refactored - and all perlisms and
pythonisms removed from the "standard" classes.
Meanwhile, feel free to ask questions or make suggestions.
With that out of the way, I will assert that in all the Python code I
have seen, use of infix operators and square brackets for indexing
overwhelmingly dominate direct use of the methods. Furthermore, it is
clear that operators like "sub_p_p_p" and "set_p_p_kic" will always
perform better than callmethodcc_sc "__sub__" and callmethodcc_sc
"__getitem__" respectively.
Therefore, Pirate emits the opcodes whenever direct use is made of
things like infix operators and square brackets. And implemented
methods like __sub__ which, while rarely used, will operate correctly by
invoking to the opcodes.
I *don't* see a need to heavily optimize for rarely used mechanisms.
I encourage you to check out Pirate. The IMCC output of pirate.py is
now remarkably close to the output of pie-thon.pl.
- Sam Ruby
Leopold Toetsch
> Almost everything we do here is going to be with method calls.
Just one additional note:
$ python
>>> from cmath import sin
>>> print sin(2+3j)
(9.15449914691-4.16890695997j)
leo
Sam Ruby
> OTOH when you have
>
> math.sin(2)
>
> Python emits:
>
> 6 44 LOAD_NAME 0 (math)
> 47 LOAD_ATTR 1 (sin)
> 50 LOAD_CONST 1 (2)
> 53 CALL_FUNCTION 1
>
> which actually is a method call on the "math" object. And the math
> object is a namespace alias of e.g. the Float PMC.
The "math" object is actually a module. Once imported, you also find it
lexically. What Pirate emits for math.sin(2) follows:
find_type $I0, "PyInt"
new $P0, $I0
$P0 = 2
find_lex $P2, 'math'
$P1=$P2.sin($P0)
In the direction I am currently heading, nothing that Pirate emits will
make any direct use of Parrot namespaces. (Yes, Parrot internally uses
namespaces for NCI methds - currently). All subroutines (except main)
are @ANON.
Do you find that surprising?
At the moment it is hard to argue how Parrot could/should change in
order to better support a language like Python. Many of the features
that exist in Parrot are based on assumptions of what language
implements will want. In several cases, I disagree with those
assumptions - i.e., YAGNI.
Am I right? Who can tell at this point. We'll know when we get there.
- Sam Ruby
Sam Ruby
> Sorry about that. These hacks of course include pieces like...
Cool. Peace.
>>.... Furthermore, it is
>>clear that operators like "sub_p_p_p" and "set_p_p_kic" will always
>>perform better than callmethodcc_sc "__sub__" and callmethodcc_sc
>>"__getitem__" respectively.
>
> Well, and that's not true. I've already shown the opposite:
> Here are agains these numbers (-O3 build, AMD 800, 5 Meg operations):
>
> MMD add PerlInt PerlInt 0.693220
> MMD add PerlInt INTVAL 0.609069
> MMD sub PerlInt PerlInt 0.475912
> MMD sub PerlInt INTVAL 0.490149
> PIR add PerlInt PerlInt 8.312403
> PIR sub PerlInt PerlInt 4.562026
>
> Please note the big speedup in the overloaded sub method.
In this table, are bigger numbers good or bad? Which of MMD or PIR
corresponds to callmethod?
I would have assumed that sub mapped to MMD, and small (i.e., good)
numbers, and that callmethod mapped to PIR and big (i.e., bad) numbers.
Am I reading this wrong?
>>I *don't* see a need to heavily optimize for rarely used mechanisms.
>
> I'm not speaking of any optimization.
>
> Rarely or not doesn't really matter, if we currently just can't do an
> equivalent of:
>
> $ python
>
>>>>from cmath import sin
>>>>sin(1+2j)
>
> (3.1657785132161682+1.9596010414216058j)
>
> as long as "sin" is just a plain opcode internally. The compiler will
> just emit a "sin" opcode. But *internally* its something like:
>
> P_cmath_namespace."sin"(...)
I don't see a Python compiler ever emitting a "sin" opcode.
The code for pie-thon was based on the incorrect assumption that
builtins are both global and reserved. The truth is that all such
symbols are lexically scoped in Python.
Pirate currently emits the following code for "print sin(1+2j)":
.sub __main__ @MAIN
new_pad 0
loadlib P1, "python_group"
find_global P0, "PyBuiltin", "__load__"
invoke
push_eh __py_catch
#
find_type $I0, "PyObject" # (find_type:100)
new $P0, $I0 # (infixExpression:394)
find_type $I1, "PyInt" # (find_type:100)
new $P1, $I1 # (expressConstant:254)
$P1 = 1 # (expressConstant:255)
find_type $I2, "PyComplex" # (find_type:100)
new $P2, $I2 # (expressConstant:254)
$P2 = "2j" # (expressConstant:255)
$P0 = $P1 + $P2 # (infixExpression:404)
find_lex $P4, 'sin' # (lookupName:142)
$P3=$P4($P0) # (callingExpression:567)
print_item $P3 # (visitPrint:675)
print_newline # (visitPrintnl:680)
#
.return ()
__py_catch:
set S0, P5['_message']
print S0
print "\n"
.end
Larry Wall
: Sam Ruby wrote:
: >I know what a singleton is. I don't know what it means to "put the
: >such a thing.
: >
:
: I think that this is in order to supports Ruby's feature of being able
: add methods to individual objects.
:
: In the following code
:
: class A
: end
:
: a = A.new
: def a.method()
: end
:
: this creates an anomynous subclass of A and makes a an instance of the
: new class.
Perl 6 also does this if you compose a role into an object rather
than into a class. You can't currently compose a method directly
unless you use an anonymous role:
$a = new A;
$a does role { method mymethod() {...} };
(Though existing property methods may be mixed in with the "but" operator.
Such roles of such properties aren't anonymous, but since Perl can intuit
the role from the property name, they're effectively anonymous.)
Larry
Leopold Toetsch
> Leopold Toetsch wrote:
> Point of order: discussions would be a lot easier if they didnt *start
> out* with perjorative terms like "ugly hacks". One of my goals is to
> eliminate the need for if (!Interp_flags_TEST(INTERP,
> PARROT_PYTHON_MODE)) sprinkled throughout the various "standard" (i.e.
> Perl) PMCs.
Sorry about that. These hacks of course include pieces like
if (!Interp_flags_TEST(INTERP, PARROT_PYTHON_MODE))
and much more, which I have introduced. It was by no means intendend
against you.
> Yes, this all needs to be radically refactored - and all perlisms and
> pythonisms removed from the "standard" classes.
Yep.
> .... Furthermore, it is
> clear that operators like "sub_p_p_p" and "set_p_p_kic" will always
> perform better than callmethodcc_sc "__sub__" and callmethodcc_sc
> "__getitem__" respectively.
Well, and that's not true. I've already shown the opposite:
Here are agains these numbers (-O3 build, AMD 800, 5 Meg operations):
MMD add PerlInt PerlInt 0.693220
MMD add PerlInt INTVAL 0.609069
MMD sub PerlInt PerlInt 0.475912
MMD sub PerlInt INTVAL 0.490149
PIR add PerlInt PerlInt 8.312403
PIR sub PerlInt PerlInt 4.562026
Please note the big speedup in the overloaded sub method.
The "add" opcodes use the existing code, the "sub" has internal opcodes
similar to:
mmd_op_v_ppp .MMD_SUBTRACT, Pdest, Pleft, Pright
and
mmd_op_v_ppp_PASM .MMD_SUBTRACT, Pdest, Pleft, Pright
But the sourcecode contains just the normal "sub" opcode. The method
call syntax should just visualize, what's happening internally.
"__getitem__" aka set_p_k_ic shows a moderate slowdown of 10%.
> Therefore, Pirate emits the opcodes whenever direct use is made of
> things like infix operators and square brackets. And implemented
> methods like __sub__ which, while rarely used, will operate correctly by
> invoking to the opcodes.
Emitting opcodes is fine. Nothing will change on the surface. I'm
speaking about what the assembler does, if it encounters Px = Py[Pz]
> I *don't* see a need to heavily optimize for rarely used mechanisms.
I'm not speaking of any optimization.
Rarely or not doesn't really matter, if we currently just can't do an
equivalent of:
$ python
>>> from cmath import sin
>>> sin(1+2j)
(3.1657785132161682+1.9596010414216058j)
as long as "sin" is just a plain opcode internally. The compiler will
just emit a "sin" opcode. But *internally* its something like:
P_cmath_namespace."sin"(...)
Or have a look at pie-thon b3.py:
T = int
class TT(T):
...
class Int(TT):
...
TT.__cmp__ = T.__cmp__
By manipulating the compare method slot the derived class sorts
differently. If we don't do a method call internally, I can hardly
imagine, how to implement Python.
Just forget the speed argument. We optimize later.
> I encourage you to check out Pirate. The IMCC output of pirate.py is
> now remarkably close to the output of pie-thon.pl.
I'll do.
> - Sam Ruby
leo
Leopold Toetsch
> Leopold Toetsch wrote:
>> Well, and that's not true. I've already shown the opposite:
>> Here are agains these numbers (-O3 build, AMD 800, 5 Meg operations):
>>
>> MMD add PerlInt PerlInt 0.693220
>> MMD add PerlInt INTVAL 0.609069
>> MMD sub PerlInt PerlInt 0.475912
>> MMD sub PerlInt INTVAL 0.490149
>> PIR add PerlInt PerlInt 8.312403
>> PIR sub PerlInt PerlInt 4.562026
>>
>> Please note the big speedup in the overloaded sub method.
> In this table, are bigger numbers good or bad? Which of MMD or PIR
> corresponds to callmethod?
Sorry, forgot to mention - execution time in seconds.
> I would have assumed that sub mapped to MMD, and small (i.e., good)
> numbers, and that callmethod mapped to PIR and big (i.e., bad) numbers.
The "add" are current opcodes/functionality - either the plain MMD add
opcode or an overloaded function implemented in PIR.
The "sub" is the scheme with the PIC locally running here. The MMD "sub"
is not a full method call as it takes arguments. It's like
call_mmd "sub", P0, P1, P2
This does once a method lookup - it consults the MMD_table for the left
and right types and caches the function pointer / type pair.
If the return method PMC is a Sub the bytecode is rewritten to an opcode
that runs the PIR "sub" function.
Summary: the method call scheme with PIC is faster for MMD operations,
slightly slower for vtable functions and almost double speed for
overloaded method functions.
Below is the full code of the benchmark.
> Am I reading this wrong?
Partly, yes.
> I don't see a Python compiler ever emitting a "sin" opcode.
Yes that what I'm talking about. We need methods.
> The code for pie-thon was based on the incorrect assumption that
> builtins are both global and reserved.
Yep, that was broken. I knew it ;)
> ... The truth is that all such
> symbols are lexically scoped in Python.
Where AFAIK the topmost lexical pad corresponds to Python's "global".
> Pirate currently emits the following code for "print sin(1+2j)":
> .sub __main__ @MAIN
> find_lex $P4, 'sin' # (lookupName:142)
> $P3=$P4($P0) # (callingExpression:567)
Cool. But to make that work you need somewhere a "sin" method in a
namespace, i.e.
METHOD PMC* sin(PMC *num) {}
in Complex.pmc. And there is another "sin" method in another namespace
doing a sinus for FLOATVALS. During __load__ you are aliasing the "sin"
method as a lexical. This step is corresponding to the "import"
statement.
OTOH when you have
math.sin(2)
Python emits:
6 44 LOAD_NAME 0 (math)
47 LOAD_ATTR 1 (sin)
50 LOAD_CONST 1 (2)
53 CALL_FUNCTION 1
which actually is a method call on the "math" object. And the math
object is a namespace alias of e.g. the Float PMC.
Given that Python and Perl and other languages will just use this
method, it should be implemented in core PMCs (Complex, Float).
The same is true for all other stuff that is currently a vtable only
like "__getitem__" or just unimplemented, like all the trig functions
for PMCs.
leo
$ cat mmd-bench.imc
.sub _main prototyped
.const int max = 5000000
.const int val = 3
$P0 = new PerlInt
$P1 = new PerlInt
$P2 = new PerlInt
$P1 = 10
$P2 = val
.local float start
start = time
.local int i
i = 0
lp1:
$P0 = $P1 + $P2
inc i
if i < max goto lp1
$N0 = time
$N0 -= start
print "MMD add PerlInt PerlInt "
print $N0
print "\n"
$P0 = new PerlInt
$P1 = new PerlInt
$P1 = 10
$I2 = val
.local float start
start = time
.local int i
i = 0
lp2:
$P0 = $P1 + $I2
inc i
if i < max goto lp2
$N0 = time
$N0 -= start
print "MMD add PerlInt INTVAL "
print $N0
print "\n"
$P0 = new PerlInt
$P1 = new PerlInt
$P2 = new PerlInt
$P1 = 10
$P2 = val
.local float start
start = time
.local int i
i = 0
lp3:
$P0 = $P1 - $P2
inc i
if i < max goto lp3
$N0 = time
$N0 -= start
print "MMD sub PerlInt PerlInt "
print $N0
print "\n"
$P0 = new PerlInt
$P1 = new PerlInt
$P1 = 10
$I2 = val
.local float start
start = time
.local int i
i = 0
lp4:
$P0 = $P1 - $I2
inc i
if i < max goto lp4
$N0 = time
$N0 -= start
print "MMD sub PerlInt INTVAL "
print $N0
print "\n"
.include "mmd.pasm"
$P0 = new PerlInt
$P1 = new PerlInt
$P2 = new PerlInt
$P1 = 10
$P2 = val
find_global $P19, "PerlInt_add_pp"
mmdvtregister .MMD_ADD, .PerlInt, .PerlInt, $P19
.local float start
start = time
.local int i
i = 0
lp5:
$P0 = $P1 + $P2
inc i
if i < max goto lp5
$N0 = time
$N0 -= start
print "PIR add PerlInt PerlInt "
print $N0
print "\n"
$P0 = new PerlInt
$P1 = new PerlInt
$P2 = new PerlInt
$P1 = 10
$P2 = val
find_global $P19, "PerlInt_sub_pp"
mmdvtregister .MMD_SUBTRACT, .PerlInt, .PerlInt, $P19
.local float start
start = time
.local int i
i = 0
lp6:
$P0 = $P1 - $P2
inc i
if i < max goto lp6
$N0 = time
$N0 -= start
print "PIR sub PerlInt PerlInt "
print $N0
print "\n"
end
.end
.sub PerlInt_add_pp prototyped
.param pmc left
.param pmc right
.param pmc dest
$I16 = left
$I17 = right
$I18 = $I16 + $I17
dest = $I18
.end
.sub PerlInt_sub_pp prototyped
.param pmc left
.param pmc right
.param pmc dest
$I16 = left
$I17 = right
$I18 = $I16 - $I17
dest = $I18
.end
Leopold Toetsch
> Now, there is one big gotcha here -- multimethod dispatch. I'm not
> entirely sure that we can do this properly and still give classes
> full control over how methods are looked for and where they go.
Thinking about MMD a bit more, I can imagine the following scheme:
0) Our current MMD_table is 2-dimensional only and totally static.
Furthermore it doesn't really fit into method lookups as it's an
assymmetric second case.
1) Single method lookup.
Current state: works for objects and NCI class methods. Does not work
dynamically for vtable methods.
The switch-over from dynamic lookup with objects to static lookup in
PMCs is done with two helper (meta-)classes: delegate.pmc for objects
and deleg_pmc.pmc for objects derived from PMCs). This adds some extra
C<if> statements in find_method_direct and looks bulky.
Proposal:
We just do a method call for all method-like, "user" methods that
are currently handled in vtables. This means that we have internally:
array."__push"()
array."__get_integer_keyed"()
scalar."__get_string"()
and what not.
But also:
complex."imag"()
complex."real"()
io_pmc."eof"()
...
The existing opcode syntax remains of course, but the method call
syntax is an equivalent.
Please remember, when reading this that we are currently not discussing
performance, we want a complete and correct implementation. We can deal
with performance implications later.
The vtable entry can remain for internal (inside Parrot) use, if we know
which PMC type is used and for internal vtable methods like init, mark,
destroy, freeze, thaw, hash (!), and so on.
As there is no distinction between a method and what's currently a
vtable this expands easily to pythonic add-ons like PyInt."__oct__"
which isn't supported directly and of course to all object methods.
There is no distinction between object and vtable class methods (except
that some cases may be heavily optimized).
2) 2-dimensional MMD
This is basically just an abstraction of 1). A MMD method gets 2 entries
in the globals table, one for the left class namespace and one for the
right class. We have e.g.
the visible opcode
Pdest = Pleft + Pright
This gets an internal opcode:
callmethod_MMD_2 "add" # 2-dim MMD dispatch opcode
or a more specialized one with signature and arguments:
callmethod_MMD_2_v_ppp "add", Pdest, Pleft, Pright
to save on register setup overhead.
This opcode does:
a) look in Pleft's class for methods named "add", remember the tuple
(function, C<searchoffset>) for each found method
b) look in PRight's class and create such tuples, where the 2nd function
argument is PRight's class. To accomplish this the namepace that is
searched is e.g. "\0right_class\0__2".
c) compare the two lists for the best match (based on some distance e.g.
the combined depth (searchoffset) of the found functions)
d) cache the result (function, left-type, right-type)
e) call function
Done.
Further as the classes for left and right are responsible for the method
lookup, the left class can e.g. just ignore the MMD-search by returning
a function and a flag that no further search should be done.
During class setup existing MMD functions in *.pmc are created by a
sequence similar to:
store_global "class_name_left", "add", func
store_global "class_name_right", 2, "add", func
or
store_global "class_name_right\0__2", "add", func
3) n-dimensional MMD
This is a straight-forward extension of 2) and handled by
callmethod_MMD_3_sig "func" (maybe)
...
callmethod_MMD_n "func", n
Comments welcome,
leo
Leopold Toetsch
> ... We can deal
> with performance implications later.
I've googled for »MMD "polymorphic inline cache"«: 1-2 of about 3 on
perl6-internals archives. You might guess the author of these mail ;)
leo
Leopold Toetsch
> subclass - To create a subclass of a class object
Is existing and used.
> add_parent - To add a parent to the class this is invoked on
> become_parent - Called on the class passed as a parameter to add_parent
What is the latter used for?
> class_type - returns some unique ID or other so all classes in one
> class family have the same ID
What is a "class family"?
> instantiate - Called to create an object of the class
Exists.
> add_method - called to add a method to the class
> remove_method - called to remove a method from a class
These are the other two parts of the C<fetchmethod> vtable I presume. When
during packfile loading a Sub PMC constant is encountered and it's a
method, C<add_method> should be called instead of C<Parrot_store_global>?
> namespace_name - returns the name of this class' namespace
Should we really separate the namespace name from the class name?
> get_anonymous_subclass - to put the object into a singleton anonymous
> subclass
How is the singleton object created in the first place?
leo
Dan Sugalski
>Dan Sugalski <d...@sidhe.org> wrote:
>
>> subclass - To create a subclass of a class object
>
>Is existing and used.
Right. I was listing the things we need in the protocol. Some of them
we've got, some we don't, and some of the stuff we have we probably
need to toss out or redo.
> > add_parent - To add a parent to the class this is invoked on
>> become_parent - Called on the class passed as a parameter to add_parent
>
>What is the latter used for?
To give the newly added parent class a chance to do some setup in the
child class, if there's a need for it. There probably won't be in
most cases, but when mixing in classes of different families I think
we're going to need this.
> > class_type - returns some unique ID or other so all classes in one
>> class family have the same ID
>
>What is a "class family"?
The metaclass's class. I think. This is meant to be an identifier
that says what kind of class a class is, so code can make some
assumptions about internal structure and such. Everything that's
based on a ParrotClass PMC, for example, would have the same
class_type ID.
> > add_method - called to add a method to the class
> > remove_method - called to remove a method from a class
>
>These are the other two parts of the C<fetchmethod> vtable I presume. When
>during packfile loading a Sub PMC constant is encountered and it's a
>method, C<add_method> should be called instead of C<Parrot_store_global>?
Yeah, I think so. I'm not too happy about it, but I think that's the
way things will end up. Most classes will then go and stuff the
methods into the namespace, but I think it'll have to be up to the
classes whether they use the namespaces for methods or not.
> > namespace_name - returns the name of this class' namespace
>
>Should we really separate the namespace name from the class name?
Yes. We're going to have cases where we've got multiple classes with
the same name but different namespaces. (Generally classes
masquerading as other classes, but I can see some other cases)
> > get_anonymous_subclass - to put the object into a singleton anonymous
>> subclass
>
>How is the singleton object created in the first place?
For now, I think singletons will all be objects of a normal class
that get pulled into a singleton class, most likely because code's
added or changed methods on the object rather than to the class.
Piers Cawley
> At 11:13 AM +0100 12/14/04, Leopold Toetsch wrote:
>>Dan Sugalski <d...@sidhe.org> wrote:
>>
>>> subclass - To create a subclass of a class object
>>
>>Is existing and used.
>
> Right. I was listing the things we need in the protocol. Some of them
> we've got, some we don't, and some of the stuff we have we probably
> need to toss out or redo.
>
>> > add_parent - To add a parent to the class this is invoked on
>>> become_parent - Called on the class passed as a parameter to add_parent
>>
>>What is the latter used for?
>
> To give the newly added parent class a chance to do some setup in the
> child class, if there's a need for it. There probably won't be in
> most cases, but when mixing in classes of different families I think
> we're going to need this.
The chap who's writing Ruby on Rails, a very capable framework reckons
that some of these 'meta' method calls that Ruby has in abundance have
really made his life a lot easier; they're not the sort of things you
need to use very often, but they're fabulously useful when you do.