New-style classes and special methods
Raj B
My question is about how special methods are stored internally in
Python objects.
Consider a new-style class which implements special methods such as
__call__ and __new__
class C(type):
def __call__(...):
<body>
class B:
__metaclass__ = C
<stuff>
b= B()
The type of C is 'type', that of B is 'C'. When B is instantiated,
the __call__ method of C is first invoked, since C is the metaclass
for B.
Internally, when a Python callable object 'obj' is called, the actual
function called seems to be
'obj->ob_type->tp_call'.
Does this that somehow the '__call__' method defined in C above is
assigned to the 'tp_call' slot in the object representing the class
C, instead of it just being stored in the dictionary like a normal
attribute? Where and how does this magic happen exactly? I'd
appreciate any level of detail.
Thanks!
Raj
Alex Martelli
Yes, special methods populate the slots in the structures which Python
uses to represent types. Objects/typeobject.c in the Python source
distribution does the hard work, particularly in function type_new (line
1722 in my current SVN checkout).
If you're not comfortable reading C code you may want to try looking at
the "Python implemented in Python" project, pypy, or perhaps
alternatives such as Jython (in Java) or better IronPython (in C#), but
I am not familiar in detail with how they deal with the issue.
Alex
Raj B
Python
> uses to represent types. Objects/typeobject.c in the Python source
> distribution does the hard work, particularly in function type_new
Thanks for that quick response. I am quite comfortable with C code
and am trying to understand exactly what happens when a new-style
class is created, and then instantiated.
I have been reading typeobject.c and type_new() inside it in detail,
and there are a few issues I am trying to figure out.
I can see a lot of *SLOT() macros in the file that seem to set the
slots to appropriate values. What I am having trouble figuring out is
the connection i.e. at what point during construction of the class
object in type_new() are those slots allotted? Is it the tp_alloc()
function which does this?
Is there some kind of descriptor or other mechanism connecting
special method names with their slots in the object representation?
(e.g. "__call__" with type->tp_call)
Also, what happens when a class inherits from multiple classes with
their own __call__ methods? Where and how is it decided which
__call__ goes into the tp_call slot?
I'm sure I'll eventually figure it out if I stare at the code hard
enough, but would totally appreciate any help I can get :)
Thanks again!
Raj
Alex Martelli
> > Yes, special methods populate the slots in the structures which
> Python
> > uses to represent types. Objects/typeobject.c in the Python source
> > distribution does the hard work, particularly in function type_new
>
>
>
> Thanks for that quick response. I am quite comfortable with C code
> and am trying to understand exactly what happens when a new-style
> class is created, and then instantiated.
>
> I have been reading typeobject.c and type_new() inside it in detail,
> and there are a few issues I am trying to figure out.
>
> I can see a lot of *SLOT() macros in the file that seem to set the
> slots to appropriate values. What I am having trouble figuring out is
> the connection i.e. at what point during construction of the class
> object in type_new() are those slots allotted? Is it the tp_alloc()
> function which does this?
I believe there are different times -- one for the fundamental struct
that represents all types, others for secondary structs that represent
e.g. numerical/arithmetic methods, sequence methods, etc, each of which
is present only if necessary for a given type.
> Is there some kind of descriptor or other mechanism connecting
> special method names with their slots in the object representation?
> (e.g. "__call__" with type->tp_call)
The online docs for the C/API interface do some effort at explaining the
type-struct and auxiliary ones, and I spent a couple of pages on that
same subject in "Python in a Nutshell", though nowhere close to doing it
justice (such docs normally address people who want to write C-coded
extensions, rather than ones whose goal is to understand the existing
Python runtime -- though many of the issues are identical).
> Also, what happens when a class inherits from multiple classes with
> their own __call__ methods? Where and how is it decided which
> __call__ goes into the tp_call slot?
The MRO is used (that stands for Method Resolution Order, and is also
implemented in the same C file, and documented there AND in a strong
essay by M. Simionato whose URL is, I believe, also in a comment in that
file) to find the relevant implementation.
> I'm sure I'll eventually figure it out if I stare at the code hard
> enough, but would totally appreciate any help I can get :)
>
> Thanks again!
You're welcome, but I hope somebody else will also step up to offer
useful comments, because (what between my talk at Google Developer's Day
tomorrow, and traveling next week to speak in Krakow and then at the
first Italian conference on Python in Florence) I'm rather overwhelmed
these days;-).
Alex
Lenard Lindstrom
> > Yes, special methods populate the slots in the structures which Python
> > uses to represent types. Objects/typeobject.c in the Python source
> > distribution does the hard work, particularly in function type_new
>
>
>
> Thanks for that quick response. I am quite comfortable with C code and
> am trying to understand exactly what happens when a new-style class is
> created, and then instantiated.
>
> I have been reading typeobject.c and type_new() inside it in detail, and
> there are a few issues I am trying to figure out.
>
> I can see a lot of *SLOT() macros in the file that seem to set the slots
> to appropriate values. What I am having trouble figuring out is the
> connection i.e. at what point during construction of the class object in
> type_new() are those slots allotted? Is it the tp_alloc() function which
> does this?
The place to start is the PyType_Type tp_new slot function type_new().
The second to last statement is a call to fixup_slot_dispatchers(). This
function goes through the dictionary looking for special methods and
adds the appropriate slot functions. This gets very involved. I had to
use the Visual Studio debugger to follow what was happening when trying
to figure out what happens when assigning a special method to a class
after it is declared.
>
> Is there some kind of descriptor or other mechanism connecting special
> method names with their slots in the object representation? (e.g.
> "__call__" with type->tp_call)
There is a special tp_call slot function, slot_tp_call(), that calls the
user defined __call__. The same goes for other special methods.
Descriptors only come into play with extension types. In this case if a
slot function is found a descriptor is added to make the slot function
accessible from Python as a special method.
>
> Also, what happens when a class inherits from multiple classes with
> their own __call__ methods? Where and how is it decided which __call__
> goes into the tp_call slot?
>
As Alex Martelli mentioned, __call__ is found using the method
resolution order. The tp_call slot function slot_tp_call() uses
lookup_method(), a variation of PyObject_GetAttribute(), to finding the
appropriate Python method. Its all documented in the C file.
> I'm sure I'll eventually figure it out if I stare at the code hard
> enough, but would totally appreciate any help I can get :)
>
Just ask.
--
Lenard Lindstrom
<le...@telus.net>