Understanding descriptors
Brian Allen Vanderburg II
still after reading some sites. Here is what I 'think', but please let
me know if any of this is wrong as I'm sure it probably is.
First when accessing an attribute on a class or instance it must be
found. For an instance, it's __dict__ is first search. If not found
the class and base class __dict__ are searched. For a class, the
__dict__ and base classes __dict__ are search.
If assigning, and the attribute is found and is not a descriptor or if
the attribute is not found, then the assignment will occur in the
__dict__ of the class or instance. If it is found and is a descriptor,
then __set__ will be call.
For reading, if the attribute is found and is a descriptor, __get__ will
be called, passing the object (if it is an instance) and class. If it
is not a descriptor, the attribute will be returned directly.
Class methods are just functions:
class C(object):
def F(self):
pass
C.__dict__['F'] # function object ...
But functions are descriptors:
C.__dict__['F'].__get__ # method wrapper ...
def f1():
pass
f1.__get__ # method wrapper ...
When a lookup is done it uses this descriptor to make a bound or unbound
method:
c=C()
C.F # unbound method object, expects explicit instance when calling the
function
c.F # bound method object provides instance implicitly when calling the
function
This is also done when adding to the classes:
C.f1 = f1
f1 # function
C.f1 # unbound method
c.f1 # bound method
To prevent this it has to be decorated so the descriptor doesn't cause
the binding:
C.f2 = staticmethod(f1)
C.f2 # functon
c.f2 # function
Here is a question, why don't instance attributes do the same thing?
c.f3 = f1
c.f3 # function, not bound method
So it is not calling the __get__ method for c.f3 After it finds c.f3 in
c.__dict__, and since it has a getter, shouldn't it call the __get__ to
return the bound method. It is good that it doesn't I know, but I just
want to know why it doesn't from an implementation view.
Brian Vanderburg II
Aahz
Brian Allen Vanderburg II <BrianVan...@aim.com> wrote:
>
> [...]
>
>When a lookup is done it uses this descriptor to make a bound or unbound
>method:
>
>c=C()
>
>C.F # unbound method object, expects explicit instance when calling the
>function
>c.F # bound method object provides instance implicitly when calling the
>function
>
>This is also done when adding to the classes:
>
>C.f1 = f1
>
>f1 # function
>C.f1 # unbound method
>c.f1 # bound method
>
>To prevent this it has to be decorated so the descriptor doesn't cause
>the binding:
>
>C.f2 = staticmethod(f1)
>C.f2 # functon
>c.f2 # function
>
>Here is a question, why don't instance attributes do the same thing?
>
>c.f3 = f1
>c.f3 # function, not bound method
>
>So it is not calling the __get__ method for c.f3 After it finds c.f3 in
>c.__dict__, and since it has a getter, shouldn't it call the __get__ to
>return the bound method. It is good that it doesn't I know, but I just
>want to know why it doesn't from an implementation view.
This is a bit beyond my expertise, but nobody else has responded, so I'll
take a stab at it:
Basically, this is part of the design philosphy that keeps everything
working the same way. When an attribute is found on an instance, it
gets returned unmodified. Period. In other words, the fact that f3
happens to return a function object gets ignored. You can force f3 to
work like a bound method by defining it as one:
def f3(self):
pass
If you use a new-style class or Python 3.0, the same design is evident
when you try to use e.g. str() to call the __str__() special method: the
instance gets ignored for looking up the method.
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
Weinberg's Second Law: If builders built buildings the way programmers wrote
programs, then the first woodpecker that came along would destroy civilization.
Bruno Desthuilliers
> I'm trying to better understand descriptors and I've got a few questions
> still after reading some sites. Here is what I 'think', but please let
> me know if any of this is wrong as I'm sure it probably is.
>
>
> First when accessing an attribute on a class or instance it must be
> found. For an instance, it's __dict__ is first search.
Actually, this is not true - binding descriptors are searched (in the
class and it's mro) _before_ the instance's __dict__ :
>>> class Foo(object):
... @apply
... def bar():
... def fget(self):
... return self._bar
... def fset(self, v):
... self._bar = v
... return property(**locals())
...
>>> f = Foo()
>>> f.bar = 42
>>> f.bar
42
>>> f.__dict__['bar'] = "boo"
>>> f.bar
42
>>> f.__dict__['bar']
'boo'
>>>
Note that this works a bit differently for non-binding descriptors
(descriptors that only implement the __get__ method), which are looked
up after the instance's __dict__
So the lookup chain is:
1/ lookup the class and bases for a binding descriptor
2/ then lookup the instance's __dict__
3/ then lookup the class and bases for a non-binding descriptor or plain
attribute
4/ then class __getattr__
Also and FWIW, there's a "step zero" : calls __getattribute__. All the
above lookup mechanism is actually implemented by object.__getattribute__.
> If not found
> the class and base class __dict__ are searched. For a class, the
> __dict__ and base classes __dict__ are search.
>
> If assigning, and the attribute is found and is not a descriptor
a binding descriptor
> or if
> the attribute is not found, then the assignment will occur in the
> __dict__ of the class or instance. If it is found and is a descriptor,
a binding descriptor
> then __set__ will be call.
> For reading, if the attribute is found and is a descriptor, __get__ will
> be called, passing the object (if it is an instance) and class. If it
> is not a descriptor, the attribute will be returned directly.
>
> Class methods are just functions:
In Python, "classmethod" has a definite meaning - it's a method that
takes the class (and not the instance) as first argument. So your
assertion should be "Methods are just functions which are attributes of
the class". Which is still not quite true. Sure, the def statement
always create function objects, regardless of whether it happens within
a class statement or not. OTHO, what you get when looking up the
corresponding attribute is definitly a method object. So the correct
formulation here is (IMHO of course) that methods are implemented by
functions that are attributes of the class. Also, this is not restricted
to functions : any callable object correctly implementing the descriptor
protocol can be used (cf the classmethod and staticmethod objects).
> class C(object):
> def F(self):
> pass
>
> C.__dict__['F'] # function object ...
yes.
> But functions are descriptors:
yes.
> C.__dict__['F'].__get__ # method wrapper ...
>
> def f1():
> pass
>
> f1.__get__ # method wrapper ...
>
> When a lookup is done it uses this descriptor to make a bound or unbound
> method:
yes.
> c=C()
>
> C.F # unbound method object, expects explicit instance when calling the
> function
> c.F # bound method object provides instance implicitly when calling the
> function
>
> This is also done when adding to the classes:
Indeed. Whether the def statement happens within the class statement or
not is irrelevant here.
> C.f1 = f1
>
> f1 # function
> C.f1 # unbound method
> c.f1 # bound method
>
> To prevent this it has to be decorated so the descriptor doesn't cause
> the binding:
>
> C.f2 = staticmethod(f1)
> C.f2 # functon
> c.f2 # function
yes.
> Here is a question, why don't instance attributes do the same thing?
Because the descriptor protocol is only invoked on class attributes.
> c.f3 = f1
> c.f3 # function, not bound method
>
> So it is not calling the __get__ method for c.f3 After it finds c.f3 in
> c.__dict__, and since it has a getter, shouldn't it call the __get__ to
> return the bound method.
Nope, cf above.
> It is good that it doesn't I know, but I just
> want to know why it doesn't from an implementation view.
If by "from an implementation view", you mean "how it works", then the
answer is above. If your question was about "why this design choice",
I'll leave the answer to someone else...
Brian Allen Vanderburg II
>
> So the lookup chain is:
>
> 1/ lookup the class and bases for a binding descriptor
> 2/ then lookup the instance's __dict__
> 3/ then lookup the class and bases for a non-binding descriptor or
> plain attribute
> 4/ then class __getattr__
>
> Also and FWIW, there's a "step zero" : calls __getattribute__. All the
> above lookup mechanism is actually implemented by
> object.__getattribute__.
Okay, so instance attributes never use their __get__/__set__/etc when
looking up.
A binding descriptor is one that has a __set__ (even if it doesn't do
anything) and it takes priority over instance variables. Properties are
binding descriptors even if they don't have a set function specified. A
non-binding descriptor doesn't have __set__ and instance variables take
priority over them.
For reading:
1. Lookup in the class/bases for a binding descriptor and if found use
its __get__
2. If instance, look up in instance __dict__ and if found return it
3. Lookup in the class/bases
a. if found and a descriptor use it's __get__
b. if found and not a descriptor return it
4. Use __getattr__ (if instance?)
For writing:
1. If instance
a. lookup in the class/bases for a binding descriptor and if found
use its __set__
b. write to instance __dict__
2. If class, write in class __dict__
I think I understand it now. Thanks.
Brian Vanderburg II