[Python-ideas] A user story concerning things knowing their own names
Greg Ewing
and pasting an overridable_property and forgetting to
change the passed-in name:
content_size = overridable_property('size',
"Size of the content area.")
which should have been
content_size = overridable_property('content_size',
"Size of the content area.")
This was quite difficult to track down, because the result
was to effectively make it an alias of *another* property
I have called 'size'. They happen to be near enough to the same
thing that the error went unnoticed for quite some time.
If I could write my overridable_property declarations
without having to repeat the name, this kind of thing would
not be able to happen.
--
Greg
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas
Westley Martínez
> I just experienced an obscure bug resulting from copying
> and pasting an overridable_property and forgetting to
> change the passed-in name:
>
> content_size = overridable_property('size',
> "Size of the content area.")
>
> which should have been
>
> content_size = overridable_property('content_size',
> "Size of the content area.")
>
> This was quite difficult to track down, because the result
> was to effectively make it an alias of *another* property
> I have called 'size'. They happen to be near enough to the same
> thing that the error went unnoticed for quite some time.
>
> If I could write my overridable_property declarations
> without having to repeat the name, this kind of thing would
> not be able to happen.
>
If it went unnoticed, what's the matter?
Nick Coghlan
> If it went unnoticed, what's the matter?
Unnoticed just means it was a silent failure rather than a noisy one.
It doesn't mean the bug wasn't resulting erroneous output.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Larry Hastings
On 03/16/2011 05:28 AM, Greg Ewing wrote:
If I could write my overridable_property declarations
without having to repeat the name, this kind of thing would
not be able to happen.
def assign(fn):
return fn(fn.__name__)
@assign
def content_size(name):
return overridable_property(name, "Size of the content area.")
How bad do you want it? ;-)
/larry/
Michael Foord
How bad do you want it? ;-)
/larry/
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas
--
http://www.voidspace.org.uk/
May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
Ian Bicking
class Table(ORM):
name = StringColumn()
Another case where I've noticed a problem is any kind of descriptor that needs its own storage; the name of the property gives a possible stable namespace for the value, but without the name you either have to pass in a name or the storage area becomes volatile. For instance, a read-only descriptor: http://svn.colorstudy.com/home/ianb/recipes/setonce.py
You can solve this in, e.g., the ORM class by doing things when a class is created -- but it requires very specific cooperation between the class and the descriptor. Everyone really does it different ways.
One could imagine an extension of the descriptor protocol, where on class creation you called something like attr.__addtoclass__(cls, name) (for all attributes of the class that define that method) -- which if you just want the name you'd simply save that name in your object and return self.
Robert Kern
> I'll note this general problem is also present in any of the declarative ORMs,
> which use silly hacks to tell descriptors their name. Like you have:
>
> class Table(ORM):
> name = StringColumn()
>
> Another case where I've noticed a problem is any kind of descriptor that needs
> its own storage; the name of the property gives a possible stable namespace for
> the value, but without the name you either have to pass in a name or the storage
> area becomes volatile. For instance, a read-only descriptor:
> http://svn.colorstudy.com/home/ianb/recipes/setonce.py
>
> You can solve this in, e.g., the ORM class by doing things when a class is
> created -- but it requires very specific cooperation between the class and the
> descriptor. Everyone really does it different ways.
>
> One could imagine an extension of the descriptor protocol, where on class
> creation you called something like attr.__addtoclass__(cls, name) (for all
> attributes of the class that define that method) -- which if you just want the
> name you'd simply save that name in your object and return self.
If we were to extend the descriptor protocol, I think it would be better to
extend it by providing __get_ex__(self, instance, owner, name), __set_ex__(self,
instance, name, value), etc. methods that would be called in preference over the
current __get__() and __set__() methods. This allows descriptor objects to be
reused.
We do this reasonably often with the Traits package (which overrides
__getattribute__() to essentially implement this __get_ex__()/__set_ex__()
protocol by different names). For example, we have a complicated trait for
specifying colors:
ColorTrait = Trait("black", Tuple, List, Str, color_table)
It would be nice to simply reuse this object everywhere:
class Figure(HasTraits):
background = ColorTrait
foreground = ColorTrait
The alternative is to only provide factories such that you always get a new
descriptor:
ColorTrait = lambda: Trait("black", Tuple, List, Str, color_table)
class Figure(HasTraits):
background = ColorTrait()
foreground = ColorTrait()
However, this makes these new descriptors inconsistent with current descriptors,
which can be shared between classes. It also prevents you from passing around
descriptor objects as first-class citizens. For example, I might want to ask the
ColorTrait for its constituents in order to construct a new one that has a
different default value. I can't ask that of a factory function.
This objection holds for any proposal that requires the descriptor object itself
to hold onto its name, no matter how it acquires that name. Personally, I don't
want descriptors that know their own name; I want descriptors that can be told
what name to use for each operation.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
Terry Reedy
>
> On 16 March 2011 11:10, Larry Hastings <la...@hastings.org
> morning, you could achieve this in current Python, no extension needed:
>
> def assign(fn):
> return fn(fn.__name__)
>
> @assign
> def content_size(name):
> return overridable_property(name, "Size of the content area.")
> And building on this sightly you could do the following for namedtuple:
> >>> from collections import namedtuple
> >>> import inspect
> >>> def make_namedtuple(fn):
> ... args = ' '.join(inspect.getargspec(fn).args)
> ... return namedtuple(fn.__name__, args)
> ...
> >>> @make_namedtuple
> ... def Point(x, y): pass
> ...
> >>> p = Point(1, 2)
> >>> p
> Point(x=1, y=2)
If make_namedtuple were added to collections, then one could import
*that* instead of 'namedtuple' itself.
--
Terry Jan Reedy
Ian Bicking
As I was thinking of __addtoclass__ it would address this, though at class instantiation time instead of attribute access time. Specifically it would be like there was a fixup stage that would look like:
def fixup(cls):
for class_instance in cls.__mro__:
for name, value in class_instance.__dict__.items():
method = getattr(value, '__addtoclass__', None)
if method is not None:
new_value = method(cls, name)
if new_value is not value:
setattr(cls, name, new_value)
If ColorTrait returns a new instance when __addtoclass__ is called, then it can, and all of its instances will be class-specific (and not shared with subclasses).
Ian
Masklinn
On 2011-03-16, at 21:49 , Terry Reedy wrote:
> On 3/16/2011 12:11 PM, Michael Foord wrote:
>>
>> On 16 March 2011 11:10, Larry Hastings <la...@hastings.org
>> As I suggested in my email on the Assignment Decorators thread this
>> morning, you could achieve this in current Python, no extension needed:
>>
>> def assign(fn):
>> return fn(fn.__name__)
>>
>> @assign
>> def content_size(name):
>> return overridable_property(name, "Size of the content area.")
>
>> And building on this sightly you could do the following for namedtuple:
>> >>> from collections import namedtuple
>> >>> import inspect
>> >>> def make_namedtuple(fn):
>> ... args = ' '.join(inspect.getargspec(fn).args)
>> ... return namedtuple(fn.__name__, args)
>> ...
>> >>> @make_namedtuple
>> ... def Point(x, y): pass
>> ...
>> >>> p = Point(1, 2)
>> >>> p
>> Point(x=1, y=2)
>
> If make_namedtuple were added to collections, then one could import *that* instead of 'namedtuple' itself.
Since `namedtuple` is a function, wouldn't it be possible to add this feature to the existing namedtuple, so that the number of names doing just about the same thing in slightly different ways doesn't explode? (it's just a different way of providing the exact same arguments after all)
Raymond Hettinger
On Mar 16, 2011, at 1:49 PM, Terry Reedy wrote:
> If make_namedtuple were added to collections, then one could import *that* instead of 'namedtuple' itself.
Post a recipe somewhere.
My bet is that it will have a near zero adoption rate.
-1 on all of these random ideas to solve what is basically a non-problem.
The name of a named tuple, or a class, or a function, or a variable
typically is used many times, not just in the definition.
If you make up a new factory function that uses inspect
magic to guess user's intend target name, you will have
introduced unnecessary complexity and will likely
introduce unexpected behaviors and bugs.
Raymond
Greg Ewing
> If it went unnoticed, what's the matter?
It only went unnoticed because until recently I hadn't
run any code that depended critically on it being right.
A bug is still a bug even if you haven't found it yet!
When it did became apparent that something was wrong,
it still took quite a long time to track down the cause.
I was looking for a bug in the get_content_size()
method, whereas it was actually calling a different
method altogether. Having attribute accesses go astray
like that is not something you expect.
--
Greg
Greg Ewing
> def assign(fn):
> return fn(fn.__name__)
>
> @assign
> def content_size(name):
> return overridable_property(name, "Size of the content area.")
>
> How bad do you want it? ;-)
Not quite badly enough to replace all my existing
overridable_property declarations with something that
ugly. :-) But I appreciate the suggestion, and I'll
keep it in mind next time I'm designing anything
similar.
Greg Ewing
> Personally, I don't want descriptors that know their own name; I
> want descriptors that can be told what name to use for each operation.
This is a somewhat different use case from mine. For
overridable_property, I *do* want the name to be known
when the descriptor is created, so that it can
precompute some things based on it.
--
Greg
Michael Foord
Larry Hastings wrote:Not quite badly enough to replace all my existing
def assign(fn):
return fn(fn.__name__)
@assign
def content_size(name):
return overridable_property(name, "Size of the content area.")
How bad do you want it? ;-)
overridable_property declarations with something that
ugly. :-) But I appreciate the suggestion, and I'll
keep it in mind next time I'm designing anything
similar.
def content_size():
return "Size of the content area."
--
Greg
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas
Terry Reedy
I am not sure how serious I was when I wrote that, but after further
thought, I retract it as a serious suggestion, if indeed it was ;-).
I think both decorators are 'cute' as examples of what one can do with
decorators, and possible cookbook recipes, but not something we really
need in the stdlib. I mostly said something because they do not require
new syntax and I do not like any of the new syntax proposals.
There are many aspects of the rationale for decorators that do not apply
to the assignment case.
1. Explicit function wrapping requires the name in tripicate rather than
duplicate. Moreover, some interface wrappers for functions need fairly
long, multi-component names. I am pretty sure such need is much less for
named tuples and properties.
2. The triplicates are *not* on the same line but may be separated by an
arbitrary number of lines. This not only impedes checking that all three
are the same, but may also impede understanding of the function code.
That sometimes really requires knowing how or what the function will be
wrapped as. This is greatly aided by bringing the wrapping function back
up to (above) the def line.
For instance, @classmethod and @staticmethod explain the non-standard
signature with 'cls' or '' instead of 'self'.
For another example, twisted.internet.defer.inLineCallbacks is a
decorator that wraps a generator function as a
twisted.internet.defer.Deferred object. Knowing that a particular
generator function implements a series of data/error callbacks tor
twisted greatly helps in understanding it. The idea is 'twisted' enough
as it is ;-).
Guido van Rossum
> I just experienced an obscure bug resulting from copying
> and pasting an overridable_property and forgetting to
> change the passed-in name:
>
> content_size = overridable_property('size',
> "Size of the content area.")
>
> which should have been
>
> content_size = overridable_property('content_size',
> "Size of the content area.")
>
> This was quite difficult to track down, because the result
> was to effectively make it an alias of *another* property
> I have called 'size'. They happen to be near enough to the same
> thing that the error went unnoticed for quite some time.
>
> If I could write my overridable_property declarations
> without having to repeat the name, this kind of thing would
> not be able to happen.
I appreciate the user story. I also appreciate what Ian said: that
this is often solved using a metaclass (in fact it is now a very
common pattern) but that everybody does it somewhat differently. And I
appreciate that Greg's use case is not solved by a metaclass (even if
we turned the pattern into a metaclass or class decorator in the
stdlib, which might be a good idea regardless).
At the same time, it seems that there aren't a lot of specific
examples besides namedtuple (which seems to cause lots of emotions and
is I think best left alone) and Greg's overridable_property. So,
unless we can come up with a really nice way (either syntactical or
perhaps through a magic builtin) to give functions like
overridable_property() access to the LHS name, and find more use
cases, I don't see this happening.
I really don't like "assignment decorators" (which aren't the same
thing at all as class or function decorators, no matter how much I
squint) nor most other solutions (e.g. ":=" -- too subtle, and might
well mean something else). But I'm not precluding that someone will
come up with a better solution. In the mean time, as long as it's just
one use case I still like spelling it using a function decorator:
@overridable_property
def content_size(): "Size of the content are"
The overridable_property can then access the __name__ and __doc__
attributes of the function passed into it, assert that it has no
arguments using the inspect module, and return an appropriate instance
of the OverridableProperty class. Voilà, decorated assignment. :-)
PS. Greg: is your current overridable_property part of some open
source code that you've published yet? Searches for it mostly seem to
turn up recent discussions here...
--
--Guido van Rossum (python.org/~guido)
Guido van Rossum
On Thu, Mar 17, 2011 at 10:12 PM, Ian Bicking <ia...@colorstudy.com> wrote:
> Did you have any opinion on __addtoclass__? It seemed to me to address most
> of the use cases (even Greg's as far as I understood it).
[Reminder: Ian's proposal is to call attr.__addtoclass__(cls, name)
for all attributes of the class that have the method, at class
construction time.]
You know, somehow I missed that when skimming. :-( It nicely
encapsulates the main reason for having a metaclass in many of these
cases.
There's another pattern where all class attributes that have a certain
property are also collected in a per-class datastructure, and at that
point we're back to the custom metaclass; but standardizing on
__addtoclass__ with the proposed signature would be a first step.
We could either have a standard metaclass that does this, or we could
just make it part of 'type' (the default metaclass) and be done with
it.
Robert Kern
> At the same time, it seems that there aren't a lot of specific
> examples besides namedtuple (which seems to cause lots of emotions and
> is I think best left alone) and Greg's overridable_property. So,
> unless we can come up with a really nice way (either syntactical or
> perhaps through a magic builtin) to give functions like
> overridable_property() access to the LHS name, and find more use
> cases, I don't see this happening.
A sizable portion of Traits needs the name information.
http://pypi.python.org/pypi/Traits
As I explained elsewhere, we currently implement this by overriding
__getattribute__ to implement a descriptor-like protocol that passes along the
name to the trait object rather than storing it on the trait object itself, but
we could re-engineer Traits make do with Ian's __addtoclass__ proposal. Being
able to use plain descriptors would allow us to rewrite Traits to avoid our
current C-implemented base class, which would let us interoperate with other
frameworks better.
I can list all of the specific features of Traits that makes use of this
information if you like.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
_______________________________________________
Greg Ewing
> PS. Greg: is your current overridable_property part of some open
> source code that you've published yet?
Yes, I'm using it in two projects at the moment:
PyGUI: http://www.cosc.canterbury.ac.nz/greg.ewing/python_gui/
Albow: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Albow/
> Searches for it mostly seem to turn up recent discussions here...
You probably won't find it with a direct search because
it's more of an internal detail of those libraries rather
than a separately-advertised feature.
--
Greg
Greg Ewing
> There's another pattern where all class attributes that have a certain
> property are also collected in a per-class datastructure,
I think __addtoclass__ could cover those as well,
if you can arrange for the relevant objects to inherit
from a class having an appropriate __addtoclass__
implementation.
If that's not convenient, another approach would be to
wrap them in something whose __addtoclass__ does the
right thing and then unwraps itself.
--
Greg
Greg Ewing
> We could either have a standard metaclass that does this, or we could
> just make it part of 'type' (the default metaclass) and be done with
> it.
If it's a metaclass, standard or not, we're back to the
same difficulties of mixing different metaclasses together.
--
Greg
Jim Jewett
> Guido van Rossum wrote:
>> There's another pattern where all class attributes that have a certain
>> property are also collected in a per-class datastructure,
> I think __addtoclass__ could cover those as well,
> if you can arrange for the relevant objects to inherit
> from a class having an appropriate __addtoclass__
> implementation.
How do you put an attribute (such as __addtoclass__ ) on a name? Or
are you proposing that the actual pattern be something more like:
x=SpecialObj()
And that normal initiation be handled either later, or as part of the
SpecialObj initiation
x=SpecialObj()=5
or
x=SpecialObj(); x=5
or
x=SpecialObj(value=5)
Doing this for every name seems likely to be wasteful. Doing it only
for certain initial values seems too magical.
Doing it only for certain attributes -- there still needs to be a way
to mark them, and I suppose we're back to either a decorator or a
special assignment operator.
@decorated_implies_an_object
x=5
x:=5
What have I missed here?
-jJ
Ian Bicking
On Fri, Mar 18, 2011 at 9:37 PM, Greg Ewing <greg....@canterbury.ac.nz> wrote:How do you put an attribute (such as __addtoclass__ ) on a name? Or
> Guido van Rossum wrote:
>> There's another pattern where all class attributes that have a certain
>> property are also collected in a per-class datastructure,
> I think __addtoclass__ could cover those as well,
> if you can arrange for the relevant objects to inherit
> from a class having an appropriate __addtoclass__
> implementation.
are you proposing that the actual pattern be something more like:
x=SpecialObj()
And that normal initiation be handled either later, or as part of the
SpecialObj initiation
x=SpecialObj()=5
or
x=SpecialObj(); x=5
or
x=SpecialObj(value=5)
What we're describing only applies to class variables; a top-level variable wouldn't be affected.
Imagine for instance a column class (ORMish) that wants to know its name:
class Column(object):
def __init__(self, **kw):
...
def __addtoclass__(self, name, cls):
self.name = name
return self
Now if you do:
class MyTable:
username = Column()
Then MyTable.username.name == 'username'
If you wanted to be able to reuse values, like Greg wants, you could do:
class Column(object):
def __init__(self, kw):
self.kw = kw
...
def copy(self):
return self.__class__(self.kw)
def __addtoclass__(self, name, cls):
new_obj = self.copy()
new_obj.name = name
return new_obj
Or you could use several different classes (e.g., BoundColumn), or... well, there's many ways to skin a cat.
Like descriptors, only objects that implement this new method would participate. It's not really like a decorator, it's much more like descriptors -- decorators like classmethod just happen to be descriptor factories.
Ian
Greg Ewing
> If you wanted to be able to reuse values, like Greg wants,
Actually I don't want to reuse values, that was someone
else. For my use case it's fine to create a new descriptor
for each use.
--
Greg
Guido van Rossum
> Actually I don't want to reuse values, that was someone
> else. For my use case it's fine to create a new descriptor
> for each use.
So, apologies if this has been brought up or rejected before, wouldn't
a class decorator work for you? That is totally capable of calling
x.__addtoclass__() (or whatever you want to call it -- it's now
between the decorator and overridable_property) for each class
attribute (really: for each value in the class __dict__) that has it,
and doesn't seem to have the problems with combining unrelated
metaclasses that you brought up: unrelated class decorators combine
just fine (especially ones like this that mutate the class but still
return the original class object).
--
--Guido van Rossum (python.org/~guido)
Greg Ewing
> So, apologies if this has been brought up or rejected before, wouldn't
> a class decorator work for you?
It would work, although it would be a bit less than satisfying,
because the property wouldn't be fully self-contained. Some of
the plumbing would still be showing, albeit less obtrusively.
--
Greg
Ian Bicking
Guido van Rossum wrote:It would work, although it would be a bit less than satisfying,
So, apologies if this has been brought up or rejected before, wouldn't
a class decorator work for you?
because the property wouldn't be fully self-contained. Some of
the plumbing would still be showing, albeit less obtrusively.
If you forget the decorator (easy to do) the errors could be lots of ugly "<PropertyThatMustKnowName at 0x4928394> object has no attribute 'name'" -- and you could make the error slightly better, but not much because the PropertyThatMustKnowName doesn't get a chance to validate itself (since you didn't use the decorator and it can't really know that).
Ian
Guido van Rossum
It would be easy enough to record the filename and line where the
constructor was called, and report those in the error message.
All in all it does sound like it could be an improvement over having
to pass the name in redundantly, and it has the advantage that it
works today.
--
--Guido van Rossum (python.org/~guido)