[Python-ideas] Different interface for namedtuple?
Carl M. Johnson
assignments and whatnot, and while I do think it would be interesting
if there were some way to abstract what happens in the creation of
classes and defs into a more generalizable structure, for the
immediate use case of NamedTuple, wouldn't an interface something like
this solve the DRY problem? --
class Point(NewNamedTuple): x, y
This interface couldn't have worked in Python 2.6 when namedtuple was
introduced, but using the __prepare__ statement in Python 3, it's
trivial to implement. As a bonus, NewNamedTuple could also support
type checking with the following syntax:
class Point(NewNamedTuple): x, y = int, int #or whatever the ABC is
for int-like numbers
In fact, since as it stands namedtuple is called "namedtuple" in
lowercase, we could just camel case "NamedTuple" to collections
instead of using the NewNamedTuple or whatever. Or, and I haven't done
any research on this, it may also be possible to implement
NewNamedTuple with backwards compatibility so it can also be used in
the traditional way.
-- Carl Johnson
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas
Carl M. Johnson
checking but with support for the kwargs of namedtuple:
>>> from collections import namedtuple
>>>
>>> # The custom dictionary
... class member_table:
... def __init__(self):
... self.member_names = []
...
... def __setitem__(self, key, value):
... self.member_names.append(key)
...
... def __getitem__(self, key):
... self.member_names.append(key)
...
>>> # The metaclass
... class TupleNamer(type):
... @classmethod
... def __prepare__(metacls, name, bases, **kwargs):
... return member_table()
...
... def __new__(cls, name, bases, clsdict, **kwargs):
... #So that the MetaClass is inheritable, don't apply
... #to things without a base
... if not bases:
... return type.__new__(cls, name, bases, {})
... #The first two things in member_names are always
... #"__name__" and "__module__", so don't pass those on
... return namedtuple(name, ' '.join(clsdict.member_names[2:]),
**kwargs)...
... def __init__(cls, name, bases, classdict, **kwargs):
... type.__init__(cls, name, bases, classdict)
...
>>> class NamedTuple(metaclass=TupleNamer): pass
...
>>> class Point(NamedTuple, rename=True): x, y, x
...
>>> Point(1, 2, 3)
Point(x=1, y=2, _2=3)
Antoine Pitrou
"Carl M. Johnson"
<cmjohnson....@gmail.com> wrote:
> >>>
> >>> # The custom dictionary
> ... class member_table:
> ... def __init__(self):
> ... self.member_names = []
> ...
> ... def __setitem__(self, key, value):
> ... self.member_names.append(key)
> ...
> ... def __getitem__(self, key):
> ... self.member_names.append(key)
IMO, this looks like too much magic for a stdlib class.
Especially the __getitem__ which will remember any reference made from
within the class definition, at the risk of surprising behaviour.
Also, your notation is actually less powerful, since you can't define
anything else than members with it (you can't add any methods or
properties to your namedtuple-derived class, except by subclassing it
even further).
Regards
Antoine.
Arnaud Delobelle
On 4 Mar 2011, at 07:17, Carl M. Johnson wrote:
> I've been following the discussion on the list about decorators for
> assignments and whatnot, and while I do think it would be interesting
> if there were some way to abstract what happens in the creation of
> classes and defs into a more generalizable structure, for the
> immediate use case of NamedTuple, wouldn't an interface something like
> this solve the DRY problem? --
>
> class Point(NewNamedTuple): x, y
>
> This interface couldn't have worked in Python 2.6 when namedtuple was
> introduced, but using the __prepare__ statement in Python 3, it's
> trivial to implement. As a bonus, NewNamedTuple could also support
> type checking with the following syntax:
>
> class Point(NewNamedTuple): x, y = int, int #or whatever the ABC is
> for int-like numbers
>
> In fact, since as it stands namedtuple is called "namedtuple" in
> lowercase, we could just camel case "NamedTuple" to collections
> instead of using the NewNamedTuple or whatever. Or, and I haven't done
> any research on this, it may also be possible to implement
> NewNamedTuple with backwards compatibility so it can also be used in
> the traditional way.
It reminds me that a while ago (I think at the time of python 2.4), before the introduction of namedtuple, I had my own implementation of a "struct" decorator to create named tuples that enforced DRY and didn't require any metaclass magic. It worked as follows.
>>> @struct
... def Point(x=0, y=0):
... "Two dimensional point with x and y coordinates"
... return x, y
...
>>> p = Point(1, 2)
>>> p
Point(1, 2)
>>> tuple(p)
(1, 2)
>>> p.x, p.y
(1, 2)
>>> type(p)
<class '__main__.Point'>
>>> Point.__doc__
'Two dimensional point with x and y coordinates'
>>> Point(y=3)
(0, 3)
As you can see it abused "def", which at the time was the only way to create decorable objects that were aware of their own name. It was implemented as follows:
def struct(f):
classname = f.__name__
prop_names = f.func_code.co_varnames[:f.func_code.co_argcount]
def _new(cls, *args, **kwargs):
return tuple.__new__(cls, f(*args, **kwargs))
def _repr(self):
return '%s%s' % (type(self).__name__, tuple(self))
def prop(i):
return property(lambda self: self[i])
attrs = {
'__slots__': (),
'__new__': _new,
'__repr__': _repr,
'__doc__': f.__doc__
}
attrs.update((name, prop(i)) for i, name in enumerate(prop_names))
return type(classname, (tuple,), attrs)
--
Arnaud
Carl M. Johnson
says "class" but you can't do normal class sorts of things like create
methods for the namedtuple subclass. There's also the inelegance that
NamedTuple is treated as a special case by the TupleNamer metaclass,
different from subclasses of NamedTuple. There's a similar issue with
ORMs where normally a subclass of Table is a description of a table
and its fields, but sometimes you want to actually create a new class
that's like Table and you can't get that done simply by subclassing.
A lot of these problems could be addressed by something like the
proposed "make" keyword. Imagine if the interfaces for NamedTuple and
Table were:
make NamedTuple(rename=True) Point:
x, y
make Table() Author:
firstname, lastname, DOB = Str(), Str(), Date()
Those strike me as nice enough declarative syntaxes. But working out
exactly how a make statement would work tends to give me a headache. I
suppose at a minimum, a making-object should have a __prepare__ method
and an __init__ method (or perhaps an __enter__ method, and an
__exit__ method). But the more one thinks about all the aspects that
would be nice for a good making-object, the more confusing it
becomes...
-- Carl Johnson
Raymond Hettinger
On Mar 4, 2011, at 8:24 PM, Carl M. Johnson wrote:
> Imagine if the interfaces for NamedTuple and
> Table were:
>
> make NamedTuple(rename=True) Point:
> x, y
>
> make Table() Author:
> firstname, lastname, DOB = Str(), Str(), Date()
To my eyes, that doesn't even look like Python anymore.
It looks a little bit like a function annotation that lost its "->"
I think this thread has completely lost perspective.
The problem being solved is just a minor inelegance
in the signature of a factory function. That signature
would be substantially the same in most commonly
used languages. It's not a deep problem crying out
for a syntactic atrocity to solve it.
Raymond
P.S. Its "namedtuple" not NamedTuple. The latter suggests
a class rather than a factory function.
Carl M. Johnson
<raymond....@gmail.com> wrote:
> P.S. Its "namedtuple" not NamedTuple. The latter suggests
> a class rather than a factory function.
"NamedTuple" is my name for a hypothetical replacement for namedtuple
that uses Metaclasses rather than being a factory function.
Greg Ewing
> make NamedTuple(rename=True) Point:
> x, y
>
> make Table() Author:
> firstname, lastname, DOB = Str(), Str(), Date()
I did some thinking about this kind of thing once when I was
considering Python as a language for interactive fiction. When
writing IF, you tend to have a lot of unique objects with
special behaviour, and to support this, dedicated IF languages
such as TADS usually have a construct that's somewhere between
a class declaration and an instantiation.
For Python, I postulated an "instance" statement that would
be used something like this:
instance Wardrobe(Thing):
name = "wardrobe"
description = "A nice mahogany double-door wardrobe."
def take(self):
print "The wardrobe is too heavy to pick up."
What this would do is first create an anonymous subclass of
Thing, and then instantiate that subclass.
This isn't quite what we're after here, though, because for
use cases such as namedtuple, we don't usually want a subclass,
and we *do* usually want to pass parameters to the constructor.
We'd also like it to fit on one line in simple cases.
It's hard to come up with a syntax that incorporates all of
these possibilites at the same time without becoming rather
long and confusing. The base class list and the construction
parameters both want to go in parentheses, and how do you
make it obvious which is which?
--
Greg
Arnaud Delobelle
On 5 Mar 2011, at 23:18, Greg Ewing wrote:
> Carl M. Johnson wrote:
>
>> make NamedTuple(rename=True) Point:
>> x, y
>> make Table() Author:
>> firstname, lastname, DOB = Str(), Str(), Date()
>
> I did some thinking about this kind of thing once when I was
> considering Python as a language for interactive fiction. When
> writing IF, you tend to have a lot of unique objects with
> special behaviour, and to support this, dedicated IF languages
> such as TADS usually have a construct that's somewhere between
> a class declaration and an instantiation.
>
> For Python, I postulated an "instance" statement that would
> be used something like this:
>
> instance Wardrobe(Thing):
>
> name = "wardrobe"
> description = "A nice mahogany double-door wardrobe."
>
> def take(self):
> print "The wardrobe is too heavy to pick up."
>
> What this would do is first create an anonymous subclass of
> Thing, and then instantiate that subclass.
That's interesting, I once wrote a simple IF engine in Python and I wanted to achieve this almost exactly! I solved it as follows: instances are declared as classes inheriting from their type and the special type named Instance. So the the class definition below binds "House" to an instance of Location.
class House(Instance, Location):
description = "a small country house"
long_description = "You are in a small country house."
objects = RedKey, WoodenChest, BackDoor
def go_south():
if BackDoor.is_open:
message('you step through the door into the garden.')
location = Garden
else:
message("You can't go THROUGH the door, it's closed!")
Instance was defined as follows:
class MetaInstance(MetaObject):
def __init__(cls, name, bases, attrs):
pass
class Instance(object):
__metaclass__ = MetaInstance
@staticmethod
def MetaInstance__new__(meta, name, bases, attrs):
bases = list(bases)
bases.remove(Instance)
cls = bases.pop()
return cls(**attrs)
MetaInstance.__new__ = MetaInstance__new__
--
Arnaud
Carl M. Johnson
-- creating namespaces, setting GUI properties, templating HTML -- are
still compelling. If you add in the examples of ORM tables and
interactive fiction processors, I think what unites all the examples
is that the make statement is a way of creating a DSL from within
Python. Can using a DSL within Python be pythonic ,though? That's a
difficult question to answer. But, on the other hand, just because
there's no clean, non-magical way of making a DSL within Python
doesn't mean people won't try and indeed, we can see the results of
their trying out there today.
For example, see Biwako <http://martyalchin.com/2011/jan/20/biwako/> a
recent project which abuses metaclasses to create a declarative syntax
for processing file formats. Here's an excerpt from that page:
> For example, here’s a very simple Biwako class that will can parse part of the
> GIF file format, allowing you to easily get to the width and height of any GIF image.
>
> from biwako import bin
>
> class GIF(bin.Structure, endianness=bin.LittleEndian, encoding='ascii'):
> tag = bin.FixedString('GIF')
> version = bin.String(size=3)
> width = bin.Integer(size=2)
> height = bin.Integer(size=2)
>
> Now you have a class that can accept any GIF image as a file (or any file-like
> object that’s readable) and parse it into the attributes shown on this class.
>
> >>> image = GIF(open('example.gif', 'rb'))
> >>> image.width, image.height
> (400, 300)
So basically, the author of this project is calling GIF a "class" but
it's not something that really operates the way a normal class does,
because it's subclassing the magical bin.Structure class and
inheriting its metaclass.
With a little searching, you can find similar examples of abuse that
are centered around the with statement rather than metaclasses. People
have made the with statement into an XML generator
<http://langexplr.blogspot.com/2009/02/writing-xml-with-ironpython-xmlwriter.html>
or an anonymous block handler
<http://code.google.com/p/ouspg/wiki/AnonymousBlocksInPython>.
Seeing examples like these make me think a re-examination of PEP 359
would be a good idea. Of course, I do think it needs a little more
work (in particular, I think the make statement should have an
equivalent of a __prepare__ method and should receive the BLOCK as a
callback instead of automatically executing it), but the core idea is
worth taking another look at.
-- Carl
Guido van Rossum
Hm... I find that example pretty clear and don't think it is in much
need of improvement. I also don't think there's anything wrong with
using class -- after all each call to GIF() returns a new object whose
attributes are defined by the definition. I'd assume that adding
methods to the class would just work and be utterly unsurprising to
anyone familiar with basic classes.
> With a little searching, you can find similar examples of abuse that
> are centered around the with statement rather than metaclasses. People
> have made the with statement into an XML generator
> <http://langexplr.blogspot.com/2009/02/writing-xml-with-ironpython-xmlwriter.html>
> or an anonymous block handler
> <http://code.google.com/p/ouspg/wiki/AnonymousBlocksInPython>.
TBH I find such abuse of 'with' much more troubling.
> Seeing examples like these make me think a re-examination of PEP 359
> would be a good idea. Of course, I do think it needs a little more
> work (in particular, I think the make statement should have an
> equivalent of a __prepare__ method and should receive the BLOCK as a
> callback instead of automatically executing it), but the core idea is
> worth taking another look at.
This I agree with. I have a feeling that last time around it sunk
primarily because people were trying to pile too many different
semantics onto the same syntax -- hopefully the infatuation with Ruby
anonymous blocks has deflated somewhat by now.
--
--Guido van Rossum (python.org/~guido)
Bruce Leban
On Sat, Mar 5, 2011 at 10:27 PM, Carl M. Johnson
<cmjohnson....@gmail.com> wrote:> With a little searching, you can find similar examples of abuse that
> are centered around the with statement rather than metaclasses. PeopleTBH I find such abuse of 'with' much more troubling.
> have made the with statement into an XML generator
> <http://langexplr.blogspot.com/2009/02/writing-xml-with-ironpython-xmlwriter.html>
> or an anonymous block handler
> <http://code.google.com/p/ouspg/wiki/AnonymousBlocksInPython>.
--- Bruce
Raymond Hettinger
. Personally, the xml writer seems like a reasonable use to me.
x = XWriter(w)
with x.element('tableofcontents'):
with x.element('section',{'page' : '10'}):
x.text('Introduction')
with x.element('section',{'page' : '12'}):
x.text('Main topic')
with x.element('section',{'page' : '14'}):
x.text('Extra topic')
<section page="10">Introduction</section>
<section page="12">Main topic</section>
<section page="14">Extra topic</section>
</tableofcontents>
Nick Coghlan
The XML example is fine, since it is just another application of
"before-and-after" coding that the with statement was designed (that
kind of thing was mentioned explicitly in the PEP 343 discussions,
although never elaborated to anything like that degree).
The bytecode hackery involved in AnonymousBlocksInPython and the
withhacks module makes for fun toys to play with, but nothing that
should even be remotely contemplated for a production system.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Bruce Leban
On Mar 6, 2011, at 10:06 PM, Bruce Leban wrote:. Personally, the xml writer seems like a reasonable use to me.I'm surprised that you like the XML writer. To me it seems much more awkward to type the python code than the XML it generates:
<snip>
At least in this example, it seems to me that the XML writer created more work and more complexity than it saved.
with x.element('foo'):for a in stuff:with x.element('bar'):a.render(x)
x.write(x.element('foo', [x.element('bar', a.render()) for a in stuff]))
Nick Coghlan
<raymond....@gmail.com> wrote:
>
> On Mar 6, 2011, at 10:06 PM, Bruce Leban wrote:
>
> . Personally, the xml writer seems like a reasonable use to me.
>
> I'm surprised that you like the XML writer. To me it seems much more
> awkward to type the python code than the XML it generates:
> w = XmlWriter.Create(System.Console.Out,XmlWriterSettings(Indent=True))
> x = XWriter(w)
>
> with x.element('tableofcontents'):
> with x.element('section',{'page' : '10'}):
> x.text('Introduction')
> with x.element('section',{'page' : '12'}):
> x.text('Main topic')
> with x.element('section',{'page' : '14'}):
> x.text('Extra topic')
However, appropriate use of keyword arguments would clean that up quite a bit:
def toc(**kwds):
return x.element('tableofcontents', kwds)
def section(**kwds):
return x.element('section', kwds)
with toc():
with section(page='10'):
x.text(intro_text)
with section(page='12'):
x.text(Main topic)
with section(page='14'):
x.text(Extra topic)
The CM style also lets you do a lot of nice things like providing
required-by-schema default values, as well as providing conveniences
to make within-document cross references easier to create.
I don't think the linked page is a particularly great exemplar of the
style, but you *can* do good things with the approach.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Guido van Rossum
> On Mon, Mar 7, 2011 at 4:06 PM, Bruce Leban <br...@leapyear.org> wrote:
>> On Sun, Mar 6, 2011 at 7:11 PM, Guido van Rossum <gu...@python.org> wrote:
>>> On Sat, Mar 5, 2011 at 10:27 PM, Carl M. Johnson
>>> <cmjohnson....@gmail.com> wrote:> With a little searching, you can
>>> find similar examples of abuse that
>>> > are centered around the with statement rather than metaclasses. People
>>> > have made the with statement into an XML generator
>>> >
>>> > <http://langexplr.blogspot.com/2009/02/writing-xml-with-ironpython-xmlwriter.html>
>>> > or an anonymous block handler
>>> > <http://code.google.com/p/ouspg/wiki/AnonymousBlocksInPython>.
>>>
>>> TBH I find such abuse of 'with' much more troubling.
>>
>> I'm curious if you are troubled by both of these or one more than the other.
>> Personally, the xml writer seems like a reasonable use to me. While I really
>> don't like the anonymous block hack (either the use or the implementation).
>
> The XML example is fine, since it is just another application of
> "before-and-after" coding that the with statement was designed (that
> kind of thing was mentioned explicitly in the PEP 343 discussions,
> although never elaborated to anything like that degree).
I don't think it's fine. The quoted example looks like it depends too
much on implicit side effects.
It is possible that I still (after all these years, and after
enthusiastically supporting its introduction) don't fully appreciate
the with-statement. I find it useful when I can clearly visualize the
alternative code, which typically involves extra flow control such as
a try/finally block. In this example it looks more like the
alternative is just more calls. I expect that in reality the
generation of XML is much more dynamic than in the example, and that
the with-statement won't provide much help (there may even be cases
where it could get in the way).
I was going to say that I don't have much experience with generating
XML, but that's not really true -- I have plenty experience generating
HTML, which is a sufficiently similar experience. In fact I just
hacked together a prototype for an HTML generating library which uses
nested function calls instead of nested with-statements to represent
the nested structure of HTML. I think I like that approach better
because it makes it easier to do some parts of the generation out of
order: I can construct various sub-lists of HTML elements and then
splice them together. E.g.
row1 = []
row2 = []
for x in <something>:
row1.append(libhtml.td(f1(x)))
row2.append(libhtml.td(f2(x)))
t = libhtml.table(libhtml.tr(row1), libhtml.tr(row2))
# Etc.
--
--Guido van Rossum (python.org/~guido)
Jim Jewett
>>>> On Sat, Mar 5, 2011 at 10:27 PM, Carl M. Johnson wrote:>
>>>> > People have made the with statement
>>>> > into an XML generator
>>>> > <http://langexplr.blogspot.com/2009/02/writing-xml-with-ironpython-xmlwriter.html>
> I don't think it's fine. The quoted example looks
> like it depends too much on implicit side effects.
What side effects?
> I find it useful when I can clearly visualize the
> alternative code, which typically involves extra flow
> control such as a try/finally block.
I had thought of the finally as the element-close tags...
> In this example it looks more like the
> alternative is just more calls.
Sure, but the close tag comes arbitrarily later, just like the close
of a file -- which was one of the canonical use cases.
> ... I just hacked together a prototype for an HTML
> generating library which uses nested function calls
> instead of nested with-statements to represent
> the nested structure of HTML. I think I like that
> approach better because it makes it easier to do
> some parts of the generation out of order: I can
> construct various sub-lists of HTML elements
> and then splice them together. E.g.
>
> row1 = []
> row2 = []
> for x in <something>:
> row1.append(libhtml.td(f1(x)))
> row2.append(libhtml.td(f2(x)))
> t = libhtml.table(libhtml.tr(row1), libhtml.tr(row2))
I think the main appeal of the above library is that you *don't* have
to follow this pattern -- you can create the broad outline before you
have the innermost details.
-jJ
Greg Ewing
> I had thought of the finally as the element-close tags...
But generation of the closing tags doesn't really have to
be done in a finally block, unless you're somehow wanting
to support throwing an exception in the middle of your
xml generation and still have it generate well-formed xml.
In the absence of such a requirement, using a with-statement
seems like overkill.
--
Greg
Nick Coghlan
> Jim Jewett wrote:
>
>> I had thought of the finally as the element-close tags...
>
> But generation of the closing tags doesn't really have to
> be done in a finally block, unless you're somehow wanting
> to support throwing an exception in the middle of your
> xml generation and still have it generate well-formed xml.
>
> In the absence of such a requirement, using a with-statement
> seems like overkill.
You don't use it as a finally block for that style of thing - if you
hit any exception, you just reraise it (really easy with
@contextmanager - simply don't put a try/finally around the yield
statement).
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Jan Kaliszewski
Greg Ewing dixit (2011-03-06, 12:18):
> For Python, I postulated an "instance" statement that would
> be used something like this:
>
> instance Wardrobe(Thing):
>
> name = "wardrobe"
> description = "A nice mahogany double-door wardrobe."
>
> def take(self):
> print "The wardrobe is too heavy to pick up."
Why don't use a class decorator? E.g.:
def instance(*args, **kwargs):
return (lambda cls: cls(*args, **kwargs))
And then simply:
@instance(...some init args...)
class Wardrobe(Thing):
...
Cheers.
*j