Source formatting for long expressions.
Christopher Armstrong
Background:
I'm writing a source persistence mechanism for Twisted
(http://twistedmatrix.com) as an alternative to the Pickle and XML persistence
that we currently have. Basically, I have code that turns Python objects into
something that looks like::
Ref(1, Instance('twisted.internet.app.Application', persistenceVersion=7,
services={}, udpPorts=[], gid=1000, connectors=[], uid=1000,
tcpPorts=[(9080, Instance('twisted.web.server.Site', sessions={},
resource=Instance('twisted.coil.web.ConfigRoot', children={},
modules=[], widgets={'config':
Instance('twisted.coil.web.AppConfiguratorPage', variables={},
dispensers=Instance('twisted.coil.coil.DispenserStorage', dispensers={}}),
app= ...
<bazillion more lines>
As you can see, this is one really long expression :-) I already have the
code that persists to/unpersists from this format (it was fairly easy with the
'jelly' serialization framework), but it's not very useful unless it's in a
pretty format that's easy for people to edit. Does anyone know of a tool that
will beak up a very, very long expression like this into multiple lines with
sane indentation? I'll probably end up writing the beast myself, but I don't
want to waste my time if there's something out there.
--
Chris Armstrong
<< ra...@twistedmatrix.com >>
http://twistedmatrix.com/users/carmstro.twistd/
Martin v. Loewis
> Does anyone know of a tool that will beak up a very, very long
> expression like this into multiple lines with sane indentation?
Emacs indentation gives this:
Ref(1, Instance('twisted.internet.app.Application',
persistenceVersion=7, services={}, udpPorts=[], gid=1000,
connectors=[], uid=1000, tcpPorts=[(9080,
Instance('twisted.web.server.Site', sessions={},
resource=Instance('twisted.coil.web.ConfigRoot', children={},
modules=[], widgets={'config':
Instance('twisted.coil.web.AppConfiguratorPage', variables={},
dispensers=Instance('twisted.coil.coil.DispenserStorage',
dispensers={}})})))])
As you can see, it tries to align further arguments to a function
together with the opening parenthesis. I would assume that any other
auto-formatters use the same style, so you won't get any "sane"
indentation until you drop the nesting level (perhaps in favour of
method invocations).
Regards,
Martin
Christopher Armstrong
mvl> As you can see, it tries to align further arguments to a function
mvl> together with the opening parenthesis. I would assume that any other
mvl> auto-formatters use the same style, so you won't get any "sane"
mvl> indentation until you drop the nesting level (perhaps in favour of
mvl> method invocations).
See http://twistedmatrix.com/users/carmstro/pypersist.py.txt for a hand-written
sample of what I'm looking for. It's basically what you quoted with a lot more
linebreaks, and instead of aligning to left-parenthesis, I just indent two
spaces for each depth-level.
I'm probably going to end up doing this myself, and I thought of something
while reading your message. I could have two steps for formatting: template-
generation and indentation. the first step would generate something like a
repr() of all of my objects, but with \ns and \ts embedded at strategic
locations. Then I could replace all \ts with current_indent_level, calculating
current_indent_level by counting (s, {s, and [s. I'm going to try this out now.
:-)
John J. Lee
> Christopher Armstrong <ra...@twistedmatrix.com> writes:
>
> > Does anyone know of a tool that will beak up a very, very long
> > expression like this into multiple lines with sane indentation?
>
> Emacs indentation gives this:
>
> Ref(1, Instance('twisted.internet.app.Application',
> persistenceVersion=7, services={}, udpPorts=[], gid=1000,
> connectors=[], uid=1000, tcpPorts=[(9080,
> Instance('twisted.web.server.Site', sessions={},
[...]
Emacs python-mode will actually align only one indent level in (instead of
aligning with the opening bracket) if you have a newline after the
bracket:
Ref(1, Instance(
'twisted.internet.app.Application',
persistenceVersion...
The tcpPorts bit should obviously be moved to another line, though.
John
Christopher Armstrong
>>>>> "mvl" == Martin v Loewis <mar...@v.loewis.de> writes:
mvl> As you can see, it tries to align further arguments to a function
mvl> together with the opening parenthesis. I would assume that any other
mvl> auto-formatters use the same style, so you won't get any "sane"
mvl> indentation until you drop the nesting level (perhaps in favour of
mvl> method invocations).
[snip..]
ca> I'm probably going to end up doing this myself, and I thought of
ca> something while reading your message. I could have two steps for
ca> formatting: template- generation and indentation. the first step would
ca> generate something like a repr() of all of my objects, but with \ns and
ca> \ts embedded at strategic locations. Then I could replace all \ts with
ca> current_indent_level, calculating current_indent_level by counting (s,
ca> {s, and [s. I'm going to try this out now. :-)
I got this working. It was fairly easy, once I got a templated source
representation::
def indentify(s):
out = []
stack = []
for ch in s:
if ch in ['[', '(', '{']:
stack.append(ch)
elif ch in [']', ')', '}']:
stack.pop()
if ch == '\t':
out.append(' '*len(stack))
else:
out.append(ch)
return string.join(out, '')
This is a fairly stupid indenter, but it works for my needs::
>>> print aot.indentify("Instance('twisted.internet.app.Application', \
\n\tfoo=bar, \n\tbaz={\n\t'quux': 1, \n\t'spam': \n\tInstance('Eggs', \
\n\tmore='state')})") #this source will really be generated by my __repr__s.
Instance('twisted.internet.app.Application',
foo=bar,
baz={
'quux': 1,
'spam':
Instance('Eggs',
more='state')})
Skip Montanaro
Chris> expression like this into multiple lines with sane indentation?
How about the pprint module?
--
Skip Montanaro (sk...@pobox.com - http://www.mojam.com/)
Boycott Netflix - they spam - http://www.musi-cal.com/~skip/netflix.html
Christopher Armstrong
>>>>> "Skip" == Skip Montanaro <sk...@pobox.com> writes:
Chris> Does anyone know of a tool that will beak up a very, very long
Chris> expression like this into multiple lines with sane indentation?
Skip> How about the pprint module?
I've looked at the source for pprint (on Python 2.1), and I saw nothing
that allows me to extend it with my own classes. If you check my other
replies to this thread, you can see my (stupid, but effective) solution.
sameer
Christopher Armstrong <ra...@twistedmatrix.com> wrote in message news:<mailman.1023668414...@python.org>...
Christopher Armstrong
>>>>> "sameer" == sameer <sam...@email.com> writes:
sameer> why not just do it in xml?
Twisted already has an XML persistence mechanism, and that is exactly what
inspired me (with disgust) to write AOT (this source-persistence mechanism).
AOT is better in every possible respect than the XML persistence IMNSHO, save
for the fact that it requires (minimal) understanding of Python syntax,
and the format for References is _slightly_ uglier (Ref(num, obj) -- since not
every object is represented with a "tag" (or class call), I can't insert
reference numbers "into" arbitrary objects).
XML is a terrible format for object persistence, due to the lack of robust
mappings. Twisted's XML uses attributes in some cases, but that is only
possible when you have string:string mappings. In *any* other case, you
have to resort to children nodes, with such ugly syntax as::
<instance class="twisted.some.Class">
<dictionary>
<string role="key" value="someAttr" />
<list>
<string value="foo" />
</list>
</dictionary>
Compare that with the equivalent AOT format::
Instance("twisted.some.Class",
someAttr=["foo"],
)
*That* is why I don't just use XML.
Michael Gilfix
> XML is a terrible format for object persistence, due to the lack of robust
> mappings. Twisted's XML uses attributes in some cases, but that is only
> possible when you have string:string mappings. In *any* other case, you
> have to resort to children nodes, with such ugly syntax as::
>
> <instance class="twisted.some.Class">
> <dictionary>
> <string role="key" value="someAttr" />
> <list>
> <string value="foo" />
> </list>
> </dictionary>
>
> Compare that with the equivalent AOT format::
>
> Instance("twisted.some.Class",
> someAttr=["foo"],
> )
>
> *That* is why I don't just use XML.
I'm not disagreeing with you because I don't think you're wrong in
this case and well, XML isn't for everything. I think XML can work
here just as well but it's just more work. I'm not sure about the
lack of robustness of the mappings - you just have to define them and
can use a DTD to verify that your mappings are intact. But you have
to write extra code for the mappings that isn't quite worth it in
such a domain specific case. XML costs extra for being general. Er,
you don't have any between version/backwards-compat issues though do
you? In which case, have you thought those out thoroughly? Forgive me
if it isn't relavent. Not sure exactly where these will be used.
-- Mike
--
Michael Gilfix
mgi...@eecs.tufts.edu
For my gpg public key:
http://www.eecs.tufts.edu/~mgilfix/contact.html
Christopher Armstrong
Michael> On Tue, Jun 11 @ 10:32, Christopher Armstrong wrote:
>> XML is a terrible format for object persistence, due to the lack of
>> robust mappings. Twisted's XML uses attributes in some cases, but that
>> is only possible when you have string:string mappings. In *any* other
>> case, you have to resort to children nodes, with such ugly syntax as::
[snip..]
>> *That* is why I don't just use XML.
Michael> I'm not disagreeing with you because I don't think you're wrong
Michael> in this case and well, XML isn't for everything. I think XML can
Michael> work here just as well but it's just more work. I'm not sure about
Michael> the lack of robustness of the mappings - you just have to define
Michael> them and can use a DTD to verify that your mappings are
Michael> intact. But you have to write extra code for the mappings that
I meant the mappings that attributes provide. Creating mappings with multiple
tags is overbearingly verbose, no matter if you formalize it or not. (Am I
understanding you correctly?) Perhaps I should've said "flexible" rather
than "robust". BTW, ONX (http://www.seairth.com/web/onx/onx.html) provides
a better type of mapping: string:arbitrary object. It's not
arbitrary object:arbitrary object, but it's a bit better.
Michael> isn't quite worth it in such a domain specific case. XML costs
Michael> extra for being general. Er, you don't have any between
Michael> version/backwards-compat issues though do you? In which case, have
Michael> you thought those out thoroughly? Forgive me if it isn't
Michael> relavent. Not sure exactly where these will be used.
Versioning the persistent data is pretty much irrelevant wrt the format used --
and yes, I have thought about it. Marmalade right now uses it's own versioned
state-getters/setters (jellyToDOM_X where X is the version number), but AOT
doesn't have its own interface -- it just uses __getstate__/__setstate__ like
pickle. I'm probably going to write a mixin class for versioned state that
supports methods like __getstate_X__ where X is the version of the persisted
state.
Note that Twisted has a class for versioning state that's orthogonal to
persisted format: twisted.persisted.styles.Versioned. This allows you
to define 'upgradeToVersionX' methods that will be called in order to
upgrade an object to the current version of the code. The only reason
to use versioned state getters/setters as opposed to this is if you want
the format of the persistent state to change for cosmectic reasons.
Michael Gilfix
> I meant the mappings that attributes provide. Creating mappings with multiple
> tags is overbearingly verbose, no matter if you formalize it or not. (Am I
> understanding you correctly?) Perhaps I should've said "flexible" rather
> than "robust". BTW, ONX (http://www.seairth.com/web/onx/onx.html) provides
> a better type of mapping: string:arbitrary object. It's not
> arbitrary object:arbitrary object, but it's a bit better.
I agree that's very verbose and cumbersome. That's why I supported your
non-use of XML. ONX is interesting though. Gonna have to take a closer look.
> Versioning the persistent data is pretty much irrelevant wrt the format used --
> and yes, I have thought about it. Marmalade right now uses it's own versioned
> state-getters/setters (jellyToDOM_X where X is the version number), but AOT
> doesn't have its own interface -- it just uses __getstate__/__setstate__ like
> pickle. I'm probably going to write a mixin class for versioned state that
> supports methods like __getstate_X__ where X is the version of the persisted
> state.
If anything, I think that's where the XML format is nice - in versioning.
Usually, you get to do considerably more code-sharing with XML formats and
the update functions are very simplistic.
> Note that Twisted has a class for versioning state that's orthogonal to
> persisted format: twisted.persisted.styles.Versioned. This allows you
> to define 'upgradeToVersionX' methods that will be called in order to
> upgrade an object to the current version of the code. The only reason
> to use versioned state getters/setters as opposed to this is if you want
> the format of the persistent state to change for cosmectic reasons.
This sounds prety cool. Will have to check it out.
Frank McIngvale
> On Tue, Jun 11 @ 10:32, Christopher Armstrong wrote:
> > XML is a terrible format for object persistence, due to the lack of robust
> > mappings. Twisted's XML uses attributes in some cases, but that is only
> > possible when you have string:string mappings. In *any* other case, you
> > have to resort to children nodes, with such ugly syntax as::
> >
> > <instance class="twisted.some.Class">
> > <dictionary>
> > <string role="key" value="someAttr" />
> > <list>
> > <string value="foo" />
> > </list>
> > </dictionary>
> >
> > Compare that with the equivalent AOT format::
> >
> > Instance("twisted.some.Class",
> > someAttr=["foo"],
> > )
> >
> > *That* is why I don't just use XML.
>
> I'm not disagreeing with you because I don't think you're wrong in
> this case and well, XML isn't for everything. I think XML can work
> here just as well but it's just more work.
>
Hi, I'm not arguing the relative merits of using XML for persistance, but
just fyi, if you want a straight XML pickler, check out:
http://www.gnosis.cx/download/Gnosis_Utils-current.tar.gz
http://freshmeat.net/projects/gnosisxml/
frank