[OT] code is data
Anton Vredegoor
developments concerning JSON (a very Pythonesque but somewhat limited
XML notation scheme, let's call it statically typed XML) Python seems to
have reached a stage where it now seems to be possible to completely
swallow lesser languages code, modify it, and spit out new source code
targeting the original language the code was written in, or even make a
translation to other languages.
The idea is that we now have a fast parser (ElementTree) with a
reasonable 'API' and a data type (XML or JSON) that can be used as an
intermediate form to store parsing trees. Especially statically typed
little languages seem to be very swallow-able. Maybe I will be able to
reimplement GFABasic (my first love computer language, although not my
first relationship) someday, just for fun.
Then there are things like cTypes (calling functions from native DLL's)
and PyPy (implementing Python in Python).
All this taken together, to me it starts looking like we're now entering
a territory that traditionally was exclusively in the Lisp domain.
Yes, Python had eval and exec for a long time already, and metatypes and
generators are having some strange unexplored possibilities too, but the
day will come soon (and at last when PyPy is reaching execution speeds
close to cPython) where Python will be able to swallow smaller
languages, and finally it will be able to swallow its own tail, like
Lisp but then more powerful (because of the widely used standard data
types and the code exchange between languages that that makes possible).
Your thoughts please.
Anton
John Roth
> With the inclusion of ElementTree (an XML-parser) in Python25 and recent
> developments concerning JSON (a very Pythonesque but somewhat limited
> XML notation scheme, let's call it statically typed XML) Python seems to
> have reached a stage where it now seems to be possible to completely
> swallow lesser languages code, modify it, and spit out new source code
> targeting the original language the code was written in, or even make a
> translation to other languages.
When I heard of the new AST based compiler, I thought it
would finally be possible to extend things cleanly. Then
I learned that they weren't going to allow modification of
the AST.
There are a lot of things you can't do in source if the
language doesn't allow it. Python is pretty good compared
to other languages, but it's still not possible to create
new control structures with short circuit semantics.
And that's one example.
I saw the "make" statement as a breath of fresh air.
Then it got shot down for what were, to me, totally
trivial reasons. That's a second one.
Sigh.
John Roth
>
>
> Your thoughts please.
>
> Anton
Paddy
> With the inclusion of ElementTree (an XML-parser) in Python25 and recent
> developments concerning JSON (a very Pythonesque but somewhat limited
> XML notation scheme, let's call it statically typed XML)
>
> Your thoughts please.
>
> Anton
Hi Anton.
If you mean this JSON: http://www.json.org/example.html
then I'd just point out that JSON isn't XML-like at all. In fact the
examples look like valid Python nested dictionaries.
- Pad.
P.S. This is good too: http://en.wikipedia.org/wiki/JSON
Ravi Teja
Paddy wrote:
> Anton Vredegoor wrote:
> > With the inclusion of ElementTree (an XML-parser) in Python25 and recent
> > developments concerning JSON (a very Pythonesque but somewhat limited
> > XML notation scheme, let's call it statically typed XML)
> <SNIP
> >
> > Your thoughts please.
> >
> > Anton
>
> Hi Anton.
> If you mean this JSON: http://www.json.org/example.html
> then I'd just point out that JSON isn't XML-like at all. In fact the
> examples look like valid Python nested dictionaries.
It is the same JSON. JSON is typically seen as a human friendly
replacement for some of the functions that XML is otherwise used for,
where the full blown XML spec is an overkill and JSON does not need
complicated parsers in some common languages because it can express
hierarchical data just like XML.
Ravi Teja
I don't share your optimism at all. Most of the things you mentioned
have existed for long. Just because some of them are now included in
the standard library isn't going to change things drastically.
Installing them earlier was never hard at all.
People like to call everything with the lightest semblence, a DSL. That
gives the feel that the language is more powerful. Ruby people do it
all the time. Python cannot be called a DSL language until, creating
them is a natural language feature (like Lisp). And that does not seem
to be happening anytime soon. Boo for example allows you to write new
constructs with it's AST library. It still cannot be called a DSL
"language".
People have however written various language interpreters (Scheme,
Forth and yes, even Basic) in Python, just for kicks. Still does not
make it a DSL language anymore than it makes C a DSL language.
At present, the closest thing to writing a DSL in Python is Logix
http://livelogix.net/logix/
Too bad though, the project is defunct and there has never been enough
interest in it.
Personally, I would like to see macros in Python (actually Logix
succeeding is good enough). But I am no language designer and the
community has no interest in it. When I absolutely need macros, I will
go elsewhere.
BJörn Lindqvist
> succeeding is good enough). But I am no language designer and the
> community has no interest in it. When I absolutely need macros, I will
> go elsewhere.
One must wonder, when is that? When do you absolutely need macros?
--
mvh Björn
Ravi Teja
Whenever there is significant boiler plate code that functions and
classes cannot eliminate alone.
Whenever there is a more elegant way to express your code.
Python 2.5 introduced conditional expressions and with statement. With
macros, one would not have to wait for the language team to implement
them. More so for features which only a small part of the community has
an interest in.
I *like* 1..5 (ada, ruby) instead of range(5). If I had macros, I would
have done it myself for *my* code.
I would like special behaviour code blocks in my programs, for say DBC
(I am aware of the work arounds).
Paddy
Ravi Teja wrote:
> BJörn Lindqvist wrote:
> > > Personally, I would like to see macros in Python (actually Logix
> > > succeeding is good enough). But I am no language designer and the
> > > community has no interest in it. When I absolutely need macros, I will
> > > go elsewhere.
> >
> > One must wonder, when is that? When do you absolutely need macros?
>
> Whenever there is significant boiler plate code that functions and
> classes cannot eliminate alone.
> Whenever there is a more elegant way to express your code.
>
Me, I am torn. I should now better. I have listened to the arguments
against Macros in Python and the ones that struck home were the
argument for maintainability:
Without macros, Python is Python. Statements do what you expect.
And the argument against DSLs altogether:
Make Python your DSL! If you design your own DSL before long you start
to embellish it with more statements or datatypes and before long it
becomes complex. If you used Python from the beginning then you would
have a community for support.
I know the arguments, but every once in a while I think if only I could
craft my own ??? statement or ....
Don't go their Paddy.
;-)
Ravi Teja
> Ravi Teja wrote:
> > BJörn Lindqvist wrote:
> > > > Personally, I would like to see macros in Python (actually Logix
> > > > succeeding is good enough). But I am no language designer and the
> > > > community has no interest in it. When I absolutely need macros, I will
> > > > go elsewhere.
> > >
> > > One must wonder, when is that? When do you absolutely need macros?
> >
> > Whenever there is significant boiler plate code that functions and
> > classes cannot eliminate alone.
> > Whenever there is a more elegant way to express your code.
> >
>
> Me, I am torn. I should now better. I have listened to the arguments
> against Macros in Python and the ones that struck home were the
> argument for maintainability:
> Without macros, Python is Python. Statements do what you expect.
Yes! I heard those arguments too. And I am not convinced.
Static language programmer: Lack of static typing removes the necessary
safeguards. The code is more error prone. Objects have behavior that is
not obvious.
Dynamic language programmer: Really? I don't seem to have any more bugs
than in my statically typed code. And my code is compact and reads
better. I don't want to go back.
No to macros proponent: Macros introduce a lot of potential for abuse.
Code will be worse to read than Perl.
Macros proponent: Really? We have been doing macros for decades. We all
think our code is better for macros, not worse. We are not going back.
I just don't get it. Don't we often invoke the "We are all adults here"
argument.
Writing a macro is not as simple as writing a function. Sort of like
metaclasses. Many will stay off them. Those that really need them will
walk that extra mile. Don't we all believe that "Simple should be
possible. Complex should be doable"
> And the argument against DSLs altogether:
> Make Python your DSL! If you design your own DSL before long you start
> to embellish it with more statements or data types and before long it
> becomes complex. If you used Python from the beginning then you would
> have a community for support.
Python has a low cognitive overhead. But it not a DSL by definition. No
language can be. The idea is that when the domain changes, a DSL should
be driven by the new domain as warranted. In other words, driven "by
the problem, not the tool".
I don't want "a DSL". I want a language that allows me to make "my
DSL" based on it. That means I don't loose the community connection. I
can still use all the rich libraries in my DSL.
I like Python for its indentation syntax, sensible semantics and
readability. I invested a lot of time in Python. After much language
hopping, I settled with Python. I like the community and the code base
available for it. The libraries just seem to be designed at the right
level of abstraction for me (as opposed to say, Java). When I need to
do something, I know where to go. But all this ties me to the language
tightly that I cannot change.
> I know the arguments, but every once in a while I think if only I could
> craft my own ??? statement or ....
My thoughts exactly.
Web frameworks, which seem to be the rage now in Python community could
have benefited tremendously from Macro capabilities since they have a
lot of boiler plate.
Fredrik Lundh
> Web frameworks, which seem to be the rage now in Python community could
> have benefited tremendously from Macro capabilities since they have a
> lot of boiler plate.
they do? methinks you haven't done much web programming lately...
</F>
Ravi Teja
You blogged on Django. Let's use that. Don't you think model creation
in Django can be represented better, given that it is done often
enough?
Let's take an example from the official tutorial
from
http://www.djangoproject.com/documentation/tutorial1/#creating-models
class Poll(models.Model):
question = models.CharField(maxlength=200)
pub_date = models.DateTimeField('date published')
class Choice(models.Model):
poll = models.ForeignKey(Poll)
choice = models.CharField(maxlength=200)
votes = models.IntegerField()
I don't use Django and I made this up quickly, so please don't pick on
subtleties.
@Poll:
question: char length 200
pub_date('date published'): date
@Choice:
poll -> Poll
choice: char length 200
votes: int
The following is my rationale. Annoted variables, symbols and code
layout visually cue more efficiently to the object nature than do
explicit text definitions. Of course, this is only sensible when there
aren't too many of any of those. In that case, the cognitive cost of
notation outweighs the representational cost of text.
Representational minimalism is troublesome in general code (ala Perl),
but not so in a DSL where the context is constrained.
I would also like to symbolize field types since they occur so commonly
in a definition file and only a few of them are commonly used. I admit
though that I find the code below a bit visually jarring and I might
use something else. But it serves to illustrate the point. I chose the
respective symbols based on their colloquial use and association with
the field types.
@Poll:
$question: length 200
%pub_date('date published')
@Choice:
poll -> Poll
$choice: length 200
#votes
Since you are on thread and are a prominent and involved member of the
Python community, I would like it if you (or any such other) can
provide feedback on the rest of my previous post rather than be
dismissive by just a small portion of it. Perhaps, that will give me
some insight how these language design decisions are rationally made (I
am not strictly a programmer by profession, much less a language
designer).
Roberto Bonvallet
> I *like* 1..5 (ada, ruby) instead of range(5). If I had macros, I would
> have done it myself for *my* code.
You can write your own preprocessor to handle things like that.
--
Roberto Bonvallet
BJörn Lindqvist
> > > go elsewhere.
> have done it myself for *my* code.
I think this example more is a symptom of a childish need to get
things your way than of a deficiency in Python. BTW, range(5) = 0..4
in Ada and Ruby.
You said "when I absolutely need macros" but none of your examples
demonstrate any "absolute need." I can't see your point.
--
mvh Björn
bruno at modulix
> With the inclusion of ElementTree (an XML-parser) in Python25 and recent
> developments concerning JSON (a very Pythonesque but somewhat limited
> XML notation scheme, let's call it statically typed XML)
JSON stands for JavaScript Object Notation, and has *nothing* to do with
XML - except for the fact that it's more and more used instead of XML
for AJAX stuff.
> Python seems to
> have reached a stage where it now seems to be possible to completely
> swallow lesser languages code, modify it, and spit out new source code
> targeting the original language the code was written in, or even make a
> translation to other languages.
If you mean "parsing source in a given format and outputting another -
modified or not - representation, in the same or another format", Python
as always been able to do so.
> The idea is that we now have a fast parser (ElementTree) with a
> reasonable 'API' and a data type (XML or JSON) that can be used as an
> intermediate form to store parsing trees. Especially statically typed
> little languages seem to be very swallow-able. Maybe I will be able to
> reimplement GFABasic (my first love computer language, although not my
> first relationship) someday, just for fun.
>
> Then there are things like cTypes (calling functions from native DLL's)
> and PyPy (implementing Python in Python).
>
> All this taken together, to me it starts looking like we're now entering
> a territory that traditionally was exclusively in the Lisp domain.
Sorry, but I just don't get the point. Parsing, working with trees and
calling native code are in no way "exclusively in the Lisp domain".
> Yes, Python had eval and exec for a long time already, and metatypes and
> generators are having some strange unexplored possibilities too, but the
> day will come soon (and at last when PyPy is reaching execution speeds
> close to cPython) where Python will be able to swallow smaller
> languages, and finally it will be able to swallow its own tail, like
> Lisp but then more powerful
I'm afraid Python is still very far from Lisp - and will never get there
(FWIW, this seems not to be the goal anyway).
> (because of the widely used standard data
> types and the code exchange between languages that that makes possible).
I still don't get the point.
> Your thoughts please.
>
> Anton
--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'on...@xiludom.gro'.split('@')])"
bruno at modulix
functions, or even structured programming. All we 'absolutely' need is
tests and gotos... (and love, of course !-).
bruno at modulix
> BJörn Lindqvist wrote:
>
>>>Personally, I would like to see macros in Python (actually Logix
>>>succeeding is good enough). But I am no language designer and the
>>>community has no interest in it. When I absolutely need macros, I will
>>>go elsewhere.
>>
>>One must wonder, when is that? When do you absolutely need macros?
>
>
> Whenever there is significant boiler plate code that functions and
> classes cannot eliminate alone.
> Whenever there is a more elegant way to express your code.
>
> Python 2.5 introduced conditional expressions and with statement. With
> macros, one would not have to wait for the language team to implement
> them. More so for features which only a small part of the community has
> an interest in.
>
> I *like* 1..5 (ada, ruby) instead of range(5). If I had macros, I would
> have done it myself for *my* code.
And that's the downside with macros - and with anything that's not
officially part of the language or it's standard lib : everybody
implements it it's own way, and you end up with dozens non-standard ways
of doing the same thing.
Not to say this is absolutely bad, but there's a balance to be found
here. One could do function decorators long before we had official
syntactic sugar for it, but it only started to be a common idiom with
the @decorator syntax. Python 2.5 introduces a 'partial' type, that is
quite easy to implement with 2.4 (and probably with older versions too),
but having it in the builtins or standard lib means it will become the
standard way to do it - no need to deal with half a dozen half-backed
implementations of it no more.
bruno at modulix
(snip)
> Annoted variables, symbols and code
> layout visually cue more efficiently to the object nature than do
> explicit text definitions. Of course, this is only sensible when there
> aren't too many of any of those. In that case, the cognitive cost of
> notation outweighs the representational cost of text.
>
> Representational minimalism is troublesome in general code (ala Perl),
> but not so in a DSL where the context is constrained.
This still impose the need to learn a new language.
> I would also like to symbolize field types since they occur so commonly
> in a definition file and only a few of them are commonly used. I admit
> though that I find the code below a bit visually jarring and I might
> use something else. But it serves to illustrate the point. I chose the
> respective symbols based on their colloquial use and association with
> the field types.
>
> @Poll:
> $question: length 200
> %pub_date('date published')
>
> @Choice:
> poll -> Poll
> $choice: length 200
> #votes
<IMHO>Yuck</IMHO>.
Anton Vredegoor
> I still don't get the point.
Well, I've got to be careful here, lest I'd be associated with the
terr.., eh, the childp..., eh the macro-enablers.
The idea is to have a way to transform a Python (.py) module into XML
and then do source code manipulations in XML-space using ElementTree.
But rest assured, there is no such module, nor will we ever need it for
anything.
Anton
"use cases are for the faint-hearted"
Michele Simionato
> I saw the "make" statement as a breath of fresh air.
> Then it got shot down for what were, to me, totally
> trivial reasons.
Which reasons? I as I recall, Guido cut it out without giving any
reason.
Of course Guido has the right to do so, but it is not respectful of
all the work people like Steven Bethard and others did :-(
Michele Simionato
bruno at modulix
> bruno at modulix wrote:
>
>> I still don't get the point.
>
>
> Well, I've got to be careful here, lest I'd be associated with the
> terr.., eh, the childp..., eh the macro-enablers.
>
> The idea is to have a way to transform a Python (.py) module into XML
> and then do source code manipulations in XML-space using ElementTree.
My my my... I'm not against the idea of dynamic source code
transformation, but for heaven's sake, *why* would one put XML in the
mix ???????
Laurent Pointal
> Anton Vredegoor wrote:
>> bruno at modulix wrote:
>>
>>> I still don't get the point.
>>
>> Well, I've got to be careful here, lest I'd be associated with the
>> terr.., eh, the childp..., eh the macro-enablers.
>>
>> The idea is to have a way to transform a Python (.py) module into XML
>> and then do source code manipulations in XML-space using ElementTree.
>
> My my my... I'm not against the idea of dynamic source code
> transformation, but for heaven's sake, *why* would one put XML in the
> mix ???????
Because its "à la mode", and is better for a commercial point of view,
even if inefficient for this problem.
Fredrik Lundh
>>> The idea is to have a way to transform a Python (.py) module into XML
>>> and then do source code manipulations in XML-space using ElementTree.
>>
>> My my my... I'm not against the idea of dynamic source code
>> transformation, but for heaven's sake, *why* would one put XML in the
>> mix ???????
because lots of people know how to describe XML transformations, and
there are plenty of tools that implement such transformations efficiently ?
> Because its "à la mode", and is better for a commercial point of view,
> even if inefficient for this problem.
why would XML be inefficient ?
</F>
Kay Schluehr
Ravi Teja wrote:
> People have however written various language interpreters (Scheme,
> Forth and yes, even Basic) in Python, just for kicks. Still does not
> make it a DSL language anymore than it makes C a DSL language.
>
> At present, the closest thing to writing a DSL in Python is Logix
> http://livelogix.net/logix/
> Too bad though, the project is defunct and there has never been enough
> interest in it.
You might be interested in EasyExtend:
http://www.fiber-space.de/EasyExtend/doc/EE.html
Unlike Logix there are no macros defined in application code and there
are no runtime macro expansions. So extension language semantics is
always fixed at compile time.
> Personally, I would like to see macros in Python (actually Logix
> succeeding is good enough). But I am no language designer and the
> community has no interest in it. When I absolutely need macros, I will
> go elsewhere.
Although Logix was written in Python and compiled to CPython bytecodes
it was a language on its own right: a Python / Lisp hybrid. Opposed to
this EasyExtend is a Python framework for language extensions and not
itself a language. A quite typical use case may not involve any new
grammar rules or terminals, but just generates code. See the coverage
fiber as an example.
bruno at modulix
> Laurent Pointal wrote:
>> Bruno Desthuilliers wrote:
>>> Anton Vredegoor wrote:
>>>> The idea is to have a way to transform a Python (.py) module into XML
>>>> and then do source code manipulations in XML-space using ElementTree.
>
>>>
>
>>> My my my... I'm not against the idea of dynamic source code
>>> transformation, but for heaven's sake, *why* would one put XML in the
>>> mix ???????
>
>
> because lots of people know how to describe XML transformations, and
> there are plenty of tools that implement such transformations efficiently ?
Efficiently enough for dynamic (runtime) use ?
Ravi Teja
BJörn Lindqvist wrote:
> > > > community has no interest in it. When I absolutely need macros, I will
> > > > go elsewhere.
> > I *like* 1..5 (ada, ruby) instead of range(5). If I had macros, I would
> > have done it myself for *my* code.
> I think this example more is a symptom of a childish need to get
> things your way than of a deficiency in Python.
I thought I had enough asterisks in there to indicate that it is a
preference that I will not be defending on rational grounds. I had a
better argument before it in the same post. But you had to choose only
the trivial one to dismiss me as childish. Didn't you? :-)
> BTW, range(5) = 0..4 in Ada and Ruby.
My bad. I usually write range(1, 5 + 1) to get 1..5.
I could write range(1, 6). But I would like to see the upper bound
explicitly. Of course, I could write a function to wrap that up.
> You said "when I absolutely need macros" but none of your examples
> demonstrate any "absolute need." I can't see your point.
Did you miss the word - *WHEN*?
I don't need them absolutely now. And I know, that I won't get them
here. And just so you don't misinterpret, I don't call that a
"deficiency". Just a mismatch between the personal and the community
mindset.
BTW, the recent language changes - decorators, conditional expressions
and with statements are not absolute either. That did not stop them
from being welcome additions.
Ravi Teja
> Ravi Teja wrote:
>
> > People have however written various language interpreters (Scheme,
> > Forth and yes, even Basic) in Python, just for kicks. Still does not
> > make it a DSL language anymore than it makes C a DSL language.
> >
> > At present, the closest thing to writing a DSL in Python is Logix
> > http://livelogix.net/logix/
> > Too bad though, the project is defunct and there has never been enough
> > interest in it.
>
> You might be interested in EasyExtend:
>
> http://www.fiber-space.de/EasyExtend/doc/EE.html
Your framework does look very interesting and might just be what I am
looking for. Will give it a try.
Thanks.
Diez B. Roggisch
>> there are plenty of tools that implement such transformations efficiently ?
>
> Efficiently enough for dynamic (runtime) use ?
Using XML-transformation for AST manipulation isn't my first choice
either - yet efficiency concerns aren't really the point here - after
all we're talking about generating code, which would be pretty useless
if the work was to be done by the transformation instead of that very
code generated ...
So the question is: do XML/XSL give an advantage here? As I said - I
personally don't think so, IMHO a standard reducer using a decent
visitor is easy enough and works well. But your (or better Fredrik's) MMV.
Diez
Ian Bicking
> Fredrik Lundh wrote:
> > Ravi Teja wrote:
> >
> > > Web frameworks, which seem to be the rage now in Python community could
> > > have benefited tremendously from Macro capabilities since they have a
> > > lot of boiler plate.
> >
> > they do? methinks you haven't done much web programming lately...
> >
> > </F>
>
> You blogged on Django. Let's use that. Don't you think model creation
> in Django can be represented better, given that it is done often
> enough?
Actually, no, it's not done that much. Creating and managing tables
isn't something done lightly. It's essential to building a new
application, but (at least in my experience, in similar systems) the
database models stabalize early and you don't spend that much time with
them. Especially not with the DSL aspects. I add and remove methods
often, but I am loathe to add and remove columns.
Now, this might seem like I'm being pedantic, but in my experience lots
of seemingly obvious DSLs end up not being that obvious. XML
generation, for instance. It's nice to have a good syntax -- and you
can get a pretty good syntax in Python (e.g., HTMLGen, stan, etc). But
efforts that go further are generally misplaced, because it's actually
not a very hard or common thing to do, even when you are slinging
around lots of XML.
Or... maybe to be more specific, the hard work later on goes into
*code*. If you are enhancing your model, you do so with methods on the
model classes, and those methods don't effect the DSL, they are just
"code". You create some raw XML in the beginning, but quickly it's
just a matter of gluing those pieces together, using functions instead
of DSLs, and that's just "code".
> Let's take an example from the official tutorial
> from
> http://www.djangoproject.com/documentation/tutorial1/#creating-models
>
> class Poll(models.Model):
> question = models.CharField(maxlength=200)
> pub_date = models.DateTimeField('date published')
>
> class Choice(models.Model):
> poll = models.ForeignKey(Poll)
> choice = models.CharField(maxlength=200)
> votes = models.IntegerField()
>
> I don't use Django and I made this up quickly, so please don't pick on
> subtleties.
>
> @Poll:
> question: char length 200
> pub_date('date published'): date
>
> @Choice:
> poll -> Poll
> choice: char length 200
> votes: int
That doesn't look that much better. How do you create it
programmatically? I know how to pass a variable to
CharField(maxlength=200); can I pass a variable to "char length 200"
just as easily? Can I use **kw? Can I read it from a CSV file and
construct the class that way? Maybe, but only by recreating all the
native patterns that I can infer easily looking at the Django class.
> The following is my rationale. Annoted variables, symbols and code
> layout visually cue more efficiently to the object nature than do
> explicit text definitions. Of course, this is only sensible when there
> aren't too many of any of those. In that case, the cognitive cost of
> notation outweighs the representational cost of text.
Words are great. Python is light on symbols, and that is good. Python
is not perfect when it comes to expressing data structures (the more I
think about it, the more PEP 359 grows on me), but real DSLs are
questionable to me.
Even the Lisps stick to an incredibly homogenous syntax (far more
homogeneous than Python) to make macros feel familiar.
> Representational minimalism is troublesome in general code (ala Perl),
> but not so in a DSL where the context is constrained.
Constrained context is a step backward! How do you add methods? How
do you do looping? How do you write *code*? If you aren't going to
allow those things, then just make a parser and build the structure
from the file, and make it a DSL implemented entirely external to
Python. That's completely okay, though in my experience it's not very
satisfying for something like a model definition (see MiddleKit for an
example of an ORM that doesn't use Python code).
Ian
K.S.Sreeram
> because lots of people know how to describe XML transformations, and
> there are plenty of tools that implement such transformations efficiently ?
>
XML Transformations (XSLT) would *certainly* be an overkill here.
They've invented a whole new declarative programming language, and we
certainly don't need that when we've got Python! :)
XML by itself feels completely out of place in this context. What we
need is, just a flexible, easy to manipulate, in-memory tree structure
(AST) for the Python source.
Regards
Sreeram
Kay Schluehr
> > I don't use Django and I made this up quickly, so please don't pick on
> > subtleties.
> >
> > @Poll:
> > question: char length 200
> > pub_date('date published'): date
> >
> > @Choice:
> > poll -> Poll
> > choice: char length 200
> > votes: int
>
> That doesn't look that much better. How do you create it
> programmatically? I know how to pass a variable to
> CharField(maxlength=200); can I pass a variable to "char length 200"
> just as easily? Can I use **kw? Can I read it from a CSV file and
> construct the class that way? Maybe, but only by recreating all the
> native patterns that I can infer easily looking at the Django class.
If it is just a different kind of representation of common data
structures as in YAML the answer might be a translation of these
declarative blocks into dicts/lists ( or derivatives of those ) at
compile time. The underlying semantics would be that of an "implicitely
embedded DSL" ( there are quite a lot in Python ). Enabling code
generation would just make them more explicit. For example XML syntax
could be considered as an alternate surface syntax for elementrees. XML
elements in Python code might be translated to aequivalent elementree
annotation syntax at compile time.
Under this considerations "choice: char length 200" and
"CharField(maxlength = 200 )" are essentially the same thing. I guess
@Choice.choice would finally represented by a property.
Regards,
Kay
Fredrik Lundh
> If it is just a different kind of representation of common data
> structures
but how do you know ?
</F>
Laurent Pointal
> Laurent Pointal wrote:
>
>>>> The idea is to have a way to transform a Python (.py) module into XML
>>>> and then do source code manipulations in XML-space using ElementTree.
>>>
>>> My my my... I'm not against the idea of dynamic source code
>>> transformation, but for heaven's sake, *why* would one put XML in the
>>> mix ???????
>
> because lots of people know how to describe XML transformations, and
> there are plenty of tools that implement such transformations efficiently ?
>
>> even if inefficient for this problem.
>
> why would XML be inefficient ?
As a storage tool, its nice, and as you says, there are many tools to
deal with.
But as an internal representation for an AST, I certainly prefer an
ad-hoc classes definitions with well defined members.
Just have a wrapper between both representations.
Laurent
A+
Kay Schluehr
The semantics is specified by the syntax transformer so it is actually
compile-time semantics relative to the base language Python . For any
custom statement/expression ( expressed by a production rule / node in
the parse-tree ) one or more target statements/expressions in standard
Python are created. The specification of the with-statement in PEP 343
can be regarded as a good example of this definition practice. The
with-statement is expanded to a "protocol" that can be expressed in
Python 2.4. In a more general case this expansion might involve
additional libraries e.g. ctypes or elementree.
bruno at modulix
>>> because lots of people know how to describe XML transformations, and
>>> there are plenty of tools that implement such transformations
>>> efficiently ?
>>
>>
>> Efficiently enough for dynamic (runtime) use ?
>
>
> Using XML-transformation for AST manipulation isn't my first choice
> either - yet efficiency concerns aren't really the point here - after
> all we're talking about generating code,
I thought we were talking about *transforming* code - just like one uses
metaclasses to transform a class definition, or @decorators to transform
a function definition...
(snip)
Diez B. Roggisch
> Diez B. Roggisch wrote:
>>>> because lots of people know how to describe XML transformations, and
>>>> there are plenty of tools that implement such transformations
>>>> efficiently ?
>>>
>>>
>>> Efficiently enough for dynamic (runtime) use ?
>>
>>
>> Using XML-transformation for AST manipulation isn't my first choice
>> either - yet efficiency concerns aren't really the point here - after
>> all we're talking about generating code,
>
> I thought we were talking about *transforming* code - just like one uses
> metaclasses to transform a class definition, or @decorators to transform
> a function definition...
Yes we were. So where does the runtime efficiency you mention come in to
play?
While the _result_ of a transformation might be a less efficient piece of
code (e.g. introducing a lock around each call to enable concurrent
access), the transformation itself is very - if not totally - static - and
usually only run once.
So except from a start up latency, it has no impact. So if for whatever
reason XSLT is someones favorite method of AST-transformation because it
fits her mindset - perfect. As I said: it wouldn't be mine either, but I
can't see your concerns about efficiency.
And XSLT certainly is suited for tree manipulation, so it might be that it
would be good for e.g. recursivly stripping type annotations of some kind
(think of e.g. type-verifying decorators that you want to get rid of for
production.)
Diez
bruno at modulix
> bruno at modulix wrote:
>
>
>>Diez B. Roggisch wrote:
>>
>>>>>because lots of people know how to describe XML transformations, and
>>>>>there are plenty of tools that implement such transformations
>>>>>efficiently ?
>>>>
>>>>
>>>>Efficiently enough for dynamic (runtime) use ?
>>>
>>>
>>>Using XML-transformation for AST manipulation isn't my first choice
>>>either - yet efficiency concerns aren't really the point here - after
>>>all we're talking about generating code,
>>
>>I thought we were talking about *transforming* code - just like one uses
>>metaclasses to transform a class definition, or @decorators to transform
>>a function definition...
>
>
> Yes we were. So where does the runtime efficiency you mention come in to
> play?
class transformations via metaclasses and function wrapping does happen
at runtime - when the class or (decorated) def statements are eval'd.
This is not the same as having a distinct preprocessing phase that would
write a new .py file.
> While the _result_ of a transformation might be a less efficient piece of
> code (e.g. introducing a lock around each call to enable concurrent
> access), the transformation itself is very - if not totally - static -
really ?
> and
> usually only run once.
Nope, it's runned each time the module is loaded (with 'loaded' distinct
from 'imported') - which can make a real difference in some execution
models...
> So except from a start up latency, it has no impact.
Having a high startup latency can be a problem in itself.
But the problem may not be restricted to startup latency. If for example
you use a metaclasse and a function that *dynamically* creates new
classes using this metaclass, then both the class statement and the
metaclass code transformation will be executed on each call to this
function.
The whole point of a code transformation mechanism like the one Anton is
talking about is to be dynamic. Else one just needs a preprocessor...
> So if for whatever
> reason XSLT is someones favorite method of AST-transformation because it
> fits her mindset - perfect. As I said: it wouldn't be mine either, but I
> can't see your concerns about efficiency.
cf above.
> And XSLT certainly is suited for tree manipulation, so it might be that it
> would be good for e.g. recursivly stripping type annotations of some kind
> (think of e.g. type-verifying decorators that you want to get rid of for
> production.)
>
> Diez
Diez B. Roggisch
>> code (e.g. introducing a lock around each call to enable concurrent
>> access), the transformation itself is very - if not totally - static -
>
> really ?
See below.
> Nope, it's runned each time the module is loaded (with 'loaded' distinct
> from 'imported') - which can make a real difference in some execution
> models...
I already mentioned that latency. If it for whatever reason really becomes
important, it would be the best to cache the result of the transformation.
Which would BTW eliminate any complexity driven runtime penalty -
regardless of the tool used. So - loading time is _not_ an issue. And I
spare you the premature optimization babble... :)
>> So except from a start up latency, it has no impact.
>
> Having a high startup latency can be a problem in itself.
See above.
> But the problem may not be restricted to startup latency. If for example
> you use a metaclasse and a function that *dynamically* creates new
> classes using this metaclass, then both the class statement and the
> metaclass code transformation will be executed on each call to this
> function.
This is an assumption I don't agree upon. The whole point of the OPs post
was about creating DSLs or alter the syntax of python itself. All that to
enhance expressiveness.
But we are still talking about CODE here - things that get written by
programmers. Even if that is piped through so many stages, it won't grow
endlessly.
Runtime (runtime meaning here not on a startup-phase, but constantly/later)
feeding of something that generates new code - I wouldn't say that is
unheard of, but I strongly doubt it occurs so often that it rules out tree
transformations that don't try and squeeze the latest bit of performance
out themselves. Which, BTW, would rule out python in itself as nothing
beats runtime assembly generation BY assembly. Don't you think?
> The whole point of a code transformation mechanism like the one Anton is
> talking about is to be dynamic. Else one just needs a preprocessor...
No, it is not the whole point. The point is
""
The idea is that we now have a fast parser (ElementTree) with a
reasonable 'API' and a data type (XML or JSON) that can be used as an
intermediate form to store parsing trees. Especially statically typed
little languages seem to be very swallow-able. Maybe I will be able to
reimplement GFABasic (my first love computer language, although not my
first relationship) someday, just for fun.
"""
No on-the-fly code generation here. He essentially wants lisp-style-macros
with better parsing. Still a programming language. Not a data-monger.
Diez
Boris Borcic
> Anton Vredegoor wrote:
>> bruno at modulix wrote:
>>
>>> I still don't get the point.
>>
>> Well, I've got to be careful here, lest I'd be associated with the
>> terr.., eh, the childp..., eh the macro-enablers.
>>
>> The idea is to have a way to transform a Python (.py) module into XML
>> and then do source code manipulations in XML-space using ElementTree.
>
> My my my... I'm not against the idea of dynamic source code
> transformation, but for heaven's sake, *why* would one put XML in the
> mix ???????
If a good transform could "reveal" xml as "python's s-expression syntax", not
only would source2source transform using xml t2t transform tools be facilitated,
but generally speaking it would be easier for "xml retro-coded" python source to
find its way through xml-enabled tools&chains. And incite developpers of such
tools to consider python a better candidate (eg than it currently is) whenever
the matter of scripting the tool comes up.
Anton Vredegoor
<...>
>> The whole point of a code transformation mechanism like the one Anton is
>> talking about is to be dynamic. Else one just needs a preprocessor...
>
> No, it is not the whole point. The point is
>
> ""
> The idea is that we now have a fast parser (ElementTree) with a
> reasonable 'API' and a data type (XML or JSON) that can be used as an
> intermediate form to store parsing trees. Especially statically typed
> little languages seem to be very swallow-able. Maybe I will be able to
> reimplement GFABasic (my first love computer language, although not my
> first relationship) someday, just for fun.
> """
>
> No on-the-fly code generation here. He essentially wants lisp-style-macros
> with better parsing. Still a programming language. Not a data-monger.
The 'problem' is that a lot of incredibly smart people are reading and
replying here who are seeing a lot more into my post than I was prepared
for :-)
Anyway, the last few weeks I have been busy transforming MsWord
documents into XML using Open Office, and next parsing this XML and
transforming it into a special subset of HTML using ElementTree's
XMLWriter class.
Then the output of the XMLWriter was put into a Zope/Plone page but I
added special markup for footnotes, making them plone objects that could
be separately edited, and I added image tags for images that were
retrieved from a separate server using an XSLT script.
To accomplish that a special zope parser was written to recognize my
nonstandard footnote and image tags, and to create the necessary
objects, and to insert them into the page.
After that I came across some turbogears code (which is stacking code at
different levels like it were those things you put under your beer
glass) and still later I saw some JSON equivalents of XML. JSON looks a
lot like Python dicts which makes it seem likely that javascript will be
able to interface with Python more efficiently.
Remember that ElementTree comes from the same place that brought us PIL
which is a package that can transform images into different types.
So if we can transform documents, images and XML, why not sourcecode?
Especially if it's not a conversion into a 'lossy' file format, (I
consider dynamically typed code versus statically typed code the analog
thing to JPEG versus bitmaps) it would be easy to convert all datatypes
into the datatypes of another language, thereby making it possible to
exchange code between languages. Algorithms just being things that
convert sets of data-objects into other sets of data-objects.
Now if one would equate standardized code exchange between languages and
within a language with macros then I guess there is nothing left for me
to do but wait till a certain google bot comes knocking at my ip-address
port 80 and transfers me to the google equivalent of Guantanamo.
But the whole point of distinguishing macros from official language
structures *is* standardization, as some other clever poster already
pointed out, so it would be extremely unfair to equate trans-language
standardized code exchange with the guerrilla type macro activities that
are plaguing the Lisp community.
Then there are some people who keep insisting they don't understand what
I'm talking about until I simplify things enough to get them on-board,
but then simply dismiss my ideas with 'you can already do that easily
with this standard python construct'. This strategy was also eloquently
refuted by some other poster, so I don't need to repeat it :-)
I've gotten a lot of things to think about, so thanks all for your
thoughts, but since this is getting way above my head I'll just wimp out
and leave the rest of the thread to the experts!
Regards,
Anton
Bruno Desthuilliers
> Diez B. Roggisch wrote:
>
> <...>
>
>>> The whole point of a code transformation mechanism like the one Anton is
>>> talking about is to be dynamic. Else one just needs a preprocessor...
>>
>>
>> No, it is not the whole point. The point is
>> ""
>> The idea is that we now have a fast parser (ElementTree) with a
>> reasonable 'API' and a data type (XML or JSON) that can be used as an
>> intermediate form to store parsing trees. Especially statically typed
>> little languages seem to be very swallow-able. Maybe I will be able to
>> reimplement GFABasic (my first love computer language, although not my
>> first relationship) someday, just for fun.
>> """
>>
>> No on-the-fly code generation here. He essentially wants
>> lisp-style-macros
>> with better parsing. Still a programming language. Not a data-monger.
>
>
> The 'problem' is that a lot of incredibly smart people are reading and
> replying here who are seeing a lot more into my post than I was prepared
> for :-)
no comment...
(snip various transformations examples)
>
> So if we can transform documents, images and XML, why not sourcecode?
>
(snip preliminary precautions)
> it would be easy to convert all datatypes
> into the datatypes of another language, thereby making it possible to
> exchange code between languages.
You mean like 'converting' javascript to python or python to ruby (or
converting any home-grown DSL to Python, etc) ?
> Algorithms just being things that
> convert sets of data-objects into other sets of data-objects.
>
> Now if one would equate standardized code exchange between languages and
> within a language with macros then I guess there is nothing left for me
> to do but wait till a certain google bot comes knocking at my ip-address
> port 80 and transfers me to the google equivalent of Guantanamo.
Lol !-)
Well, given this quote from another of your posts:
"""
The idea is to have a way to transform a Python (.py) module into XML
and then do source code manipulations in XML-space using ElementTree.
"""
I effectively understood something like a python to python
transformation, which of course led me to something very very like macros.
> But the whole point of distinguishing macros from official language
> structures *is* standardization, as some other clever poster already
> pointed out, so it would be extremely unfair to equate trans-language
> standardized code exchange with the guerrilla type macro activities that
> are plaguing the Lisp community.
>
> Then there are some people who keep insisting they don't understand what
> I'm talking about until I simplify things enough to get them on-board,
count me in then :(
> but then simply dismiss my ideas with 'you can already do that easily
> with this standard python construct'. This strategy was also eloquently
> refuted by some other poster, so I don't need to repeat it :-)
>
> I've gotten a lot of things to think about, so thanks all for your
> thoughts, but since this is getting way above my head I'll just wimp out
> and leave the rest of the thread to the experts!
No way you will escape from your responsabilities so easily !-)
Ravi Teja
> *code*. If you are enhancing your model, you do so with methods on the
> model classes, and those methods don't effect the DSL, they are just
> "code". You create some raw XML in the beginning, but quickly it's
> just a matter of gluing those pieces together, using functions instead
> of DSLs, and that's just "code".
> That doesn't look that much better. How do you create it
> programmatically? I know how to pass a variable to
> CharField(maxlength=200); can I pass a variable to "char length 200"
> just as easily? Can I use **kw? Can I read it from a CSV file and
> construct the class that way? Maybe, but only by recreating all the
> native patterns that I can infer easily looking at the Django class.
I am looking at it from the cognitive perspective. You are looking at
it from the compiler perspective.
I think you are talking about full blown DSLs like SQL which try to be
self contained for a given domain. The ones I am referring are only
thin layers on Python.
> Words are great. Python is light on symbols, and that is good.
Agreed. When I came to Python from Perl, I loved the clean syntax.
Scalars, arrays, hashes occur too frequently in Perl code that using
symbols to denote them causes more noise than cognitive assistance. On
the other hand, using symbols to denote an occational special construct
is helpful (as in decorators).
> Even the Lisps stick to an incredibly homogenous syntax (far more
> homogeneous than Python) to make macros feel familiar.
Yes! The parser friendly, "everything is a list" syntax does help. I
did consider that to be an essential feature to enable dynamic macros.
However I changed my mind when I saw Logix macro syntax.
> Constrained context is a step backward! How do you add methods? How
> do you do looping? How do you write *code*? If you aren't going to
> allow those things, then just make a parser and build the structure
> from the file, and make it a DSL implemented entirely external to
> Python. That's completely okay, though in my experience it's not very
> satisfying for something like a model definition (see MiddleKit for an
> example of an ORM that doesn't use Python code).
I agree that constrained context is a step back in terms flexibility.
But it is a strategic step backwards, in this case to trade for
representational benefits. The extent of constraints is a judgement
call. And proof of utility can only be emperical.
However I think that you are seeing my sample differently than I meant
it. I did not mean to create a special syntax file that would be parsed
as a text file such that it would loose all the benefits of Python. It
is just a thin layer over Python code for specific representational
benefits. Kay Schluehr does a good job of identifying it as such in his
reply.
Ian Bicking
> > Or... maybe to be more specific, the hard work later on goes into
> > *code*. If you are enhancing your model, you do so with methods on the
> > model classes, and those methods don't effect the DSL, they are just
> > "code". You create some raw XML in the beginning, but quickly it's
> > just a matter of gluing those pieces together, using functions instead
> > of DSLs, and that's just "code".
>
> > That doesn't look that much better. How do you create it
> > programmatically? I know how to pass a variable to
> > CharField(maxlength=200); can I pass a variable to "char length 200"
> > just as easily? Can I use **kw? Can I read it from a CSV file and
> > construct the class that way? Maybe, but only by recreating all the
> > native patterns that I can infer easily looking at the Django class.
>
> I am looking at it from the cognitive perspective. You are looking at
> it from the compiler perspective.
I don't think that distinction is very meaningful. As a programmer I
have to understand both. I have to be able to synthesize correct
expressions. I need to understand the compiler. I understand the
Python compiler well, and it gives me reasonably good feedback when I
get things wrong, and it has a lot of flexibility along several
orthogonal lines.
We're talking about programming languages, so it's useless to consider
a cognitive perspective without considering the compiler perspective.
> I think you are talking about full blown DSLs like SQL which try to be
> self contained for a given domain. The ones I am referring are only
> thin layers on Python.
I understand you, but it's very unclear to me how you can make a thin
layer that's not horribly leaky or stunted. In my experience one or
both are likely in DSLs, and the result is horrible and only useful as
a toy.
If you are really serious about the implementation, sure. But DSL
almost screams out that it lacks seriousness, and a long-term
commitment to debuggability, generality, and documentation.
> However I think that you are seeing my sample differently than I meant
> it. I did not mean to create a special syntax file that would be parsed
> as a text file such that it would loose all the benefits of Python. It
> is just a thin layer over Python code for specific representational
> benefits. Kay Schluehr does a good job of identifying it as such in his
> reply.
I understood the distinction you were making. But you were also
speaking generally about generally programmable syntax, and I don't
think that's a good idea, and it's very unclear how that mixes with
Python. You can always do:
model = make_model("""
my funny syntax
""")
And you actually get something that can be a close peer to normal
Python code. But if you are talking about Python and some novel syntax
interleaved, then the details seem to matter a lot, because the
implementation is substantial and potentially invasive.
There are also very *specific* things that can be discussed that are
more conventional, and driven as much by the ease of implementation as
expressiveness. I think that is much more productive, because I think
the general case is a bad idea. The 'make' syntax is an example of
this.
Ian
Anton Vredegoor
> You mean like 'converting' javascript to python or python to ruby (or
> converting any home-grown DSL to Python, etc) ?
Yes, but also what some other posters mentioned, making Pythons internal
parsing tree available to other programs (and to Python itself) by using
a widely used standard like XML as its datatype.
>> Then there are some people who keep insisting they don't understand what
>> I'm talking about until I simplify things enough to get them on-board,
>
> count me in then :(
Sorry about that.
>> but then simply dismiss my ideas with 'you can already do that easily
>> with this standard python construct'. This strategy was also eloquently
>> refuted by some other poster, so I don't need to repeat it :-)
>>
>> I've gotten a lot of things to think about, so thanks all for your
>> thoughts, but since this is getting way above my head I'll just wimp out
>> and leave the rest of the thread to the experts!
>
> No way you will escape from your responsabilities so easily !-)
Ok, count me back in then too :-) Of course I will be available for
further discussion. If more than ten people demand a PEP and no better
champion is available (very unlikely) I'll even write a proposal.
Anton
Paul Boddie
> Bruno Desthuilliers wrote:
>
> > You mean like 'converting' javascript to python or python to ruby (or
> > converting any home-grown DSL to Python, etc) ?
>
> Yes, but also what some other posters mentioned, making Pythons internal
> parsing tree available to other programs (and to Python itself) by using
> a widely used standard like XML as its datatype.
http://pysch.sourceforge.net/ast.html
I was going to write a long reply to one of your previous messages, but
the above link references a project which may intersect with some of
your expectations. Meanwhile, it should be noted that the availability
of Python AST processing tools is not a recent thing: the compiler
module has been around for a long time, and it is possible to modify
the AST and to generate bytecode from it; my own experiments have
centred on producing other representations from the AST, and other more
successful projects (eg. ShedSkin) produce other languages (eg. C++)
from the AST.
Paul
Ravi Teja
> have to understand both.
> get things wrong, and it has a lot of flexibility along several
> orthogonal lines.
> We're talking about programming languages, so it's useless to consider
> a cognitive perspective without considering the compiler perspective.
As some one who has followed your excellent articles, posts and tools
over the last many years, I don't doubt your expertise at all.
I admit I am not be entirely looking at this as a programmer. I am
working in informatics for the last few years. At least in this domain,
it is useful to seperate the perspectives. Sort of like designing a
functional language based primarily on mathematical abstractions rather
than imperative realities. Yes! Once the implementation is underway,
the practical realities are unavoidable but they interfere in thinking,
otherwise.
> I understand you, but it's very unclear to me how you can make a thin
> layer that's not horribly leaky or stunted. In my experience one or
> both are likely in DSLs, and the result is horrible and only useful as
> a toy.
I am not making this up out of thin air. There is already an
implementation, Logix, which allows you make thin layer DSLs on top of
itself (which in turn sits on top of Python). It did not fly but I
could not technically fault it (except maybe that it is not very
optimized at the moment). Maybe with your better understanding of
language implementations, you can.
> I understood the distinction you were making. But you were also
> speaking generally about generally programmable syntax, and I don't
> think that's a good idea, and it's very unclear how that mixes with
> Python. You can always do:
>
> model = make_model("""
> my funny syntax
> """)
>
> And you actually get something that can be a close peer to normal
> Python code. But if you are talking about Python and some novel syntax
> interleaved, then the details seem to matter a lot, because the
> implementation is substantial and potentially invasive.
NO! I have a feeling that you are interpreting me as a language
radical. I do not think that Python syntax should have too many
novelties. Currently Python feels a cohesive whole. A well designed
entity rather than something that evolved by accretion of features like
Perl. With active attempts to prune inconsistencies for that end
(Python 3000). And I value all that.
But Logix does not add ANY new syntactic constructs to Python. There is
a special import function that imports a macro supporting Python
language variant module where one can create special syntactic
constructs. I am quite satisfied with the macro capabilities of Logix.
But I am just grumbling that the Python community does not find macros
interesting. But without enough users, it simply dies away.
Fredrik Lundh
> You blogged on Django. Let's use that. Don't you think model creation
> in Django can be represented better, given that it is done often
> enough?
nope, because 1) it's not done very often, and 2) the existing syntax is
already very minimal, and defined in terms of a language that I
already understand.
there might be cognitive theories that argue that the length of the
symbols used to describe something is more important than the symbols
you use, and how they can be "chunked" by the brain, but sturgeon's law
applies to cognitive scientists too ;-)
> Since you are on thread and are a prominent and involved member of the
> Python community, I would like it if you (or any such other) can
> provide feedback on the rest of my previous post rather than be
> dismissive by just a small portion of it. Perhaps, that will give me
> some insight how these language design decisions are rationally made (I
> am not strictly a programmer by profession, much less a language
> designer).
see Ian's posts for some excellent discussion.
</F>
Anton Vredegoor
> Anton Vredegoor wrote:
>> Yes, but also what some other posters mentioned, making Pythons internal
>> parsing tree available to other programs (and to Python itself) by using
>> a widely used standard like XML as its datatype.
>
> http://pysch.sourceforge.net/ast.html
Very interesting, it led me to some sxml describing pages and it also
tricked me into reading some Python documentation that I had always
considered to be hiding some arcane deep Python magic. I guess now that
Python is officially entering tree territory (as opposed to providing
third party functionality) it seems unavoidable that Python's officially
endorsed tree datatype will also be used for some of its internal
structures, thereby making it more accessible to programmers like me and
to outside programmers.
> I was going to write a long reply to one of your previous messages, but
> the above link references a project which may intersect with some of
> your expectations. Meanwhile, it should be noted that the availability
Somehow I get the impression of getting away with my posts luckily,
while you now are duping other interested readers into not reading your
innermost feelings about this subject. Let's get it in the open, don't
spare me :-)
> of Python AST processing tools is not a recent thing: the compiler
> module has been around for a long time, and it is possible to modify
> the AST and to generate bytecode from it; my own experiments have
> centred on producing other representations from the AST, and other more
> successful projects (eg. ShedSkin) produce other languages (eg. C++)
> from the AST.
Well maybe this trick of vehemently denying the existence of something
on Usenet worked again, by materializing the thing as a reaction.
However, I knew of the existence of such languages but I am mostly
interested in standardized code interchange, like for example with JSONP
which fetches some external javascriptcode from another server using
JSON and places the translated javascript into a webpage at the request
of the clients browser or so it seems. Maybe a Python webserver could
also emit pieces of javascript code by getting them from a *Python* code
library after translating Python code on the fly?
That would open up the web to Python programmers without browsers
needing to understand Python. Like Jython, but now as separately
distributed functions from different servers.
Anton
Kay Schluehr
Anton Vredegoor wrote:
> Paul Boddie wrote:
>
> > Anton Vredegoor wrote:
>
> >> Yes, but also what some other posters mentioned, making Pythons internal
> >> parsing tree available to other programs (and to Python itself) by using
> >> a widely used standard like XML as its datatype.
> >
> > http://pysch.sourceforge.net/ast.html
>
> Very interesting, it led me to some sxml describing pages and it also
> tricked me into reading some Python documentation that I had always
> considered to be hiding some arcane deep Python magic. I guess now that
> Python is officially entering tree territory (as opposed to providing
> third party functionality) it seems unavoidable that Python's officially
> endorsed tree datatype will also be used for some of its internal
> structures, thereby making it more accessible to programmers like me and
> to outside programmers.
I had the same initial impression with the "deep" aspects of Python
parser API but I learned soon that I just stumbled about an awkward
kind parse-tree representation. The structures are as simple as they
could be - due to the nice LL(1) Python grammar which are reflected.
I think node manipulation is just a first basic step for defining
source transformers i.e. they provide a sound basis. For usability
purposes a template language is a far better solution.
Here is my own attempt. Defining a repeat statement ( for demonstration
purposes ) using EasyExtend requires following translation:
repeat:
<suite>
until:
<test>
==>
while 1:
<suite>
if <test>:
break
The placeholders <suite> and <test> correspond to nodes in the
parse-tree ( and constants in symbol.py ). The repeat-stmt is defined
by an added grammar rule. The corresponding node in the parse-tree is
dispatched against Transformer.handle_repeat_stmt() during parse-tree
traversal. Matching against <suite> and <test> in the repeat_stmt node
is done by applying:
suite_node = find_node(repeat_node, symbol.suite)
test_node = find_node(repeat_node, symbol.test, level=1)
What becomes tedious is the creation of the corresponding while_stmt
node which is the expected result of the transformation. I actually
want to use the declaration as it is written above and insert the
suite_node and the test_node where the placeholders <suite> and <test>
are defined. Using EasyExtend one can define an extension language
where the placeholders ( and some enhanced versions of those ) become
language elements! Given this one can put everything together:
import macro # module that corresonds to the macro extension
language
class FastTransformer(Transformer):
def handle_repeat_stmt(self, repeat_node):
suite_node = find_node(repeat_node, symbol.suite)
test_node = find_node(repeat_node, symbol.test, level=1)
target_stmt = """
while 1:
<suite>
if <test>:
break
"""
while_node = macro.expand(target_stmt,
{'suite': suite_node,
'test': test_node},
goal =
symbol.while_stmt)
return any_stmt(while_node)
This is pretty safe and works recursively: if the <suite> node contains
another <repeat_stmt> node it will be replaced as well.
Max Erickson
>
> However, I knew of the existence of such languages but I am
> mostly interested in standardized code interchange, like for
> example with JSONP which fetches some external javascriptcode
> from another server using JSON and places the translated
> javascript into a webpage at the request of the clients browser
> or so it seems. Maybe a Python webserver could also emit pieces
> of javascript code by getting them from a *Python* code library
> after translating Python code on the fly?
>
> Anton
I have no idea how close it is to what you are talking about, but pypy
does have some sort of python->javascript support:
http://codespeak.net/pypy/dist/pypy/doc/getting-
started.html#translating-the-flow-graph-to-javascript-code
About two thirds down, if the above link is broken:
http://codespeak.net/pypy/dist/pypy/doc/getting-started.html
max
Bruno Desthuilliers
(snip)
> However, I knew of the existence of such languages but I am mostly
> interested in standardized code interchange, like for example with JSONP
> which fetches some external javascriptcode from another server using
> JSON and places the translated javascript into a webpage at the request
> of the clients browser or so it seems.
This is AJAX with JSON instead of XML (should we call this AJAJ ?-).
It's quite handy, since it saves both the extra bits to be transfered
(XML is way much verbose than JSON) and the XML to javascript parsing.
> Maybe a Python webserver could
> also emit pieces of javascript code by getting them from a *Python* code
> library after translating Python code on the fly?
If you restrict this on data, it's already done (Turbogears uses this
for AJAX).
> That would open up the web to Python programmers without browsers
> needing to understand Python. Like Jython, but now as separately
> distributed functions from different servers.
>
> Anton
Paul Boddie
> Paul Boddie wrote:
> >
> > the above link references a project which may intersect with some of
> > your expectations. Meanwhile, it should be noted that the availability
>
> Somehow I get the impression of getting away with my posts luckily,
> while you now are duping other interested readers into not reading your
> innermost feelings about this subject. Let's get it in the open, don't
> spare me :-)
I was just going to say that your Web application could have used XSLT
more or less exclusively if the data arrives in XML (and obviously
leaves in some kind of XML), provided you aren't doing exotic
processing on the data.
[...]
> Well maybe this trick of vehemently denying the existence of something
> on Usenet worked again, by materializing the thing as a reaction.
My intention was to correct the perception that before a bunch of work
was done on exposing the Python AST (the so-called "AST branch") there
weren't any tools to get nice ASTs which could be used to generate
code. Of course there are such tools, with the compiler module being in
the standard library itself - not to be confused with the parser module
which everyone seems to mention and about whose representation everyone
seems to complain, but that's also obviously in the standard library
too.
I'm not sure about the ready availability of Python-to-Python
transformers, though. I have made HTML "reports" of Python programs -
annotated source code with type information in CSS-based pop-ups - and
a plain text serialiser for the compiler module's AST wouldn't be as
difficult as that. The hardest part of working with the compiler API
would be in the modification of the AST, and perhaps it is in that area
that the "AST branch" shows promise.
[...]
> Maybe a Python webserver could also emit pieces of javascript code by
> getting them from a *Python* code library after translating Python
> code on the fly?
http://subway.python-hosting.com/file/crackajax/trunk/crackajax.py
I personally don't go for this kind of thing, but people have given it
some thought.
Paul
Ravi Teja
Fredrik Lundh wrote:
> there might be cognitive theories that argue that the length of the
> symbols used to describe something is more important than the symbols
> you use and how they can be "chunked" by the brain
Expert communication is known to work differently. For example, take a
doctor's note. It's minimalist representation and domain specific
notational standardizations are absolute gibberish for a reader without
the knowledge of medicine (or more specifically, the sub-specialty) and
the context of the patient. But it is very efficient for the audience
it is intended for. Now, the hand writing - that's a wholly different
story :-).
BTW, I am studying clinical information representations. This
discussion was interesting to me because I am trying to see programming
language representation as a better discussed and more structured
microcosm of the relatively fuzzy domain I am studying. Of course,
there are many important differences and I may be drawing the parallels
liberally at this point but it gives me an interesting perspective.