Porting Mako to IronPython
Eloff
guys there were very hostile towards IronPython, and not interested in
helping me, so I decided to work on Mako instead (which I prefer as a
template engine anyway)
Kudos for using plain old UnitTest, it was easy to run the test suite
under IronPython. Of course almost nothing passes, but mostly because
_ast is missing on IronPython and the compiler package is broken.
So I'm about halfway finished porting _ast to IronPython, and it looks
like I'll be able to pull it off with a nearly perfect compatibility.
The ASTs are a little different, but functionally identical (composed
of identical types, with identical attributes, and identical
inheritance trees)
As I work on this, I'm using class SourceGenerator in mako._ast_utils
to dump asts to Python source, and I'm finding a lot of bugs with it.
How important is this class to Mako? Should I spend time fixing it?
Some things that are broken:
def foo(): return
def foo():
return # note no indent
power expressions (e.g. 2**8)
function decorators
extended slices
I get the feeling I'm just seeing the tip of the iceberg here. I'm not
sure how to fix the indent bug (help here would be great), but I've
fixed power expressions and function decorators. Looking into extended
slices now.
Anyway I'll be in touch as I work on Mako. The good news is IronPython
doesn't seem to need many special code paths *cough* Jython. It often
just works where CPython did.
-Dan
Michael Bayer
This is more pjenvey's area of expertise since he's rewritten all the
ast stuff to work wtih Jython/GAE, but it seems like SourceGenerator
is only used in conjunction with the built in _ast module of py 2.5
+. Otherwise its not used and the source generation is done within
Mako (looking at it I can see that's the part I wrote...its been
awhile).
Eloff
> but it seems like SourceGenerator
> is only used in conjunction with the built in _ast module of py 2.5
> +.
IronPython I guess it will be used there as well. Does pjenvey read
mako-discuss? I suppose he's the guy to talk to about fixing
SourceGenerator.
-Dan
Michael Bayer
right but, basically there's two ExpressionGenerator classes. one
uses SourceGenerator and the other doesn't. Both should in theory be
usable. Also SourceGenerator, for our purposes, really only has to
generate Python expressions. It doesn't need to generate code
blocks. Mako's unit tests are pretty thorough and if you can get
all the lexer/template/compiler/defs tests to pass, and run the
documentation generator, you're in pretty good shape.
Michael Bayer
(Armin's listed as the author....)
Eloff
> right but, basically there's two ExpressionGenerator classes. one
> uses SourceGenerator and the other doesn't. Both should in theory be
> usable. Also SourceGenerator, for our purposes, really only has to
> generate Python expressions. It doesn't need to generate code
> blocks.
bugs with expressions then.
>Mako's unit tests are pretty thorough and if you can get
> all the lexer/template/compiler/defs tests to pass, and run the
> documentation generator, you're in pretty good shape.
it important here?
> heh...seems like _ast_util.py is taken from jinja in the 1st place ?
> (Armin's listed as the author....)
own AST code. A lot of the classes in _ast_utils found their way into
the stdlib in Python 2.6+, see the ast module.
-Dan
Michael Bayer
On Oct 12, 2008, at 12:17 PM, Eloff wrote:
>
>> Mako's unit tests are pretty thorough and if you can get
>> all the lexer/template/compiler/defs tests to pass, and run the
>> documentation generator, you're in pretty good shape.
>
> That's the plan. What's the documentation generator though, and why is
> it important here?
its in doc/build/genhtml.py, and its just another big elaborate set of
templates to make sure they work.
Philip Jenvey
Very cool to hear about your porting work Eloff. Adding _ast support
to IronPython must be no small task.
_ast_utils originally came came from the initial version of Armin's
ast module, just straight from his sandbox repository. Then like you
said, the version of it in 2.6 is stripped down to only the basic
Visitor classes (i.e. no SourceGenerator).
I've fixed a couple bugs in SourceGenerator but I didn't know there
were more. As Mike said mako only uses it to generate a few things,
and its doubtful any of its bugs actually affect mako. I suppose we
should at least add a comment to the code about its flakyness, though.
--
Philip Jenvey
Eloff
>
> Very cool to hear about your porting work Eloff. Adding _ast support
> to IronPython must be no small task.
>
of C#, but it's fast, memory efficient, and it passes everything in
mako's test_ast.py. I made numerous fixes to SourceGenerator, but like
you say most of them were either in statements or in rarely used
expressions (e.g. ExtSlice)
Unfortuantly it had to be implemented as a patch to IronPython, and
they don't yet accept patches. I'll contribute the patch to IPCE, and
people can use it to compile their own. I'll also upload patched
binaries. A patch from IPCE also needed to be applied to inspect.py
from the CPython stdlib.
I'm taking a break from fixing things, but I've got 3/4 of mako's
tests passing now. Most of the remaining errors/failures seem to be
str/unicode issues, I don't see them being a problem.
Do you folks currently run tests on Jython after making changes? Would
you run them on IronPython, assuming I bundled everything needed into
a zip? I'm worried about fixing all these little things and then
coming back in a few months to find dozens more have crept in.
-Dan
Michael Bayer
On Oct 13, 2008, at 5:57 PM, Eloff wrote:
>
> Do you folks currently run tests on Jython after making changes? Would
> you run them on IronPython, assuming I bundled everything needed into
> a zip? I'm worried about fixing all these little things and then
> coming back in a few months to find dozens more have crept in.
I just run against cPython. pjenvey might run against Jython. If
you were maintaining IronPython compatibility, then we'd trust that
you're running tests on that platform.
Eloff
> If you were maintaining IronPython compatibility, then we'd trust that
> you're running tests on that platform.
out breakages, but I don't want to be forever tracking down and fixing
bugs, it's not fun. I don't think it's unreasonable to ask committers
to check that they are not breaking things on IronPython when
committing. By all means if they are having difficulty adapting their
code to IronPython I will be happy to help. To abuse a common
metaphor, I'm happy to teach you guys how to fish, but I'm not going
to stand here handing out fish forever. I'm just wasting my time with
that kind of solution, as soon as I'm unable or uninterested in fixing
mako, my work will all be for naught.
The good news is I've been working with the IronPython guys, and we've
figured out a way to use _ast as a standalone module for IronPython 2
so that it will run with the official distribution. Just drop it in
the dlls directory and you're golden. The bad news is some of these
remaining bugs are difficult to chase down and fix. I've been getting
busier again so I'm trying to fix a minimum of 7 failed tests per
week. At that rate hopefully everything will be working in a month and
we can talk about merging the changes into trunk.
Michael Bayer
On Oct 16, 5:11 pm, Eloff <dan.el...@gmail.com> wrote:
>
> I don't mind running tests on mako on a regular basis, and pointing
> out breakages, but I don't want to be forever tracking down and fixing
> bugs, it's not fun. I don't think it's unreasonable to ask committers
> to check that they are not breaking things on IronPython when
> committing. By all means if they are having difficulty adapting their
> code to IronPython I will be happy to help. To abuse a common
> metaphor, I'm happy to teach you guys how to fish, but I'm not going
> to stand here handing out fish forever. I'm just wasting my time with
> that kind of solution, as soon as I'm unable or uninterested in fixing
> mako, my work will all be for naught.
theory of operation is pretty much nailed down and isn't going to have
any dramatic changes (at most I can see optional C extensions someday,
but probably not even that).
Eloff
> Mako's theory of operation is pretty much nailed down and isn't going to have
> any dramatic changes (at most I can see optional C extensions someday,
> but probably not even that).