Proposal for Template Engine Refactoring

[Armin Ronacher]

Hi,

I would like to participate in this year's Google Summer of Code by
introducing a new backend for the Django (and Jinja2) templating
language.
As the author of Jinja2 I recently decided to rewrite the code
generation
of the engine to better support alternative Python implementations
like
PyPy and restricted environments like Google's Appengine.

The Jinja2 backend is already usable for other templating engines as
well
for as long as the sematics are compatible with the ones from Jinja2.
However with a better backend design the Django semantics should be
possible to implement in there as well.

As such as I am proposing to keep this backend independent of either
Django and Jinja2. I would maintain inside a separate repository and
with
each release of Jinja2 and Django port fixes into their respective
code
bases. This could also be used as a basis for other template engines
or
domain specific languages if necessary.

The reason I don't want to keep this a separate package but actually
copy
the code over to Django and Jinja2 are twofold. First of all I don't
want
to introduce a new dependency for the template engines themselves.
Secondly I really want to keep this an implementation detail for
Jinja2 at
the very least. In Django we don't want to see dependencies either.

Reasons (coming from the current Jinja2 backend) for a new version:

1. implement a drop-in interpreter as alternative to the template
compilation for better error reporting during development.
2. implement an ast.py based compiler in order to avoid the debug
traceback hackery that is currently in place to support debugging
of
templates with accurate line numbers.
3. Better support for jit interpreted Python installations like PyPy
by
removing the need to call locals() or sys._getframe which is
currently
required by Jinja2 in order to implement includes.
4. Python generating compiled for GAE. This one would not have the
same
error handling capabilities as the interpreter and the AST based
compiler, but has the advantage of running everywhere and would
allow
module based importing like the one currently in use in Jinja2.

Reasons why this is interesting for Django:

1. It would greatly speed up execution of core tags. The idea is to
use
code generation for the builtin tags and to generate a context as
necessary when droppping into old-style custom tags.
2. Provide a new API that can be used for new tags that allow code
generation behind the scenes.
3. Better debugging capabilties by providing actual Python tracebacks
for
errors in templates.

How much speedup will we see in Django? Probably not as much as
people
are hoping as Django's current semantics just cannot be sped up to
Jinja2
levels by keeping the extension API unchanged. However because there
are
already some abstractions in place we can probably achieve a
significant
speedup for certain common template types that it reduces the need to
switch to Jinja2.

Why am I proposing this as a GSOC Django project instead of hacking on
this under the umbrella of the PSF for Jinja2 alone? First of all,
speeding up Jinja2 is a boring task right now as it is very close to
raw
Python performance already. Secondly there was a proposal by Alex
last
year and there is already a proposal here about template engine
compilation backends and if I am going to do this for Jinja2 anyways,
I
can keep it generic enough to also support Django.


Regards,
Armin

[Armin Ronacher]

Great. Google groups decided to reflow my mail. Here in actually
readable version: http://paste.pocoo.org/show/362024/

Regards,
Armin

[Jonathan S]

+1 on this.

Are you planning on keeping the API for registering template tags
compatible? (django.template.Library.register.tag)
I think this may be required for backwards-compatibility, but
personally, I'd like to see a cleaner API for this, which doesn't
expose the parser details. (Writing template tags should not involve
any token manipulation.)

[Armin Ronacher]

Hi,

On Mar 29, 2:23 pm, Jonathan S <jonathan.slend...@gmail.com> wrote:
> Are you planning on keeping the API for registering template tags
> compatible?  (django.template.Library.register.tag)

It will be supported because otherwise upgrading won't be possible,
but how well this works in detail would have to be investigated and
what a cleaner implementation looks like. Jinja2 has an expression
parser and encourages people to create a AST that does what they want
in extensions. However in Jinja2 I never encouraged people to write
custom tags so the API is very limited there.


Regards,
Armin

[Jacob Kaplan-Moss]

Hi Armin --

Can you speak a little more about how you'd see the workflow working
with this new library? It seems a bit complex to have a library that's
*copied* between Django and Jinja: I'd worry about patches getting
lost.

For context, this has happened a few times with the simplejson and
elementtree libraries in core Python: patches made to the upstream got
"lost" before merging into Python's stdlib, and patches made against
the stdlib didn't make it upstream. For a while there the json library
in the stdlib was ever so slightly different from the simplejson
library of supposedly the same version.

Do you have ideas about how we could avoid this problem?

(For the record, I wouldn't be opposed to having your proposed library
be an external dependency. At some point the Python packaging
ecosystem will be good enough that we can stop bundling, and I think
we're getting pretty close to that point. But understood if that's a
battle you don't want to fight :)

Jacob

[Armin Ronacher]

Hi,

On Mar 29, 3:54 pm, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:
> Can you speak a little more about how you'd see the workflow working
> with this new library? It seems a bit complex to have a library that's
> *copied* between Django and Jinja: I'd worry about patches getting
> lost.

I think that problem can mostly be avoided. The interpreter /
compiler / ast combo library is probably very minimal in size. I don't
see a big problem having to copy them into the Django svn every once
in a while. That could even be automated.

> For context, this has happened a few times with the simplejson and
> elementtree libraries in core Python: patches made to the upstream got
> "lost" before merging into Python's stdlib, and patches made against
> the stdlib didn't make it upstream.

stdlib is a more complex problem because it's tightly tied into the
Python release cycle and has a separate set of developers. Even if I'm
not a django developer I am never far from the project so it will be
the same developer doing it during GSOC and afterwards. The extra work
involved in making this work with both Jinja2 and Django and merging
every once in a while should be minimal. The only problem would be
other people doing changes to the codebase directly in the Django SVN.
Which is why I would recommend having a dedicated separate repository
for this thing (might as well be subversion if you want) where I and
all Django developers have access for bugfixes if necessary. And also,
once that thing is stable I don't think there is a lot to be done on
the actual compiler. The Django specific stuff will not end up in that
backend anyways.


Regards,
Armin