[GSoC] Switching to Jinja2 proposal

[Christopher Medrela]

Hello! GSoC 2014 is coming and I'm thinking about issue to work on.

The template system is one of the components that are of special interest to me.

One of the major issues is that rendering templates is slow. The problem could

be solved by compiling template to Python bytecode, but it seems to be really

hard to achieve, given that there was an unsuccessful attempt.

Why not switching to Jinja2? I thought that somebody else proposed this idea

but I couldn't find any discussion; so, please, point me to the discussion if

the idea was discussed, otherwise let's discuss it!

The pros are obvious: 1) fast templates, 2) less code to maintain, 3) lot's of

companies use Jinja2 (because of slowness of default template system) and

builtin support for Jinja2 would be beneficial for them (thing about

integrating Jinja2 with settings like TEMPLATE_DEBUG).

Now the cons. First of all, one of the design decision is that Django has no

dependencies. We can overwhelm it by "static linking" -- I mean copying Jinja2

code into Django. At the first glance, it may look like a horrible idea, but

think about it in a different way. If we don't switch to Jinja2, we have to

maintain entire template system and fix every bug as well as implement new

features. If we switch, Jinja2 developers can do this job for us. We only need

to forward tickets to Jinja2 developers and update the static linkage.

The second big problem is that switching is a big change and backward

compatibility matters. We will need to support both the deprecated Django

template system and the new one. However, it doesn't mean double work -- we

don't need to implement new features for the deprecated system, only bug fixes

will be required. Also note, that a lot of companies uses Jinja2 and switching

from third-package Jinja2 to Jinja2-builtin-Django isn't an enormous change at

all.

I'd like to hear your opinion. Feel free to comment!

BTW, I'd like to have an internship in the late summer. It's impossible to

work at GSoC and have an internship at the same time, but I really want to do

both, so I need to start GSoC as early as possible, at 21 April or even

earlier. Is it possible?

[Russell Keith-Magee]

On Sun, Feb 9, 2014 at 6:16 AM, Christopher Medrela <chris....@gmail.com> wrote:

Hello! GSoC 2014 is coming and I'm thinking about issue to work on.

The template system is one of the components that are of special interest to me.

One of the major issues is that rendering templates is slow. The problem could

be solved by compiling template to Python bytecode, but it seems to be really

hard to achieve, given that there was an unsuccessful attempt.

This should set off a red flag for you. The GSoC project to byte code compile Django's templates was implemented by Armin, the same person who wrote Jinja2 - and yet the project didn't fully succeed. It's worth investigating *why* this idea failed, because it flags one of the reasons why "just adopt Jinja2" may not be a viable options.

 

Why not switching to Jinja2? I thought that somebody else proposed this idea

but I couldn't find any discussion; so, please, point me to the discussion if

the idea was discussed, otherwise let's discuss it!

It's been proposed in jest, and it's been accepted in jest as well :-) However, I don't think there's been a serious proposal to this effect.

 

The pros are obvious: 1) fast templates, 2) less code to maintain, 3) lot's of

companies use Jinja2 (because of slowness of default template system) and

builtin support for Jinja2 would be beneficial for them (thing about

integrating Jinja2 with settings like TEMPLATE_DEBUG).

Now the cons. First of all, one of the design decision is that Django has no

dependencies. We can overwhelm it by "static linking" -- I mean copying Jinja2

code into Django. At the first glance, it may look like a horrible idea, but

think about it in a different way. If we don't switch to Jinja2, we have to

maintain entire template system and fix every bug as well as implement new

features. If we switch, Jinja2 developers can do this job for us. We only need

to forward tickets to Jinja2 developers and update the static linkage.

We're unlikely to vendor a copy of Jinja2. If we went down this road, we'd be much more likely to look at using dependencies defined in setup.py.

 

The second big problem is that switching is a big change and backward

compatibility matters. We will need to support both the deprecated Django

template system and the new one. However, it doesn't mean double work -- we

don't need to implement new features for the deprecated system, only bug fixes

will be required. Also note, that a lot of companies uses Jinja2 and switching

from third-package Jinja2 to Jinja2-builtin-Django isn't an enormous change at

all.

Untrue. It *can* be a *very* big change. One of the biggest problems is backwards compatibility for custom template tags. There are a lot of these out there in the wild, and the way Django defines custom template tags is one of the major reasons that a bytecode approach to Django templates is difficult.

 

I'd like to hear your opinion. Feel free to comment!

Personally, I'm -0 on this proposal as described.

Although Jinja2 and Django template share a common base syntax, Jinja2 includes a bunch of features that I'm not wild about. Django's templates are *deliberately* hobbled to prevent the injection of business logic into templates. Jinja2 template allow for function calls, array subscripting, and all sorts of other programming language structures. I'm not saying these things are inherently bad; I'm saying there's a reason why Django hasn't included them, and I'm not wild about the idea of switching to a default template language that allows them.

I'd be a more supportive of two different spins on this project idea:

 1) Try to pick up where the 2012 GSoC project left off, and continue the work to byte code compile Django's templates. However, this project is unlikely to get off the ground without a concrete proposal to get around the problems encountered the first time around.

 2) Work on the internals of Django to decouple the template engine, so that (a), Django's template language is a standalone in the same way that Jinja2 is, and/or (b) there's a clean interface so that it's possible to define a clean Jinja2 module that you can drop into your Django stack.

A good solution for (2) would set the groundwork for a *long term* transition to Jinja2 templates, but would also allow long term backwards compatibility, as the Django template language could be maintained long term, even if it is dropped as an officially supported option by Django itself.

BTW, I'd like to have an internship in the late summer. It's impossible to

work at GSoC and have an internship at the same time, but I really want to do

both, so I need to start GSoC as early as possible, at 21 April or even

earlier. Is it possible?

It depends on exactly how long the overlapping period is. If it's a matter of a week or two, and we have a good proposal from a student, I suspect we'd be happy to internally shift the dates for the GSoC by a fortnight to accommodate them. In the past, we've accommodated students who have a 2 week exam period in the middle of the GSoC; I don't see why we wouldn't extend the same courtesy to an overlap at the end of the GSoC. However, the longer the overlap, the less likely we are to be accommodating.

Yours,

Russ Magee %-)

[Curtis Maloney]

Can I suggest a 3) to this?

After getting involved with the internals of the template engine recently, I came to the suspicion that a lot of the speed issues come from highly defensive coding.

Now, this is generally to be expected when safety is more important than speed, but I am moderately firm of the opinion that some careful analysis of the template code could help reveal places where the same guard [such as mark_safe, force_text, etc] is being applied repeatedly but could be avoided.

So my (3) to this is to analyse the template code paths sufficiently to be able to identify places where these guards can be omitted safely.

--

Curtis

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAJxq8489OcnOg_KE6PsAghqF6smSY8hRW3gKsmoGT%2Bec1kBMiw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Kevin Christopher Henry]

Russell makes the very good point that Jinja2 isn't just a faster version of the Django template engine - it's philosophically at odds with the original design and intent of the Django template engine.

Personally, I prefer Jinja2's approach and would love to see it become the standard. (The benefits of improved performance, and of moving templates out of core, are nice as well.) But before there's a switch we would need to discuss this philosophical difference and come to a consensus that the Jinja2 approach is, at least, acceptable.

In the absence of a plan to eventually switch to Jinja2, I'm not sure a re-architecting (option 2) is a great use of time. Why? Because in my experience the existing solutions work fine. I'm using Jinja2 and the django-jinja adapter in my projects and everything pretty much just works.

As Russell pointed out, the one thing that doesn't just work is trying to use custom template tags. I haven't looked into this problem at all, but presumably it's a hard one since to my knowledge none of the Jinja2 adapters allow you to do this. I think that the single best way to smooth adoption of Jinja2 would be to figure out a way for a Jinja2 template to use arbitrary Django template tags.

[Russell Keith-Magee]

Hi Curtis,

If true, this would certainly be a reasonable improvement.

However, I'm not sure it would be a great GSoC project. 

This would have to be posed as a "optimise the template language" project, which is problematic when we don't know for certain ahead of time where the problems lie. We're not likely to accept a 12 week fishing expedition, because there's a possibility that the student will run out of things to fix after 2 weeks.

The only way that this would be an acceptable GSoC project would be to present a pre-analysis that demonstrated that there was 12 weeks worth of things than needed to be optimised. However, my experience with optimisations of this sort has been that *finding* the problem is usually 90% of the work. Once you know *what* is slow, cleaning up a code path or caching a slow lookup is comparatively easy. 

However, if someone wants to look for sources of optimisation outside of GSoC, I'd encourage them to do so.

Yours,

Russ Magee %-)

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAG_XiSAXUZDMJ04k88nh%2BZW9J2khCGrpQTe9WiK5U_%3Df4Ad%2Bhw%40mail.gmail.com.

[Jonathan Slenders]

For any refactoring of the template engine, going to jinja2, etc... it would very much help to look at this ticket first:

https://code.djangoproject.com/ticket/20434

It's a backwards compatible patch, which allows template tags to be written in a much more declarative way, without exposing any internals of the parser in the API. If all template tags are eventually rewritten in this system, we could deprecate the old way and have a chance to play with the parser. (But even if we wouldn't depracate old template tags. It's still cleaner.)

This would probably also allow the same template tags of Django to be used by Jinja2.

I wrote that patch a year ago, so not sure whether it still merges.

[Carl Meyer]

Hi Chris and Russ,

On 02/08/2014 05:11 PM, Russell Keith-Magee wrote:
>
> On Sun, Feb 9, 2014 at 6:16 AM, Christopher Medrela
> <chris....@gmail.com <mailto:chris....@gmail.com>> wrote:
>
> Why not switching to Jinja2? I thought that somebody else proposed
> this idea
> but I couldn't find any discussion; so, please, point me to the
> discussion if
> the idea was discussed, otherwise let's discuss it!
>
> It's been proposed in jest, and it's been accepted in jest as well :-)
> However, I don't think there's been a serious proposal to this effect.

I've proposed it before, and not in jest. Though perhaps not fully
seriously either, as full seriousness would mean supplying a patch.

> The pros are obvious: 1) fast templates, 2) less code to maintain,
> 3) lot's of
> companies use Jinja2 (because of slowness of default template
> system) and
> builtin support for Jinja2 would be beneficial for them (thing about
> integrating Jinja2 with settings like TEMPLATE_DEBUG).
>
> Now the cons. First of all, one of the design decision is that
> Django has no
> dependencies. We can overwhelm it by "static linking" -- I mean
> copying Jinja2
> code into Django. At the first glance, it may look like a horrible
> idea, but
> think about it in a different way. If we don't switch to Jinja2, we
> have to
> maintain entire template system and fix every bug as well as
> implement new
> features. If we switch, Jinja2 developers can do this job for us. We
> only need
> to forward tickets to Jinja2 developers and update the static linkage.
>
> We're unlikely to vendor a copy of Jinja2. If we went down this road,
> we'd be much more likely to look at using dependencies defined in setup.py.

I agree.

> The second big problem is that switching is a big change and backward
> compatibility matters. We will need to support both the deprecated
> Django
> template system and the new one. However, it doesn't mean double
> work -- we
> don't need to implement new features for the deprecated system, only
> bug fixes
> will be required. Also note, that a lot of companies uses Jinja2 and
> switching
> from third-package Jinja2 to Jinja2-builtin-Django isn't an enormous
> change at
> all.
>
>
> Untrue. It *can* be a *very* big change. One of the biggest problems is
> backwards compatibility for custom template tags. There are a lot of
> these out there in the wild, and the way Django defines custom template
> tags is one of the major reasons that a bytecode approach to Django
> templates is difficult.

Yes. For this reason I think that any deprecation of DTL would need to
be very slow, if it was deprecated at all. Perhaps it could be moved
into an external project for continued use by legacy projects with large
bodies of custom template tag code.

> I'd like to hear your opinion. Feel free to comment!
>
>
> Personally, I'm -0 on this proposal as described.
>
> Although Jinja2 and Django template share a common base syntax, Jinja2
> includes a bunch of features that I'm not wild about. Django's templates
> are *deliberately* hobbled to prevent the injection of business logic
> into templates. Jinja2 template allow for function calls, array
> subscripting, and all sorts of other programming language structures.
> I'm not saying these things are inherently bad; I'm saying there's a
> reason why Django hasn't included them, and I'm not wild about the idea
> of switching to a default template language that allows them.

I don't buy this argument, for a number of reasons:

1) On the wider spectrum of Python templating languages, Django and
Jinja2's design philosophies are quite similar. Jinja2 does have a few
syntactic constructs that make it more powerful (which make it more
pleasant to work with, IMO), but over-emphasizing these is a case of the
narcissism of small differences. I would be opposed to switching to a
template language with a fundamentally different philosophy (i.e. one
that allows large blocks of pure Python code in templates, or one that
is tied to XML syntax), but I consider Jinja2 a strictly superior
implementation of essentially the same design philosophy as DTL.

2) DTL is indeed hobbled compared to Jinja2, but if the goal of this
hobbling is to "prevent the injection of business logic into templates",
it is a failure, and couldn't be otherwise, because the premise is
flawed. If a template developer has no interest in maintaining a clear
distinction between business logic and presentation logic, DTL gives
them far more than enough rope to hang themselves with; heck, all you
need is a functional {% if %} and you can easily put tons of business
logic into templates (and I've seen plenty of DTL-using projects that do
so.) Having worked on many projects using DTL and many using Jinja2, I
have observed no correlation between the template language used and the
amount of business logic that ends up in templates.

3) Not all logic is business logic. Presentation of data can also
require logic (sometimes non-trivial logic), and often this logic
belongs in the templates. DTL forces you to make choices between doing
presentation-related data transformation in view code (ick) or
implementing custom tags (using an API that is far too difficult and
low-level, resulting in a proliferation of confusingly similar built-in
decorators and third-party projects to attempt to paper over it with a
reasonable API) for simple things that would be more concise and
readable if done directly in the template. And when you do need to do
something in Python code, Jinja2's API for that could hardly be simpler
(define a Python function, make it available in the template context,
and call it), in stark contrast to DTL.

I also think Jinja macros result in clearer and more maintainable
template partials compared to using Django includes for the same
purpose, because macros are explicit about the context variables they
require. (The closest equivalent in Django is an inclusion tag, but
these are harder to work with than macros because the relevant
information to understand the inclusion tag is split up between a
snippet of Python code in one place and a template in another place.)

But mostly, I think the rendering speed alone is enough reason to switch
(given that we have good evidence from Armin's GSoC that getting DTL to
a comparable rendering speed while maintaining backwards compatibility
may not be possible). It's a problem that we are unable to do things
that would otherwise be no-brainers (e.g. rendering form widgets using
templates rather than by concatenating strings of HTML in Python code)
because our default templating language is too slow.

> I'd be a more supportive of two different spins on this project idea:
>
> 1) Try to pick up where the 2012 GSoC project left off, and continue
> the work to byte code compile Django's templates. However, this project
> is unlikely to get off the ground without a concrete proposal to get
> around the problems encountered the first time around.

I think this is unlikely to be successful, and not worth a second GSoC's
worth of effort when Jinja2 is already available.

> 2) Work on the internals of Django to decouple the template engine, so
> that (a), Django's template language is a standalone in the same way
> that Jinja2 is, and/or (b) there's a clean interface so that it's
> possible to define a clean Jinja2 module that you can drop into your
> Django stack.
>
> A good solution for (2) would set the groundwork for a *long term*
> transition to Jinja2 templates, but would also allow long term backwards
> compatibility, as the Django template language could be maintained long
> term, even if it is dropped as an officially supported option by Django
> itself.

This is basically the same approach that I would favor, although I would
immediately make Jinja2 the "blessed" option (i.e. the one used in the
tutorial and featured in the documentation), with DTL maintained (in or
out of core) only for backwards-compatibility for legacy projects.

Carl

[Aymeric Augustin]

On 10 févr. 2014, at 20:16, Carl Meyer <ca...@oddbird.net> wrote:

> I've proposed it before, and not in jest. Though perhaps not fully
> seriously either, as full seriousness would mean supplying a patch.

I’ve heard an ex-BDFL accepting the idea. The other ex-BDFL was there and he
didn’t throw the first one through the window. Hence, the idea is accepted ;-)

I would do it if I had the time.

(As a full-time employee and father, my time isn’t exchangeable for money.)

The MVP would look like this:

1) Design a convention to tell DTL templates apart from Jinja2 templates. My
favorite option is {# syntax: django #} or {# syntax: jinja2 #} in the first
line of the file. Using a file extension would work too. For files that do not
follow the convention, Django would default to DTL for two releases and then
to Jinja2. Let the bikeshed begin ;-)

2) Convert all built-in templates Jinja2. Provide suitable replacements for
template filters and tags we care about. This is a great opportunity to remove
some historical cruft like the add filter.

3) Update various bits of template-related infrastructure appropriately. Given
than jingo has less than 500 LoC excluding tests, and that Django's Template
abstraction is very straightforward, I'm cautiously optimistic.

This is a great opportunity to improve the pluggability of the template engine.
If we do a good job, it will be possible to turn the legacy DTL into a separate
project. We may have to make it less stateful to get there, maybe building on
Christopher Medrela’s patches (#17093).

I have only one significant concern. Jinja2 is still a one-man-project [1] and
that man has taken a public stance on Python 3 that is at odds with Django's.
It's often misinterpreted as "Python 3 sucks" after a cursory reading.

That said, I'm pretty sure that Jinja2 is more robust that the DTL, even under
Python 3. I'm only worried about the PR. Whether we like it or not, software
needs PR. I don't want to have to answer ridiculous comments such as "Look,
Django is migrating to a template language anchored in Python's past!"

[1] https://github.com/mitsuhiko/jinja2/graphs/contributors

--
Aymeric.

[Luke Sneeringer]

On Feb 10, 2014, at 12:16 PM, Carl Meyer <ca...@oddbird.net> wrote:

It's a problem that we are unable to do things
that would otherwise be no-brainers (e.g. rendering form widgets using
templates rather than by concatenating strings of HTML in Python code)
because our default templating language is too slow.

The deliberate hobbling aspect also makes situations like this that should be no-brainers into painful slogs. Form widgets are a great example beyond this, too, because customizing HTML for particular form elements (e.g. adding size="30" to one input field) is an absolute pain. The blessed way involves overwriting the field, which requires copying every single relevant attribute of the model (and then double-maintaining changes).

The Flask extension's mechanism is {{ field(size=30) }}. Done.

And under this example, it's DTL that pushes you to put that logic in the wrong place; it doesn't belong in a forms.py file in most places. I've even seen cases where the only reason an explicit form exists is to make small template-level changes.

Best Regards,

Luke

[Curtis Maloney]

From my own experience [writing formulation] I think it _can_ be easy to write efficient template-level form rendering, with the right pre-processing on the form/field objects.  So once again the issue comes back to "it's not entirely the language, but the objects you pass it."

Just my 2c.

--

Curtis

[Russell Keith-Magee]

On Tue, Feb 11, 2014 at 9:31 AM, Luke Sneeringer <lu...@sneeringer.com> wrote:

On Feb 10, 2014, at 12:16 PM, Carl Meyer <ca...@oddbird.net> wrote:

It's a problem that we are unable to do things
that would otherwise be no-brainers (e.g. rendering form widgets using
templates rather than by concatenating strings of HTML in Python code)
because our default templating language is too slow.

The deliberate hobbling aspect also makes situations like this that should be no-brainers into painful slogs. Form widgets are a great example beyond this, too, because customizing HTML for particular form elements (e.g. adding size="30" to one input field) is an absolute pain. The blessed way involves overwriting the field, which requires copying every single relevant attribute of the model (and then double-maintaining changes).

The Flask extension's mechanism is {{ field(size=30) }}. Done.

Please don't fall into the trap of assuming I'm mentally impaired in some way. 

I've been using the Django template language for a long time. I'm aware of what you can and can't do. I'm fully aware of syntactic sugar that Jinja2 provides, and the ways that it can be used.

I'm also aware of the ways it can be abused. I'd be able to mount a reasonable argument that {{ field(size=30) }} is a problem, not a feature (whats the magical significance of 30? What happens if you make a site-wide decision to extend all size=30 fields to size=40? etc. And, to be clear, I'm not interested in having *this* specific argument in long form. Just be aware that I'd make it, and that it's the argument that underpins the original design decisions of Django's template language).

In the light of this full awareness, *my* analysis has concluded that I'm willing to live with a little inconvenience, and gain an ecosystem in which newcomers aren't encouraged to turn their templates into PHP. 

I'm also aware that this is *my* analysis, and that others will come to a different conclusion, based on their own values, priorities, experiences, and engineering taste. 

This doesn't make *either* analysis wrong. It means different people value different things. 

Yours,

Russ Magee %-)

[Luke Sneeringer]

Sent from my iPad

On Feb 10, 2014, at 7:00 PM, Russell Keith-Magee <rus...@keith-magee.com> wrote:

On Tue, Feb 11, 2014 at 9:31 AM, Luke Sneeringer <lu...@sneeringer.com> wrote:

On Feb 10, 2014, at 12:16 PM, Carl Meyer <ca...@oddbird.net> wrote:

It's a problem that we are unable to do things
that would otherwise be no-brainers (e.g. rendering form widgets using
templates rather than by concatenating strings of HTML in Python code)
because our default templating language is too slow.

The deliberate hobbling aspect also makes situations like this that should be no-brainers into painful slogs. Form widgets are a great example beyond this, too, because customizing HTML for particular form elements (e.g. adding size="30" to one input field) is an absolute pain. The blessed way involves overwriting the field, which requires copying every single relevant attribute of the model (and then double-maintaining changes).

The Flask extension's mechanism is {{ field(size=30) }}. Done.

Please don't fall into the trap of assuming I'm mentally impaired in some way. 

I intended no offense. I have great respect for you and for your intellect and reasoning. 

I'm also aware of the ways it can be abused. I'd be able to mount a reasonable argument that {{ field(size=30) }} is a problem, not a feature (whats the magical significance of 30? What happens if you make a site-wide decision to extend all size=30 fields to size=40? etc. And, to be clear, I'm not interested in having *this* specific argument in long form. Just be aware that I'd make it, and that it's the argument that underpins the original design decisions of Django's template language).

I'm also aware that this is *my* analysis, and that others will come to a different conclusion, based on their own values, priorities, experiences, and engineering taste. 

This doesn't make *either* analysis wrong. It means different people value different things. 

However, Jinja's approach makes it very easy to decide on the side of the line that you are proposing. Django's approach makes it very hard to go in the converse route.

Best Regards,

Luke

[Aymeric Augustin]

On 11 févr. 2014, at 03:00, Russell Keith-Magee <rus...@keith-magee.com> wrote:

> I'd be able to mount a reasonable argument that {{ field(size=30) }} is a problem, not a feature (whats the magical significance of 30? What happens if you make a site-wide decision to extend all size=30 fields to size=40? etc. And, to be clear, I'm not interested in having *this* specific argument in long form. Just be aware that I'd make it, and that it's the argument that underpins the original design decisions of Django's template language).

With all due respect, you can find a better argument… Everyone will hardcode the 30 in Python code just as well as in template code…

--
Aymeric.

[Andrey Antukh]

Hi Aymeric

I agree with you in all most all your points, with exception to this point.

Jinja2 as far as I know, is one of the first template engine that had python3 suport. I think

these type of comments should not exists. 

As you have said previously, jinja2 is a one man project, but if django adopts it, probably

it get a boost.

Andrey

 

[1] https://github.com/mitsuhiko/jinja2/graphs/contributors

--
Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/70B2B842-9977-4F60-B743-4662019DE56A%40polytechnique.org.

For more options, visit https://groups.google.com/groups/opt_out.

--

Andrey Antukh - Андрей Антух - <andrei....@kaleidos.net> / <ni...@niwi.be>

http://www.niwi.be

https://github.com/niwibe

[Christopher Medrela]

I'm really astonished how much feedback I got!

On Sunday, February 9, 2014 1:11:19 AM UTC+1, Russell Keith-Magee wrote:

On Sun, Feb 9, 2014 at 6:16 AM, Christopher Medrela <chris....@gmail.com> wrote:

Hello! GSoC 2014 is coming and I'm thinking about issue to work on.

The template system is one of the components that are of special interest to me.

One of the major issues is that rendering templates is slow. The problem could

be solved by compiling template to Python bytecode, but it seems to be really

hard to achieve, given that there was an unsuccessful attempt.

This should set off a red flag for you. The GSoC project to byte code compile Django's templates was implemented by Armin, the same person who wrote Jinja2 - and yet the project didn't fully succeed. It's worth investigating *why* this idea failed, because it flags one of the reasons why "just adopt Jinja2" may not be a viable options. 

Armin said in his proposal [1] that he wanted to "rewrite Jinja2 code

generation to better support alternative Python implementations like PyPy" and

I guess that was the reason why he didn't adopt "just switch to Jinja2"

approach.

Also note, that his proposal was about writing a backend for both Jinja2 and

DTL. That was really ambitious so there is no risk like "he failed writing

backend for two template engines so switching to Jinja2 will probably also

fail".

[1] https://groups.google.com/forum/#!searchin/django-developers/gsoc$20compilation/django-developers/amUQgS4QvOM

The pros are obvious: 1) fast templates, 2) less code to maintain, 3) lot's of

companies use Jinja2 (because of slowness of default template system) and

builtin support for Jinja2 would be beneficial for them (thing about

integrating Jinja2 with settings like TEMPLATE_DEBUG).

Now the cons. First of all, one of the design decision is that Django has no

dependencies. We can overwhelm it by "static linking" -- I mean copying Jinja2

code into Django. At the first glance, it may look like a horrible idea, but

think about it in a different way. If we don't switch to Jinja2, we have to

maintain entire template system and fix every bug as well as implement new

features. If we switch, Jinja2 developers can do this job for us. We only need

to forward tickets to Jinja2 developers and update the static linkage.

We're unlikely to vendor a copy of Jinja2. If we went down this road, we'd be much more likely to look at using dependencies defined in setup.py. 

So we need to discuss introducing dependencies. Django design decision were

made long time ago when python packaging mechanism was bad, so avoiding

dependencies was obvious, but this argument is no longer valid. So are we

ready to introduce Jinja2 dependency in Django 1.8? This is important question

because introducing dependencies is necessary condition for

switching/supporting Jinja2 and if we don't reach a consensus, it'll be better

to stop discussion and focus on more productive activities (like considering

other GSoC ideas).

BTW, I'd like to have an internship in the late summer. It's impossible to

work at GSoC and have an internship at the same time, but I really want to do

both, so I need to start GSoC as early as possible, at 21 April or even

earlier. Is it possible?

It depends on exactly how long the overlapping period is. If it's a matter of a week or two, and we have a good proposal from a student, I suspect we'd be happy to internally shift the dates for the GSoC by a fortnight to accommodate them. In the past, we've accommodated students who have a 2 week exam period in the middle of the GSoC; I don't see why we wouldn't extend the same courtesy to an overlap at the end of the GSoC. However, the longer the overlap, the less likely we are to be accommodating.

Before approx. April, I can't say how much do I need to shift dates. I'm going

to apply for these internships which won't collide with GSoC and more we shift

the dates, more internships are available for me.

Although Jinja2 and Django template share a common base syntax, Jinja2 includes a bunch of features that I'm not wild about. Django's templates are *deliberately* hobbled to prevent the injection of business logic into templates. Jinja2 template allow for function calls, array subscripting, and all sorts of other programming language structures. I'm not saying these things are inherently bad; I'm saying there's a reason why Django hasn't included them, and I'm not wild about the idea of switching to a default template language that allows them.

But on the other side DTL encourage you to put too much in views. What you

can't do in a template, you can do in a view and pass it to the template via

context. And this isn't good too.

If we switched to Jinja2 and made it the "blessed" option, we could document

only a subset of Jinja2 features, so newbies (who read Django documentation

and didn't read Jinja2 docs -- that is the majority of newbies) won't inject

business logic into templates. And at the same time, experienced users can

read Jinja2 docs and learn all features and they won't be limited by

weaknesses of DTL. That's the best of two worlds.

On Sunday, February 9, 2014 8:45:06 PM UTC+1, Kevin Christopher Henry wrote:

Russell makes the very good point that Jinja2 isn't just a faster version of the Django template engine - it's philosophically at odds with the original design and intent of the Django template engine.

Personally, I prefer Jinja2's approach and would love to see it become the standard. (The benefits of improved performance, and of moving templates out of core, are nice as well.) But before there's a switch we would need to discuss this philosophical difference and come to a consensus that the Jinja2 approach is, at least, acceptable.

OK, so the list of Django principles that Jinja2 doesn't meet is:

1) Separate logic from presentation

2) Don’t invent a programming language

3) Safety and security

IMO DTL meets these principles at great cost. DTL is a proof that you cannot

create a language that prevents users from including business logic in

template while keeping the language powerful as well as comfortable and not

forcing people to write some parts of templates outside templates.

I see that we should prevent newbies, but it comes at huge cost. Maybe we

should change these design principles and sacrifice preventing new users for

convenience of experienced users?

In the absence of a plan to eventually switch to Jinja2, I'm not sure a re-architecting (option 2) is a great use of time. Why? Because in my experience the existing solutions work fine. I'm using Jinja2 and the django-jinja adapter in my projects and everything pretty much just works.

If we support both DTL and Jinja2 and allow to mix templates (so you can

include DTL template from Jinja2 one and vice versa), it can be a great use

of time, even if DTL will stay as the "blessed" option:

1) Django users, who would like to switch from DTL to Jinja2, could do it

   incrementally, instead of "rewrite everything at once" option, which is

   just impossible for large projects.

2) The same argument is true for builtin templates. We could rewrite some

   builtin DTL templates especially these slow ones. We could move rendering

   form widgets from Python code to templates!

And because DTL stays as the blessed option, you don't have the problem of

template system's principles.

This is what I'm proposing now: 1) supporting Jinja2 out-of-box and 2) allowing to

mix DTL and Jinja2 templates; then 3) decoupling DTL and 4) rewriting Django

builtin templates in Jinja2. After doing it, we can reconsider eventually

switching to Jinja2.

On Monday, February 10, 2014 9:56:37 PM UTC+1, Aymeric Augustin wrote:

I have only one significant concern. Jinja2 is still a one-man-project [1] and 
that man has taken a public stance on Python 3 that is at odds with Django's. 
It's often misinterpreted as "Python 3 sucks" after a cursory reading. 

What did Armin said about Python 3 exactly? The only problem I can see is that

Jinja supports 3.3+ while Django 1.8 will support 3.2 (I guess). So 1) we need

to drop support for 3.2 in Django 1.8 or 2) we need to tamper with Jinja2 to

support 3.2 (Armin is not going to help us do it).

[Aymeric Augustin]

2014-02-11 13:42 GMT+01:00 Christopher Medrela <chris....@gmail.com>:

 

What did Armin said about Python 3 exactly?

He wrote an extensive argumentation about "why Python 2 [is] the better

language for dealing with text and bytes" [1] as well as a number of tweets

and a few other blog posts along the same lines.

While his arguments are technically correct, I disagree with his conclusions

because he's speaking with the point of view of an expert maintaining

libraries at the boundary between unicode and bytes (like werkzeug). However,

most Python users aren't experts and aren't maintaining such libraries. In my

experience working with Python programmers ranging from intern to veteran, the

unicode model of Python 3 is a strict improvement over Python 2 in terms of

pitfalls hit in day-to-day programming. YMMV.

[1] http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/

-- 

Aymeric.

[Schmitt, Christian]

I'm not a django-developer, but I'm creating a lot of applications with Django and I would never want to switch to Jinja2.

Why?

The first thing is that Django Templates are simple to understand, they are not formed as a new DSL or a Programmable way of a Template language.

Their Syntax is clear and simple and it's really easy to extend the tags for new programmers aswell.

I only had a quick overview of Jinja2 but the Syntax aswell as other things just looking more like a programming language for templates than a simple templating engine.

Also the Template Language or the Django interface is most of the time not the bottleneck of any application (there are some exceptions where the template engine or some django views will be a bottleneck but most of the time there is a easy way to solve it like using c)

Nearly 90% of the time you will have trouble with the database.

Changing the Template Language will maybe looking good on benchmark papers, but i don't think that it will help scaling websites to need less servers. (Django also has a really good and easy way to scale up really good, thanks to the DatabaseRouters, Cache Engine, Session Engine, etc. you could Scale out/up really, really well)

The next thing is that the internet is changing and template languages that are on the server side getting less focus.

Mostly applications are more and more and API which will getting consumed by a Framework like AngularJS or KnockoutJS.

So there is no need for a faster template language, since the Django Template Language is fast enough for the Django Admin or other applications where a Template Engine is still needed.

I hope that the most people understood my points and that my english isn't too bad.

For me switching to jinja2 wouldn't make any sense. 

Also we changed a lot of app loading stuff, so my applications needing to be migrated. While I think that I have a lot of stuff that needs to be refactored when I switch to Django 1.7 I don't want to have a lot of backwards incompatible stuff in Django 1.8 or Django 1.9, too.

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANE-7mU31zMRkVrJAZY_tde-xdiU63p2s38pB2vVPpeDpPBkxQ%40mail.gmail.com.

[Gwildor Sok]

There are a few problems with Christian's assumptions:

* Not everyone uses a JS Framework. Personally, we use a lot of static pages, and when we do want to do some fancy stuff, we use pjax to replace content on the page, but in the backend this is still done by rendering a full template through a Django view.
* The templating language is also used for small stuff, and the switch to Jinja would enable using the templating language for even more stuff. The biggest issue that comes to mind are template-based widgets.

Personally, I'm in favor of switching to Jinja. The speed bonus and the ability to call functions with arguments are great features for me. One downside I can think of is that Jinja does not escape variables by default, which might become a XSS security issue.

[Curtis Maloney]

At this point someone should start asking for real-world examples with measurements.

I'm personally of the "let's see if we can improve DTL first, then revisit the question" stance.

But in order to show any improvements, we must have timings for the current tool.

Can people who've suffered slowness please donate fragments of template they've found particularly slow?

On 12 February 2014 21:29, Gwildor Sok <gwild...@gmail.com> wrote:

There are a few problems with Christian's assumptions:

* Not everyone uses a JS Framework. Personally, we use a lot of static pages, and when we do want to do some fancy stuff, we use pjax to replace content on the page, but in the backend this is still done by rendering a full template through a Django view.

We run a hybrid side, and fairly high traffic [7k req/min is a fairly typical] and I've yet to find a major template performance issue that didn't turn out to be a DB hit hiding in an object, or similar.

That's not to say I don't think it can be faster...

 

* The templating language is also used for small stuff, and the switch to Jinja would enable using the templating language for even more stuff. The biggest issue that comes to mind are template-based widgets.

Actually, django-sniplates and django-formulation both allow using another template as a "bag of macros".  I'm also working on a "just macros" rework of the idea.  formulation even has a "reuse" tag so you can define template macros [using blocks] within your template.

 

Personally, I'm in favor of switching to Jinja. The speed bonus and the ability to call functions with arguments are great features for me.

I guess it's time I finally write my "What should I pass in the Context" blog post... to point out that your designers are [usually] not coders, and shouldn't have to understand your data structures or schema.  Whilst in some ways perhaps excessively, DTL does make you consider your data structures.

 

One downside I can think of is that Jinja does not escape variables by default, which might become a XSS security issue.

That's quite a large downside!

 

--

Curtis

[Carl Meyer]

On 02/12/2014 02:25 PM, Curtis Maloney wrote:
> At this point someone should start asking for real-world examples with
> measurements.
>
> I'm personally of the "let's see if we can improve DTL first, then
> revisit the question" stance.
>
> But in order to show any improvements, we must have timings for the
> current tool.
>
> Can people who've suffered slowness please donate fragments of template
> they've found particularly slow?

Have a look through the comments of
https://code.djangoproject.com/ticket/15667

Carl

[Donald Stufft]

Crate had one, i’d have to pull it out but it was a pretty simple template. The sticking

point was it had a 30k item loop which was significantly faster in Jinja2.

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAG_XiSBmOU%3D4orZnW13bw-ZLT_O416unynhtnQ9%3DhgPtL9Bbtw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Marc Tamlyn]

Point of order: Jinja2 does now have autoescaping (http://jinja.pocoo.org/docs/extensions/#autoescape-extension)

Do not take this as any opinion on the matter

[Carl Meyer]

On 02/12/2014 02:25 PM, Curtis Maloney wrote:

> On 12 February 2014 21:29, Gwildor Sok <gwild...@gmail.com

> One downside I can think of is that Jinja does not escape variables
> by default, which might become a XSS security issue.
>
> That's quite a large downside!

Jinja2 supports autoescape-by-default, it's a configuration parameter
when creating a template environment. Obviously Django would set it to
True. So this is not an issue.

Carl

[Aymeric Augustin]

In addition, Django implements the (non-standard) __html__ protocol: https://code.djangoproject.com/ticket/7261.

I believe escaping will work correctly without many, if any, changes.

-- 

Aymeric.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMwjO1GCQG1q5W_BmU%3DNTA0zmydnTuhVH%3DFrtdDR-kXVSLTZ8Q%40mail.gmail.com.

[Christopher Medrela]

My last post was pretty long and the most important questions and statements

have left unanswered, so I will repeat them.

What I'm proposing now is more conservative proposal. Firstly, Django will

support Jinja2 out-of-the-box, but DTL will remain the "blessed" option.

Secondly, Django will allow to mix DTL and Jinja2 templates (so you can

include/inherit DTL template from Jinja2 one and vice versa).

After doing it, I could focus on 3) decoupling DTL or/and 4) rewriting Django

builtin templates in Jinja2 or/and 5) moving rendering form widgets from

Python code to Jinja2 templates.

After that all, we could start again the war DTL vs Jinja2, but please focus

on the new proposal now.

Questions are:

1) What do you think about the new proposal? Would it be useful?

2) Jinja2 doesn't support 3.2. Will Django 1.8 support 3.2?

3) Supporting Jinja2 out-of-the-box means introducing dependencies. Are we

   ready for this?

On Tuesday, February 11, 2014 2:07:19 PM UTC+1, Aymeric Augustin wrote:

2014-02-11 13:42 GMT+01:00 Christopher Medrela <chris....@gmail.com>:

 

What did Armin said about Python 3 exactly?

He wrote an extensive argumentation about "why Python 2 [is] the better

language for dealing with text and bytes" [1] as well as a number of tweets

and a few other blog posts along the same lines.

While his arguments are technically correct, I disagree with his conclusions

because he's speaking with the point of view of an expert maintaining

libraries at the boundary between unicode and bytes (like werkzeug). However,

most Python users aren't experts and aren't maintaining such libraries. In my

experience working with Python programmers ranging from intern to veteran, the

unicode model of Python 3 is a strict improvement over Python 2 in terms of

pitfalls hit in day-to-day programming. YMMV.

[1] http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/

-- 

Aymeric.

OK, so Armin finds Python 2 better than Python 3. But why is it at odds with

Django? He didn't say that he is not going to support Python 3. So where is

the risk that concerns you?

[Donald Stufft]

On Feb 15, 2014, at 11:43 AM, Christopher Medrela <chris....@gmail.com> wrote:

My last post was pretty long and the most important questions and statements

have left unanswered, so I will repeat them.

What I'm proposing now is more conservative proposal. Firstly, Django will

support Jinja2 out-of-the-box, but DTL will remain the "blessed" option.

Secondly, Django will allow to mix DTL and Jinja2 templates (so you can

include/inherit DTL template from Jinja2 one and vice versa).

After doing it, I could focus on 3) decoupling DTL or/and 4) rewriting Django

builtin templates in Jinja2 or/and 5) moving rendering form widgets from

Python code to Jinja2 templates.

After that all, we could start again the war DTL vs Jinja2, but please focus

on the new proposal now.

Questions are:

1) What do you think about the new proposal? Would it be useful?

2) Jinja2 doesn't support 3.2. Will Django 1.8 support 3.2?

3) Supporting Jinja2 out-of-the-box means introducing dependencies. Are we

   ready for this?

If we have Jinja2 I don’t see any reason to keep the DTL as the blessed option.

On Tuesday, February 11, 2014 2:07:19 PM UTC+1, Aymeric Augustin wrote:

2014-02-11 13:42 GMT+01:00 Christopher Medrela <chris....@gmail.com>:

 

What did Armin said about Python 3 exactly?

He wrote an extensive argumentation about "why Python 2 [is] the better

language for dealing with text and bytes" [1] as well as a number of tweets

and a few other blog posts along the same lines.

While his arguments are technically correct, I disagree with his conclusions

because he's speaking with the point of view of an expert maintaining

libraries at the boundary between unicode and bytes (like werkzeug). However,

most Python users aren't experts and aren't maintaining such libraries. In my

experience working with Python programmers ranging from intern to veteran, the

unicode model of Python 3 is a strict improvement over Python 2 in terms of

pitfalls hit in day-to-day programming. YMMV.

[1] http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/

-- 

Aymeric.

OK, so Armin finds Python 2 better than Python 3. But why is it at odds with

Django? He didn't say that he is not going to support Python 3. So where is

the risk that concerns you?

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/79dbbf71-6b70-48d1-8510-cef471812677%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Babatunde Akinyanmi]

On 15 Feb 2014 18:13, "Donald Stufft" <don...@stufft.io> wrote:
>
>
> On Feb 15, 2014, at 11:43 AM, Christopher Medrela <chris....@gmail.com> wrote:
>
>> My last post was pretty long and the most important questions and statements
>> have left unanswered, so I will repeat them.
>>
>> What I'm proposing now is more conservative proposal. Firstly, Django will
>> support Jinja2 out-of-the-box, but DTL will remain the "blessed" option.
>> Secondly, Django will allow to mix DTL and Jinja2 templates (so you can
>> include/inherit DTL template from Jinja2 one and vice versa).
>>
>> After doing it, I could focus on 3) decoupling DTL or/and 4) rewriting Django
>> builtin templates in Jinja2 or/and 5) moving rendering form widgets from
>> Python code to Jinja2 templates.
>>
>> After that all, we could start again the war DTL vs Jinja2, but please focus
>> on the new proposal now.
>>
>> Questions are:
>>
>> 1) What do you think about the new proposal? Would it be useful?
>>
>> 2) Jinja2 doesn't support 3.2. Will Django 1.8 support 3.2?
>>
>> 3) Supporting Jinja2 out-of-the-box means introducing dependencies. Are we
>>    ready for this?
>
>
> If we have Jinja2 I don’t see any reason to keep the DTL as the blessed option.

Exactly

[Russell Keith-Magee]

On Sun, Feb 16, 2014 at 12:43 AM, Christopher Medrela <chris....@gmail.com> wrote:

My last post was pretty long and the most important questions and statements

have left unanswered, so I will repeat them.

What I'm proposing now is more conservative proposal. Firstly, Django will

support Jinja2 out-of-the-box, but DTL will remain the "blessed" option.

As a broad statement, this sounds fine; but what does this mean in practice? What does "out of the box" support look like?

 

Secondly, Django will allow to mix DTL and Jinja2 templates (so you can

include/inherit DTL template from Jinja2 one and vice versa).

Including doesn't sound like it would be any problem, but inheriting? Is that really going to be possible?

After doing it, I could focus on 3) decoupling DTL or/and 4) rewriting Django

builtin templates in Jinja2 or/and 5) moving rendering form widgets from

Python code to Jinja2 templates.

After that all, we could start again the war DTL vs Jinja2, but please focus

on the new proposal now.

Questions are:

1) What do you think about the new proposal? Would it be useful?

 

I think the broad feature is useful; I can't comment as to whether it would make the lives of any existing Django-Jinja users any easier.

2) Jinja2 doesn't support 3.2. Will Django 1.8 support 3.2?

Django 1.6 and 1.7 both support 3.2; we haven't had the project-level discussion about deprecating 3.2 support.

Historically, this would have been based on the usage of Python 3.2 in the wild, driven primarily by operating systems that shipped 3.2 as the default. That's not going to apply here. I'm not sure how we'd judge the rate of 3.2 adoption in practice.

3) Supporting Jinja2 out-of-the-box means introducing dependencies. Are we

   ready for this?

I think we are. Pip and setuptools handle dependencies really well; if we don't want to add it as a default dependency, we can easily check for an ImportError when Jinja support is enabled and give an appropriate error message, same as we do for YAML support.

 

On Tuesday, February 11, 2014 2:07:19 PM UTC+1, Aymeric Augustin wrote:

2014-02-11 13:42 GMT+01:00 Christopher Medrela <chris....@gmail.com>:

 

What did Armin said about Python 3 exactly?

He wrote an extensive argumentation about "why Python 2 [is] the better

language for dealing with text and bytes" [1] as well as a number of tweets

and a few other blog posts along the same lines.

While his arguments are technically correct, I disagree with his conclusions

because he's speaking with the point of view of an expert maintaining

libraries at the boundary between unicode and bytes (like werkzeug). However,

most Python users aren't experts and aren't maintaining such libraries. In my

experience working with Python programmers ranging from intern to veteran, the

unicode model of Python 3 is a strict improvement over Python 2 in terms of

pitfalls hit in day-to-day programming. YMMV.

[1] http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/

-- 

Aymeric.

OK, so Armin finds Python 2 better than Python 3. But why is it at odds with

Django? He didn't say that he is not going to support Python 3. So where is

the risk that concerns you?

For my part, my concern is that the tone of his discussions and comments about Python3 suggests that while he currently supports Python3, he does so begrudgingly, and he might, at some point in the future, stop. That would put Django in a precarious position.

Yours,

Russ Magee %-)

[Russell Keith-Magee]

On Sun, Feb 16, 2014 at 1:12 AM, Donald Stufft <don...@stufft.io> wrote:

On Feb 15, 2014, at 11:43 AM, Christopher Medrela <chris....@gmail.com> wrote:

My last post was pretty long and the most important questions and statements

have left unanswered, so I will repeat them.

What I'm proposing now is more conservative proposal. Firstly, Django will

support Jinja2 out-of-the-box, but DTL will remain the "blessed" option.

Secondly, Django will allow to mix DTL and Jinja2 templates (so you can

include/inherit DTL template from Jinja2 one and vice versa).

After doing it, I could focus on 3) decoupling DTL or/and 4) rewriting Django

builtin templates in Jinja2 or/and 5) moving rendering form widgets from

Python code to Jinja2 templates.

After that all, we could start again the war DTL vs Jinja2, but please focus

on the new proposal now.

Questions are:

1) What do you think about the new proposal? Would it be useful?

2) Jinja2 doesn't support 3.2. Will Django 1.8 support 3.2?

3) Supporting Jinja2 out-of-the-box means introducing dependencies. Are we

   ready for this?

If we have Jinja2 I don’t see any reason to keep the DTL as the blessed option.

 

Amongst other reasons:

a) Some of us prefer DTL to Jinja2 :-)

b) The Python3 support issue is a lingering concern

c) There is a lot of legacy code and tutorials that need to continue to work as is.

This is an area where there is a lot of benefit to moving slowly, IMHO. We're talking about a major part of the Django experience; Introducing a new option *and* switching to it in a single release sounds like a recipe for confusion to me.

As I see it, there are three possible options here:

 1) Add Jinja2 as a supported option, but stick with DTL as the default.

 2) Add Jinja2 as a supported option, and indicate that long term, we're going to switch to Jinja as the default

 3) Add Jinja2 as a supported option and move to it immediately as the default.

My personal preferences would be in that order. This is something where historically, we would have gone to the BDFLs for a call; I suspect the core team might need to have a quick discussion and make a call, lest this turn into the mother of all bike sheds.

Yours,

Russ Magee %-)

[Carl Meyer]

Hi Russ,

On 02/15/2014 05:16 PM, Russell Keith-Magee wrote:
> a) Some of us prefer DTL to Jinja2 :-)

Just as a point of clarification: have you used Jinja2 for a non-trivial
project, and are there specific areas where you, personally, for your
own use, prefer how you do something in DTL vs how it's done in Jinja2?
Or is this preference for DTL entirely a matter of theoretical concern
about what others might do given the increased power of Jinja2?

I ask because the latter is all I've so far heard, and I have a hard
time imagining cases of the former where the DTL approach couldn't
easily be emulated in Jinja2.

> b) The Python3 support issue is a lingering concern
>
> c) There is a lot of legacy code and tutorials that need to continue to
> work as is.
>
> This is an area where there is a lot of benefit to moving slowly, IMHO.
> We're talking about a major part of the Django experience; Introducing a
> new option *and* switching to it in a single release sounds like a
> recipe for confusion to me.
>
> As I see it, there are three possible options here:
>
> 1) Add Jinja2 as a supported option, but stick with DTL as the default.
> 2) Add Jinja2 as a supported option, and indicate that long term, we're
> going to switch to Jinja as the default
> 3) Add Jinja2 as a supported option and move to it immediately as the
> default.

I agree that that there is no harm in moving slowly (apart from the
added burden of maintaining support for two template engines for a
period of time).

From my perspective, the most important benefit comes not when Jinja2
becomes the default/documented option (although I do favor moving in
that direction), but when it becomes a guaranteed-available option, such
that we can use it internally in Django performance-sensitive template
renderings (i.e. form widgets). If we don't achieve even that in the
first iteration, then we haven't achieved much that isn't already doable
via third-party adapters such as jingo or django-jinja.

Carl

[Carl Meyer]

Hi Christopher,

On 02/15/2014 09:43 AM, Christopher Medrela wrote:
> What I'm proposing now is more conservative proposal. Firstly, Django will
> support Jinja2 out-of-the-box, but DTL will remain the "blessed" option.
> Secondly, Django will allow to mix DTL and Jinja2 templates (so you can
> include/inherit DTL template from Jinja2 one and vice versa).
>
> After doing it, I could focus on 3) decoupling DTL or/and 4) rewriting
> Django
> builtin templates in Jinja2 or/and 5) moving rendering form widgets from
> Python code to Jinja2 templates.

This sounds reasonable to me.

> After that all, we could start again the war DTL vs Jinja2, but please focus
> on the new proposal now.
>
> Questions are:
>
> 1) What do you think about the new proposal? Would it be useful?

Yes, as long as Jinja2 is a hard dependency, such that we can rely on
its availability for internal Django use (form widgets).

> 2) Jinja2 doesn't support 3.2. Will Django 1.8 support 3.2?

Donald might be able to offer better hard numbers based on e.g. PyPI
usage, but my impression is that usage of 3.2 is very low, and dropping
it for 1.8 would not be a major problem.

> 3) Supporting Jinja2 out-of-the-box means introducing dependencies. Are we
> ready for this?

I think so, yes.

Carl

[Donald Stufft]

These numbers are about a month old, but https://gist.github.com/dstufft/8455306

>
>> 3) Supporting Jinja2 out-of-the-box means introducing dependencies. Are we
>> ready for this?
>
> I think so, yes.
>
> Carl
>

[Christopher Medrela]

I'm sorry for such long time without any reply, but I was investigating possible

approaches of mixing Django and Jinja2 templates.

On Sunday, February 16, 2014 1:08:49 AM UTC+1, Russell Keith-Magee wrote:

On Sun, Feb 16, 2014 at 12:43 AM, Christopher Medrela <chris....@gmail.com> wrote:

My last post was pretty long and the most important questions and statements

have left unanswered, so I will repeat them.

What I'm proposing now is more conservative proposal. Firstly, Django will

support Jinja2 out-of-the-box, but DTL will remain the "blessed" option.

As a broad statement, this sounds fine; but what does this mean in practice? What does "out of the box" support look like?

The two main principles would be: 1) reusing existing interfaces and contracts

2) making switching from Djinja/jingo/django-jinja/coffin as easy as possible.

Last month, Djinja and jingo were downloaded 650 times while coffin and

django-jinja have each one 2k downloads. Coffin focus on imitating DTL.

Therefore, the "out of box" support should be similar to django-jinja.

Basically, the same functions will apply to Jinja2 as well as to Django

templates. If you want to render a template, you write

``render_to_response(template_name)``, whether the template is Jinja2 or Django

template. Templates will be distinguished by extension.

How can it be achieved?

 - We need to move all existing functions (like `render_to_response`) from

   django.template to a new module. Let's call it `django.dtl`.

 - Then, we can create dummy implementations of `render_to_response` (and all

   other functions) that checks the template extension and dispatch to

   corresponding function from `django.dtl` or `jinja2`.

 - It would be nice to use the same loaders for both django.dtl and jinja2. This

   requires to rewrite existing Django loaders so that they fulfill the contract

   of `jinja2.Loader`. 

Secondly, Django will allow to mix DTL and Jinja2 templates (so you can
include/inherit DTL template from Jinja2 one and vice versa).

Including doesn't sound like it would be any problem, but inheriting? Is that really going to be possible?

It seems to be possible. Consider parent Jinja2 template:

    Parent

    {% block overridden %}parent overridden block{% endblock %}

    {% block nonoverridden %}parent nonoverridden block{% endblock %}

as well as child Jinja2 template:

    {% extends "parent.jinja2" %}

    {% block overridden %}child overridden block{% endblock %}

Jinja2 compiles templates by generating valid Python code and passing it to

`exec` function. The appropriate code for the former is:

    from __future__ import division

    from jinja2.runtime import LoopContext, TemplateReference, Macro, Markup, TemplateRuntimeError, missing, concat, escape, markup_join, unicode_join, to_string, identity, TemplateNotFound

    name = 'parent'

    def root(context, environment=environment):

        if 0: yield None

        yield u'Parent '

        for event in context.blocks['overridden'][0](context):

            yield event

        for event in context.blocks['nonoverridden'][0](context):

            yield event

    def block_nonoverridden(context, environment=environment):

        if 0: yield None

        yield u'parent nonoverridden block'

    def block_overridden(context, environment=environment):

        if 0: yield None

        yield u'parent overridden block'

    blocks = {'nonoverridden': block_nonoverridden, 'overridden': block_overridden}

    debug_info = '1=8'

and for the child is:

    from __future__ import division

    from jinja2.runtime import LoopContext, TemplateReference, Macro, Markup, TemplateRuntimeError, missing, concat, escape, markup_join, unicode_join, to_string, identity, TemplateNotFound

    name = 'child'

    def root(context, environment=environment):

        parent_template = None

        if 0: yield None

        parent_template = environment.get_template('parent.jinja2', 'child')

        for name, parent_block in parent_template.blocks.iteritems():

            context.blocks.setdefault(name, []).append(parent_block)

        for event in parent_template.root_render_func(context):

            yield event

    def block_overridden(context, environment=environment):

        if 0: yield None

        yield u'child overridden block'

    blocks = {'overridden': block_overridden}

    debug_info = '1=8'

As you can see, child template loads parent template using

`environment.get_template` method; then iterates over its blocks; finally

delegates to it by calling `root_render_func` and passing the context.

We can write Jinja2 loader that checks the template extension; if it's "jinja2",

it delegates to another loader that is responsible for loading Jinja2 templates;

otherwise, it loads appropriate Django template and wraps it with an adapter.

The adapter must have the same interface as `jinja2.Template` -- probably only

`blocks` and `root_render_func` need to be implemented. `root_render_func` will

be similar to `Template.render` and `blocks` is a map from block names to

functions rendering these blocks (these functions will be similar to

`BlockNode.render`). We also need to tamper with `BlockNode.render` method in

the case when the child template is DTL and the parent is written in Jinja2 so

BlockNode can pass control to the block defined in parent template.

As I see it, there are three possible options here:

 

 1) Add Jinja2 as a supported option, but stick with DTL as the default.
 2) Add Jinja2 as a supported option, and indicate that long term, we're going to switch to Jinja as the default
 3) Add Jinja2 as a supported option and move to it immediately as the default.

 

My personal preferences would be in that order. This is something where historically, we would have gone to the BDFLs for a call; I suspect the core team might need to have a quick discussion and make a call, lest this turn into the mother of all bike sheds.

First option is a necessary condition for the others and it seems to be big

enough GSoC project, therefore I will focus on it. In the future, if we changed

mind, we could move to other options. 

On Sunday, February 16, 2014 11:44:52 PM UTC+1, Donald Stufft wrote:

On Feb 16, 2014, at 4:23 PM, Carl Meyer <ca...@oddbird.net> wrote: 

> 

>> 2) Jinja2 doesn't support 3.2. Will Django 1.8 support 3.2? 
> 
> Donald might be able to offer better hard numbers based on e.g. PyPI 
> usage, but my impression is that usage of 3.2 is very low, and dropping 
> it for 1.8 would not be a major problem. 

These numbers are about a month old, but https://gist.github.com/dstufft/8455306 

Thank you very much, Donald! In short, the numbers are:

2.7 -- 6450k total downloads

3.2 --   60k

3.3 --  250k

It seems like the support for Python 3.2 could be dropped. 

[Camilo Torres]

 - Then, we can create dummy implementations of `render_to_response` (and all

   other functions) that checks the template extension and dispatch to

   corresponding function from `django.dtl` or `jinja2`.

Hello,

Could this create a little problem for the people in charge of the template construction?. Let's say the templates are HTML, many tools uses the file extension to 'detect' the file type, so these people can get confused when their tools do not recognize the templates as expected; in addition, the people in charge of templates could not be developers, but graphic designers or authoring tools users.

Regards,

Camilo

[Ryan Hiebert]

That argument makes sense to me: '.html' would have to be assumed to be DTL, because in order to get Jinja2 you'd have to specify it in the extension.

[Kevin Christopher Henry]

Hi Christopher,
 

 ... checks the template extension and dispatch to

 corresponding function from `django.dtl` or `jinja2`.

The mechanism for distinguishing the two kinds of template needs to be more flexible. For example, let's say I want to override a third-party template with my own Jinja template. In that case I need to use the same name (.html and all), but I want it to be processed by Jinja. The way django-jinja solves this is with a setting that provides a regular expression to determine which template names get processed by Jinja. That may not be the best way (I can imagine that regular expression getting hairy with a lot of "or" clauses), but just looking at file extensions isn't enough.

I don't have a comment on the merits of your approach to inheriting from DTL templates, but personally this is not something I've ever needed to do. I'm sure there are use cases for this, that's just my experience.

By contrast, something that I very often want to do is use third-party Django template tags in my Jinja2 templates. Right now there's just no way to do that (that I know of). So, if you're taking requests, please solve that problem. :-) And if you're forced to prioritize, I think that would be more useful than being able to inherit from DTL templates.

Cheers,
Kevin

 

[Christopher Medrela]

I've made some benchmarks. The results are:

jinja2 empty for  0.028 s

jinja2 include    1.094 s

django empty for  0.435 s

django include    2.134 s

Where "empty for" is an empty loop repeated 50 times:

    {% for i in range_ %} {% endfor %}

And include is a similar template where the loop body includes another template

(which is empty) inside for loop:

    {% for i in range_ %}{% include "i" %}{% endfor %}

The full code is available here: http://pastebin.com/DWjE1axV

Note that including is only 1.5x faster in Jinja2 compared to Django.

Surprisingly, the main speedup is in executing for loop (19x), not it's body.

On Friday, February 21, 2014 11:07:30 PM UTC+1, Kevin Christopher Henry wrote:

Hi Christopher,
 

 ... checks the template extension and dispatch to

 corresponding function from `django.dtl` or `jinja2`.

The mechanism for distinguishing the two kinds of template needs to be more flexible. For example, let's say I want to override a third-party template with my own Jinja template. In that case I need to use the same name (.html and all), but I want it to be processed by Jinja. The way django-jinja solves this is with a setting that provides a regular expression to determine which template names get processed by Jinja. That may not be the best way (I can imagine that regular expression getting hairy with a lot of "or" clauses), but just looking at file extensions isn't enough.

Unfortunately, you're right. So I will follow Aymeric Augustin's idea: at the

beginning of every template there will be "{# syntax: django #}" or "{# syntax:

jinja2 #}". If the template lacks such declaration, django template is assumed.

I don't have a comment on the merits of your approach to inheriting from DTL templates, but personally this is not something I've ever needed to do. I'm sure there are use cases for this, that's just my experience.

By contrast, something that I very often want to do is use third-party Django template tags in my Jinja2 templates. Right now there's just no way to do that (that I know of). So, if you're taking requests, please solve that problem. :-) And if you're forced to prioritize, I think that would be more useful than being able to inherit from DTL templates.

What kind of support do you except for third-party template tags? Suppose, that

`cycle` tag is not builtin. Would it be acceptable to write sth like that:

    dtl cycle '1' '2' as rows

It could be quite easily implemented as a Jinja2 extensions. Of course, I guess

that you'd prefer this style:

    cycle '1' '2' as rows

Unfortunately, it cannot be done easily. Each of template tags should map to a

new Jinja2 extension. But the list of extensions must be given to `Environment`

constructor and cannot be changed (at least this is what documentation says).

What can we do?

 1) We can tamper with Jinja2. Maybe it is possible to change list of extensions

    after environment creation? I will investigate it.

 2) We need to know all template tags before rendering first template. All tags

    will be available in all templates. The drawback is that you have no modules

    that you can `load` when desired.

BTW, what about built-in template tags? Are there any issues?

[Curtis Maloney]

I must admit to being conflicted on this project.

I'd love to see the measurements and profiling involved in establishing how much "faster" Jinja2 is, mostly to see where DTL could do with some love...

However, I don't believe for most people their templates represent enough of a time cost to make this project worth it.  Certainly, in every project I'm involved in, it's typically the actual work that takes most of the time.  Add caching loaders and fragment caching, and, well...

For people where it is, Django already supports pluggable template engines through Loaders... which also solves the "how to distinguish" problem...

--

Curtis

--

You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ecf638ce-5811-4234-bd40-964178763090%40googlegroups.com.

[Curtis Maloney]

Did I miss the link to your benchmark code?
Does it time just rendering, or parsing also?
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Christopher Medrela]

The link is here: http://pastebin.com/DWjE1axV

It times only rendering -- templates are parsed and cached in the setup.

[Kevin Christopher Henry]

What kind of support do you except for third-party template tags? Suppose, that

`cycle` tag is not builtin. Would it be acceptable to write sth like that:

    dtl cycle '1' '2' as rows

It could be quite easily implemented as a Jinja2 extensions. Of course, I guess

that you'd prefer this style:

    cycle '1' '2' as rows

Good question. In my opinion, making the tag invocation look the same as it does in DTL should not be an important goal. These are different languages, after all, with different syntaxes and different calling conventions. For example, many things that need to be tags in DTL are just functions in Jinja. (django-jinja took this approach by implementing the built-in 'url' as a function rather than a tag.)

Another relevant issue is that there could be name ambiguities. In DTL you specify a module with the load tag, which allows you to have multiple apps that use the same template tag name. Without the load tag you need some way to specify which 'cycle' you're referring to. (This is not just an issue with wrapping DTL tags, it's also going to be an issue if multiple apps try to register global Jinja functions with the same name.)

[Curtis Maloney]

Yah... my oops ... I was reading on my phone and missed the link :)

--

You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/eda8c6ea-79b6-4bb1-a4b5-7af8d2dfd3bf%40googlegroups.com.

[Christopher Medrela]

Hello! I'm sorry for such long time without any feedback. Unfortunately, I'm

not going to send Django proposal. My original plan was to send two proposals:

one for Django and one for Breeze (Scala numerical processing library) but I

lack time and I will focus on Breeze. Therefore, I won't have enough time to

work at adding Jinja2 support this summer.

Thank you very much for all responses! Lots of DTL aspects were discussed here

and we reached consensus in some issues (like adding out-of-the-box support of

Jinja2) so I hope that the time you spent at this thread is not lost time.

Maybe there will be some other students eager to work at template system?

[Gwildor Sok]

So, there won't be a GSoC project for this, but it would be a shame to let the lengthy discussion and momentum generated go to waste. Can a decision on this matter be made and implemented regardless of the GSoC projects?

Personally, I'm all for it, mostly because of the speed gained, which would finally allow ticket #15667 to be solved, which is a really important one to me because it addresses one of the core modules of Django and one which is behaving pretty sloppy compared to the better ones, in my humble opinion.

[Christopher Medrela]

On Sunday, March 23, 2014 11:51:06 AM UTC+1, Gwildor Sok wrote:

So, there won't be a GSoC project for this, but it would be a shame to let the lengthy discussion and momentum generated go to waste. Can a decision on this matter be made and implemented regardless of the GSoC projects?

I think that the initial decisions were made already -- we agreed on

out-of-the-box support for Jinja2 and rewriting some templates into Jinja2.

You are asking for implementing a big feature but who should do it? Everybody

is a volunteer here, contributors are scare resource. As I said, I won't have

enough time in the next six months to work at this feature. Maybe somebody

will tempt to implement that for money? This feature seems to be a good

kickstarter project -- it's an isolated issue with a clear objective.

[Asif Saifuddin]

Hi, chris,

So as django have the jinja 2 support out of the box what do you think the direction of the ticket? https://code.djangoproject.com/ticket/15667

should the form rendering based on jinja2? or whats on our mind? If you guide me with some direction then I might go for the ticket. looking forward to hear from you.

Regards

[Christopher Medrela]

Asif, I'm not developing Django since some time ago and I really don't know

answer for your questions.