RFC: django.template refactoring (#7806)

220 views
Skip to first unread message

Johannes Dollinger

unread,
Sep 16, 2008, 2:36:30 PM9/16/08
to django-d...@googlegroups.com
Why should django.template be refactored? Filter and Variable parsing
is inconsistent. Many ad hoc parsers in defaulttags are fragile.
Whitespace handling is ungraceful.

The patch provided in #7806[1] splits __init__.py into separate
modules[2] and introduces a TokenStream class that allows parsing of
literals, vars (called lookups) and filter expressions.

Here's how it would work:

@register.tag
@uses_token_stream
def mytag(parser, bits):
expr = bits.parse_expression(required=True)
return MyNode(expr)

`uses_token_stream` replaces the Token argument to the parser
function with a TokenStream object.
If the token is not fully parsed, a TemplateSyntaxError is raised.
For better examples, have a look at the patched version of
`django.template.defaulttags`.


TokenStream API (first stab)
============================
``def __init__(self, parser, source)``
parser: a django.template.compiler.Parser instance
source: a string or a django.template.compiler.Token instance

TokenStream tokenizes its source into "string_literal",
"numeric_literal", "char", and "name" tokens, stored in self.tokens
as (type, lexeme) pairs. Whitespace will be discarded.
You can "read" tokens via the following methods, the current position
in self.tokens is maintained in self.offset.

A "char" token is a single character in ':|=,;<>!?%&@"\'/()[]{}`*+-'.
A name is any sequence of characters that does contain neither "char"
characters nor whitespace and is not a string or numeric literal.

>>> TokenStream(parser, r'"quoted\" string"|filter:3.14 as
name').tokens
[('string_literal', '"quoted\\" string"'), ('char', '|'), ('name',
'filter'), ('char', ':'), ('numeric_literal', '3.14'), ('name',
'as'), ('name', 'name')]
>>> TokenStream(parser, r' "quoted\" string" | filter : 3.14 as
name').tokens
[('string_literal', '"quoted\\" string"'), ('char', '|'), ('name',
'filter'), ('char', ':'), ('numeric_literal', '3.14'), ('name',
'as'), ('name', 'name')]

Low level API
-------------
``def pop(self)``
Returns a pair (tokentype, lexeme) and offset+=1; tokentype is one of
"string_literal", "numeric_literal", "char", "name".

``def pop_lexem(self, match)``
Returns True and offset+=1 if the next token's lexeme equals `match`.
Returns False otherwise.

``def pop_name(self)``
Returns the next token's lexeme and offset+=1 if its tokentype is
"name". Returns None otherwise.

``def pushback(self)``
offset-=1


High level API
--------------
These methods raise TokenSyntaxError and leave offset untouched if
the expected result cannot be parsed.
Each accepts a boolean required=False kwarg which turns
TokenSyntaxErrors into TemplateSyntaxErrors if True.

``def parse_string(self, bare=False)``
Returns the value of the following string literal. If bare=True,
unquoted strings will be accepted.

``def parse_int(self)``
Returns the value of the following numeric literal, if it is an int.

``def parse_value(self)``
Returns an Expression that evaluates the following literal, variable
or translated value.

``def parse_expression(self)``
Returns an Expression that evaluates the following value or
filterexpression.

``def parse_expression_list(self, minimum=0, maximum=None, count=None)``
Returns a list `e` of expressions; minimum <= len(e) <= maximum.
count=n is a shortcut for minimum=maximum=n.


I'm unhappy with the naming of TokenStream and uses_token_stream
(using_token_stream?). And I just noticed the english spelling of
lexeme has a trailing "e".


[1] http://code.djangoproject.com/ticket/7806
[2] See [1] for details. Yes, this is orthogonal to the concern of
the ticket. I gave up splitting the patch when I stumbled upon the
first circular dependency issue - a half-assed effort - and decided
it's not worth it.

Brian Beck

unread,
Sep 16, 2008, 3:09:07 PM9/16/08
to Django developers
On Sep 16, 2:36 pm, Johannes Dollinger
<johannes.dollin...@einfallsreich.net> wrote:
> Why should django.template be refactored? Filter and Variable parsing
> is inconsistent. Many ad hoc parsers in defaulttags are fragile.
> Whitespace handling is ungraceful.
>
> The patch provided in #7806[1] splits __init__.py into separate
> modules[2] and introduces a TokenStream class that allows parsing of
> literals, vars (called lookups) and filter expressions.

Regardless of how it happens, I did notice when writing a patch for
#9093 that django.template really could use some refactoring. +1

Malcolm Tredinnick

unread,
Sep 16, 2008, 7:37:00 PM9/16/08
to django-d...@googlegroups.com

On Tue, 2008-09-16 at 20:36 +0200, Johannes Dollinger wrote:
> Why should django.template be refactored? Filter and Variable parsing
> is inconsistent. Many ad hoc parsers in defaulttags are fragile.
> Whitespace handling is ungraceful.
>
> The patch provided in #7806[1] splits __init__.py into separate
> modules[2] and introduces a TokenStream class that allows parsing of
> literals, vars (called lookups) and filter expressions.

As was pointed out the first time you brought this up, keep in mind that
there still need to be ways to manually control the lexing phase. Not
every template tag has the same requirements there.

Also, since the Variable class is part of the public API (we document
how to write custom template tags), you need to preserve backwards
compatibility. For example, anything written according to
custom-template-tags.txt should continue to work unchanged. There are
possibly some reasonable other methods in template/__init__.py that
should continue to remain available as well.

That being said, I'm completely in favour of a bit of shuffling in those
files to make things easier to understand and fix a bunch of little
inconsistencies. However, I'd encourage doing it in slightly smaller
steps than the original patches I read so that we can see that backwards
compatibility is maintained and do the smallest number of changes
necessary to tidy things up. Basically, keep the refactoring and the
feature/API addition portions distinct.

For example, it's useful (and necessary) to make the Variable class be a
lot more consistent with respect to also resolving the effect of apply
filters to variables (right now, anything using Variable doesn't have
filter effects applied, which was probably just an oversight
originally).

Helper methods that make it easy for the common cases to say "I expect
three paramters matching the 'x y as z' pattern where x and z are
strings and y must be an integer" would be very handy. This is where
method like your parse_int and parse_string methods are additions that
would be useful. Things like parse_expression is probably overkill for a
high-level API, since resolving the effect of filters should really just
be a natural part of Variable.resolve_context(). I'm not sure I see the
use-case at the moment for parse_expression; perhaps you could elaborate
with an example.

Regards,
Malcolm

Simon Willison

unread,
Sep 16, 2008, 7:55:50 PM9/16/08
to Django developers
On Sep 16, 7:36 pm, Johannes Dollinger
<johannes.dollin...@einfallsreich.net> wrote:
> @register.tag
> @uses_token_stream
> def mytag(parser, bits):
>      expr = bits.parse_expression(required=True)
>      return MyNode(expr)
>
> `uses_token_stream` replaces the Token argument to the parser  
> function with a TokenStream object.

I'm completely in favour of some refactoring in django.template, but
I'm not at all keen on that decorator syntax. Nested decorators get
ugly quickly in my opinion, especially considering our need to keep
supporting Python 2.3 syntax.

Would @register.tag(token_stream=True) work instead, or am I missing
something?

Cheers,

Simon

Johannes Dollinger

unread,
Sep 17, 2008, 8:26:10 AM9/17/08
to django-d...@googlegroups.com

Am 17.09.2008 um 01:37 schrieb Malcolm Tredinnick:

> As was pointed out the first time you brought this up, keep in mind
> that
> there still need to be ways to manually control the lexing phase. Not
> every template tag has the same requirements there.
>
> Also, since the Variable class is part of the public API (we document
> how to write custom template tags), you need to preserve backwards
> compatibility. For example, anything written according to
> custom-template-tags.txt should continue to work unchanged. There are
> possibly some reasonable other methods in template/__init__.py that
> should continue to remain available as well.

It is completely optional. If you don't use @uses_token_stream you'll
get a Token instance that can be parsed as hitherto. And afaics the
old API is still there and functional.

> That being said, I'm completely in favour of a bit of shuffling in
> those
> files to make things easier to understand and fix a bunch of little
> inconsistencies. However, I'd encourage doing it in slightly smaller
> steps than the original patches I read so that we can see that
> backwards
> compatibility is maintained and do the smallest number of changes
> necessary to tidy things up. Basically, keep the refactoring and the
> feature/API addition portions distinct.

Basically, agreed.

> For example, it's useful (and necessary) to make the Variable class
> be a
> lot more consistent with respect to also resolving the effect of apply
> filters to variables (right now, anything using Variable doesn't have
> filter effects applied, which was probably just an oversight
> originally).
> Helper methods that make it easy for the common cases to say "I expect
> three paramters matching the 'x y as z' pattern where x and z are
> strings and y must be an integer" would be very handy. This is where
> method like your parse_int and parse_string methods are additions that
> would be useful. Things like parse_expression is probably overkill
> for a
> high-level API, since resolving the effect of filters should really
> just
> be a natural part of Variable.resolve_context(). I'm not sure I see
> the
> use-case at the moment for parse_expression; perhaps you could
> elaborate
> with an example.

parse_expression() does not resolve filters. It returns an Expression
(Lookup, Literal, or FilterExpression) instance that has a resolve
(context) method.
Lookup handles looking up variables in a context. FilterExpression
groups an Expression and (func, arg) pairs. Literal boxes literal
values.

# Here's how your example could be parsed with the proposed API:
x = bits.parse_string(required=True)
y = bits.parse_int(required=True)
if not bits.pop_lexeme('as'):
raise TemplateSyntaxError
z = bits.parse_string(required=True)

# A {% with %} parser example:
def do_with(parser, bits):
expr = bits.parse_expression(required=True)
if not bits.pop_lexeme('as'): # this should be more compact
raise TemplateSyntaxError
name = bits.pop_name()
if not name:
raise TemplateSyntaxError
nodelist = parser.parse_nodelist(('endwith',))
return WithNode(expr, name, nodelist)
do_with = register.tag('with', uses_token_stream(do_with))

Johannes Dollinger

unread,
Sep 17, 2008, 8:27:28 AM9/17/08
to django-d...@googlegroups.com
> Would @register.tag(token_stream=True) work instead, or am I missing
> something?

Yes, that would work.

Ben Ford

unread,
Sep 17, 2008, 9:42:34 AM9/17/08
to django-d...@googlegroups.com
I take it that most are aware of:

http://lucumr.pocoo.org/cogitations/2008/09/16/why-jinja-is-not-django-and-why-django-should-have-a-look-at-it/

It seems like a very well thought out and thorough write up.

Ben

2008/9/17 Johannes Dollinger <johannes....@einfallsreich.net>


> Would @register.tag(token_stream=True) work instead, or am I missing
> something?

Yes, that would work.



Malcolm Tredinnick

unread,
Sep 17, 2008, 6:52:14 PM9/17/08
to django-d...@googlegroups.com

On Wed, 2008-09-17 at 14:42 +0100, Ben Ford wrote:
> I take it that most are aware of:
>
> http://lucumr.pocoo.org/cogitations/2008/09/16/why-jinja-is-not-django-and-why-django-should-have-a-look-at-it/
>
> It seems like a very well thought out and thorough write up.

Um .. welll. :-(

Parts of it are very well thought out and if it had been a post on "how
Jinja works" it would have been excellent. Other parts are completely
unconstrained by facts or acknowledgement that Django's standard
templates and Jinja's ones have different goals (which is important,
since this isn't an apples to apples comparison at all). So that the
flip side is on the record, here's my single contribution to it:

The good news is that, as far as I can see, all the actual bugs he notes
are already in tickets. I went through and checked quickly yesterday and
found all but one, which I'm pretty sure is in there, so I'll look again
today.

Whilst reading that, keep in mind that Armin apparently misunderstands
(or possibly just omits to clearly state) what a template object
instance is in Django. It's a state machine for rendering templates and
includes the current state as well as the transitions. This isn't an
invalid or particularly novel approach to executing a domain specific
language. In fact, it makes a lot of sense if you want to write
self-contained node-based state instances. Jinja and Django templates
have different compilation strategies and return different types of
objects and the difference is a vital distinction in any discussion of
appropriate usage. The blog post appears to imply that it would be
natural for any random data structure to be used as a global variable
(by extrapolation, since he doesn't establish that a Template instance
-- not a class, the instance of a compiled template -- is special in any
sense, so it must be somehow "normal") and that those structures will
work beautifully when multiple threads cause its internal state to be
modified.

This has two problems: (1) it's pretty much false by default (Python !=
Haskell, Erlang, etc and mutable objects are very normal in Python), and
(2) it would imply that using lots of globals to store stuff in
multi-threaded programs is a good idea that people should be doing more
often.

There's a name for that sort of programming style and it's not
"Excellent, Maintainable Code". :-)

Using the same logic, Python is not thread-safe at all and shouldn't be
used, since lists, dictionaries, sets, and pretty much any random
instance of a class object contain state in the instance that will
change upon use. Tuples and strings are specially noted to be immutable
and so are usable in those situations without extra locking, but they're
special-cases, not the norm, in a pass-by-reference language. Using
module-level globals sparingly is something we certainly encourage.

Fortunately, this isn't a real issue, since there's nowhere that we
suggest the assumption of immutability would be valid (and, really, a
decent programmer is going to know that any arbitrary data structures
have the same behaviour unless it's noted as immutable; minimal
experience says that making assumptions contrary to that always leads to
trouble). You create the template object as part of a view and render it
and things are (mostly -- see next para) fine. There's no real need to
put template instances into globals.

The "mostly" bit is because we do have some reset-ability issues with
things like the cycle tag -- so rendering it multiple times in the same
thread of execution, such as creating mass mail, will lead to differing
results.

Also, so that it's on the record: Armin denigrating Eric Holscher for
ticket #7501 was a low-blow for multiple reasons. One being that Armin
either misunderstood or ignored the type of object that a Template
instance is, as noted above. Another being that nominally avoiding
mentioning specifics whilst giving enough information that somebody
could easily find it out anyway is simply sleazy. It devalues Eric's
work for that part of the audience who don't know the history and makes
the author of the piece just look small. To fill in the missing pieces,
I explicitly asked Eric to change the title at the Lawrence Sprint, as I
was busy doing something else at the time and had been meaning to change
it for a while and Eric was going through doing the scut work of making
the changes through the Web browser interface whilst Jacob and I fed him
decisions on various points. I made this change in full awareness of the
difference between multi-threaded and multi-run use-cases and knowing
that Template instances are just like lists, dictionaries,
sets, ...(insert 150 other data structures here).

All the bits in that blog post about Jinja, once understood in the
context of why Jinja is a different templating approach to Django, were
well done in that article. Reading the Jinja source is a pretty
interesting thing to do, too and people should definitely play with the
language if they haven't already. But it's not a one-for-one replacement
for Django templates, nor is it a superset of functionality in either
direction. It has different goals and neither Jinja nor Djangos' goals
are invaildated by the other existing.

The bugs noted in Django templates are things we've noted down and
Johannes changes address a large number of them. We've always accepted
speed improvements that don't completely break things, but we also
realise that template rendering is just one component of the timeslice
for a request and isn't the major piece unless your net time is pretty
tiny anyway. With sensible caching strategy and real-world data and
computation stuff, the amortised effect is that it's "fast enough", for
most cases and making it possible to easily enough use a different
templating language when it's not. That's a pretty good goal. Django
ships with a template language for designers, not for Python
programmers; another large difference between Django and Jinja
templates, leading to the somewhat-perceived dichotomy over "logic in
templates" (although in a lot of cases the big arguments people have
come down to which template tags and filters should ship by default).

It's always nice when somebody sits down and write out an holistic state
of play of a particular portion of anything. It's very encouraging when,
reading through, you realise that there aren't any really big surprises
in the factual areas and it helps clarify where the differences of
opinion are. Sanity checks of that nature cannot be undervalued. They
can be written in a less confrontational style, however.

Regards,
Malcolm

zvoase

unread,
Sep 17, 2008, 8:02:51 PM9/17/08
to Django developers
I don't know if anyone's noticed, but the templating language seems
(at least from a usage standpoint) to be a lot like a Lisp
interpreter.
I know that sounds completely weird and random, but it's true. It's
like a very small subset of Scheme or something (the small subset
thing is necessary because you're not trying to put too much logic in
the template). I suppose the main difference is that blocks are
started and ended with explicit {% sometag %} and {% endsometag %}
statements instead of opening and closing parentheses.
If you need to refactor any of it, you might want to look at some
implementations of Lisp and steal some ideas.

Just a thought from a Lisper :)

Regards,
Zack

On Sep 18, 12:52 am, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
> On Wed, 2008-09-17 at 14:42 +0100, Ben Ford wrote:
> > I take it that most are aware of:
>
> >http://lucumr.pocoo.org/cogitations/2008/09/16/why-jinja-is-not-djang...

alex....@gmail.com

unread,
Sep 17, 2008, 8:18:02 PM9/17/08
to Django developers
If I follow you are saying that it is lispy in that flow control
statements and functions are handled just the same way(meaning you can
define your own statements if you like)?

zvoase

unread,
Sep 17, 2008, 8:25:59 PM9/17/08
to Django developers
I guess so, but also from the overall *feel* of using it; I find
myself switching into Lisp mode when I use it. That probably makes no
sense, but it's just this feeling I had. I guess it feels as though
there's some big eval/apply machine behind it all. And the fact that
templates are compiled to a Template object reminds me of the fact
that all lisp programs are really just data structures. Template
Contexts also feel like frame stacks (in fact, they're near as dammit
an implementation of frame stacks).

On Sep 18, 2:18 am, "alex.gay...@gmail.com" <alex.gay...@gmail.com>
wrote:

Armin Ronacher

unread,
Sep 18, 2008, 7:25:47 PM9/18/08
to Django developers
On Sep 18, 12:52 am, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
> On Wed, 2008-09-17 at 14:42 +0100, Ben Ford wrote:
> Parts of it are very well thought out and if it had been a post on "how
> Jinja works" it would have been excellent. Other parts are completely
> unconstrained by facts or acknowledgement that Django's standard
> templates and Jinja's ones have different goals (which is important,
> since this isn't an apples to apples comparison at all).
I'm starting to feel like Don Quixote here (the Windmills being a
community that learned to live with the implications of the
limitations of their beloved template engine). After my blog post I
continued to dig in the Django sources and had to notice that it's
impossible to reused Template objects at all so you are pretty much
forced to reparse them over and over again. (Not reparsing would for
example also break the FilterExpression embedded i18n support because
it translates on compilation, not on resolving)

With that in mind you can pretty much everything said about thread
safety in my blog post.

The rest of my reply is missing now. I actually wrote a mail here
that was a lot longer than my blog post from yesterday but noticed
that I repeated myself with slightly different examples. I don't want
to start the discussions again because I know the first reply will be
again that missing support for expressions is good, that the AST
evaluator is a blessing and in general the Django template engine is
the best ever happened to mankind. I know I'm sort of exaggerating
here but in general that's what I read out of the comments I've seen
on the reddit discussion thread about that topic.

Please apologise my blog post, it wasn't meant to criticise the Django
template engine concept in any way, just to inspire a possible
improved version of Django templates. I think Jinja does solve a few
things better than Django (and probably some worse) but I can't be a
mistake to look at how things work there to get an inspiration of how
to make things better. The primary inspiration for myself for Jinja
was Django, and I'm not ashamed to say that. When I started
developing Jinja it was a simple port of the concept into a standalone
library not depending on Django with unicode support. What you can
see now in Jinja2 is the second iteration of a vastly changed template
engine, that however is still based on the same principles the Django
template engine is: no XML, template designer friendly, easy to get
started, extensible.

If I can help out in any way to get some of the Jinja concepts ported
to Django, or just explain why things work that way and not different
in detail I would love to do so. I think Django can't lose, it just
takes someone to have a look at it.

Regards,
Armin

Vinay Sajip

unread,
Sep 19, 2008, 5:15:47 AM9/19/08
to Django developers


On Sep 17, 11:52 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
> On Wed, 2008-09-17 at 14:42 +0100, Ben Ford wrote:
> > I take it that most are aware of:
>
> >http://lucumr.pocoo.org/cogitations/2008/09/16/why-jinja-is-not-djang...
>
> > It seems like a very well thought out and thorough write up.
>
> Um .. welll. :-(
>
> Parts of it are very well thought out and if it had been a post on "how
> Jinja works" it would have been excellent. Other parts are completely
> unconstrained by facts or acknowledgement that Django's standard
> templates and Jinja's ones have different goals (which is important,
> since this isn't an apples to apples comparison at all).

There are philosophical differences between Django and Jinja
templating about how much power there should be in the templating
engine. Django's philsophy is to keep the power to the minimum
required - "templates are not programs" - and I think we all
understand this, but apart from that point, can you [or one of the
other core devs] be more specific about what you think these different
goals are?

>
> The good news is that, as far as I can see, all the actual bugs he notes
> are already in tickets. I went through and checked quickly yesterday and
> found all but one, which I'm pretty sure is in there, so I'll look again
> today.
>
> Whilst reading that, keep in mind that Armin apparently misunderstands
> (or possibly just omits to clearly state) what a template object
> instance is in Django. It's a state machine for rendering templates and
> includes the current state as well as the transitions. This isn't an
> invalid or particularly novel approach to executing a domain specific
> language. In fact, it makes a lot of sense if you want to write
> self-contained node-based state instances. Jinja and Django templates
> have different compilation strategies and return different types of
> objects and the difference is a vital distinction in any discussion of
> appropriate usage.

I don't think that template users care about the internals as much as
they care about correct behaviour, including in a threaded
environment.

> The blog post appears to imply that it would be
> natural for any random data structure to be used as a global variable
> (by extrapolation, since he doesn't establish that a Template instance
> -- not a class, the instance of a compiled template -- is special in any
> sense, so it must be somehow "normal") and that those structures will
> work beautifully when multiple threads cause its internal state to be
> modified.
>
> This has two problems: (1) it's pretty much false by default (Python !=
> Haskell, Erlang, etc and mutable objects are very normal in Python), and
> (2) it would imply that using lots of globals to store stuff in
> multi-threaded programs is a good idea that people should be doing more
> often.
>
> There's a name for that sort of programming style and it's not
> "Excellent, Maintainable Code". :-)
>
> Using the same logic, Python is not thread-safe at all and shouldn't be
> used, since lists, dictionaries, sets, and pretty much any random
> instance of a class object contain state in the instance that will
> change upon use. Tuples and strings are specially noted to be immutable
> and so are usable in those situations without extra locking, but they're
> special-cases, not the norm, in a pass-by-reference language. Using
> module-level globals sparingly is something we certainly encourage.

I'm not sure how relevant these references to Haskell, Erlang etc.
are. Python, like most other languages, does not guarantee intrinsic
thread safety of its data structures, nor do we expect this. Armin's
point relates to *shared* data across threads, viz. the templates
themselves. When concurrent requests are handled by different threads,
they create separate instances of instrinsically unsafe data
structures, e.g. context and response objects, but as these are not
shared among threads, we don't care that they're not thread-safe
implementations. However, the templates *are* shared, and so they have
a responsibility not to store state in themselves, especially as the
context is there for precisely that purpose. Otherwise, when two
concurrent requests invoke the same template containing a cycle tag,
the cycle tag will not perform properly because the different threads
trample over each other's state. This is, it seems to me, quite
different to needing to set the cycle tag to a known state at the
start of template execution.
See my comment above. I don't believe it is relevant to invoke the
lack of intrinsic thread safety in lists, dictionaries etc. as the
myriad instances of these objects are not shared between threads
during concurrent processing of requests. The templates, however, are.
Therefore, they should be thread-safe, in the sense of allowing the
same instance to be used by multiple concurrent threads, without any
surprises. This can be easily achieved by storing any needed state in
the context [or other private object created per template execution].
If it is not, I'm not sure that Django templates are thread-safe, and
Armin would appear to have a valid point.

Armin could certainly have chosen better words when commenting on the
change to the #7501 ticket summary, but there is a real point here -
there are actually (it seems) two bugs, which have been conflated by
the change in title. The bug Malcolm is talking about is not setting
the cycle tag to a known state at the start of template execution.
This is not the same as the bug Armin reported - that of the cycle
state being trampled on by different threads during their concurrent
execution, due to the cycle tag's state being stored in the template
itself rather than in a per-template-execution object such as the
context.

Looking at the code of the cycle tag,

class CycleNode(Node):
def __init__(self, cyclevars, variable_name=None):
self.cycle_iter = itertools_cycle([Variable(v) for v in
cyclevars]) # Malcolm's bug (possibly)
self.variable_name = variable_name

def render(self, context):
value = self.cycle_iter.next().resolve(context) # Armin's bug
if self.variable_name:
context[self.variable_name] = value
return value

The first commented line may be Malcolm's bug - the cycle_iter
instance should be initialised to a known state, which should happen
whenever a template is executed, which may or may not be when the
CycleNode instance is created (which, I would guess, is at template
load/parse time).

The second commented line is probably Armin's bug. The cycle_iter
instance should not be held in the CycleNode instance, because that,
as part of the template, is shared between threads. Instead, the
cycle_iter should be held in the context or another separate object
which is specifically for use in one template execution only.

In general, couldn't one assert that any statement in the render
method which mutates self [ above, this is done by
self.cycle_iter.next() ] could lead to incorrect behaviour? It would
be useful to know what other examples Armin has found. For example, I
see in IfChangedNode.render the assignments "self._last_seen =
compare_to" and "self._last_seen = None" which could potentially cause
the same kind of problem, if the same node instance is accessed by
multiple threads.

I'm not looking to step on anyone's toes here - just trying to clarify
my understanding of the thread safety (or otherwise) of the current
template engine implementation.

Regards,

Vinay Sajip

Andreas

unread,
Sep 19, 2008, 4:05:19 AM9/19/08
to Django developers
What really annoys me is all the talk about how important it is that
its designed for non-programmers but if i recall corectly jacob said
on djangocon that they never aimed for making django a general purpose
webframework, they just made something that fit their needs. With that
said, how many django projects actually involves "template designers"?
Such a job position doesnt exists here in sweden. I bet my five cents
that in most cases the programmer also makes the templates with
semantic html, and the css and the js. Most developers today has
competence that spans all over these areas. So shouldnt we make
something that fit our needs instead of making something for someone
else? Else id like to see some "template designers" in this
discussion.

(Of course Its loose coupled and i can use anything i want, which i
actually do)

Jacob Kaplan-Moss

unread,
Sep 19, 2008, 11:16:45 AM9/19/08
to django-d...@googlegroups.com
On Fri, Sep 19, 2008 at 4:15 AM, Vinay Sajip <vinay...@yahoo.co.uk> wrote:
> There are philosophical differences between Django and Jinja
> templating about how much power there should be in the templating
> engine. Django's philsophy is to keep the power to the minimum
> required - "templates are not programs" - and I think we all
> understand this, but apart from that point, can you [or one of the
> other core devs] be more specific about what you think these different
> goals are?

http://docs.djangoproject.com/en/dev/misc/design-philosophies/#template-system

In particular:

"""
The goal is not to invent a programming language. The goal is to offer
just enough programming-esque functionality, such as branching and
looping, that is essential for making presentation-related decisions.

The Django template system recognizes that templates are most often
written by designers, not programmers, and therefore should not assume
Python knowledge.
"""

Jacob

Vinay Sajip

unread,
Sep 19, 2008, 4:22:44 PM9/19/08
to Django developers


On Sep 19, 4:16 pm, "Jacob Kaplan-Moss" <jacob.kaplanm...@gmail.com>
wrote:
> On Fri, Sep 19, 2008 at 4:15 AM, Vinay Sajip <vinay_sa...@yahoo.co.uk> wrote:
> > There are philosophical differences between Django and Jinja
> > templating about how much power there should be in the templating
> > engine. Django's philsophy is to keep the power to the minimum
> > required - "templates are not programs" - and I think we all
> > understand this, but apart from that point, can you [or one of the
> > other core devs] be more specific about what you think these different
> > goals are?
>
> http://docs.djangoproject.com/en/dev/misc/design-philosophies/#templa...
>
> In particular:
>
> """
> The goal is not to invent a programming language. The goal is to offer
> just enough programming-esque functionality, such as branching and
> looping, that is essential for making presentation-related decisions.
>
> The Django template system recognizes that templates are most often
> written by designers, not programmers, and therefore should not assume
> Python knowledge.
> """
>

Jacob,

Thanks for replying - I did say "apart from the templates-are-not-
programs point", which you've emphasised here. And I was asking about
*differences* between Jinja and Django, because that was what
Malcolm's post was about. From the information that you linked to, on
a point-by-point reading and comparing with Jinja, it seems that Jinja
was avowedly inspired by Django templates and, in large measure,
follows the same philosophy. So I don't think Armin's comparison was
such an apples-and-oranges one. The one big difference is in Jinja
allowing more complex expressions, including function calls, in
templates. If this expressive power was toned down, there would appear
to be little difference other than implementation details. (I won't go
through the points blow-by-blow, unless anyone expresses an interest.)

On the question of expressive power, is "elif" explicitly left out
because it's considered an advanced concept for designers to
understand - more complex than "if" and "else"? Or can it be added
without breaking the design principles behind Django templates? And is
it really strongly felt that whereas "{% ifequal a b %}" is OK for
designers to understand, "{% if a = b %}" is considered too advanced?

From Armin's recent post

http://groups.google.com/group/django-developers/browse_thread/thread/ba0644b835e3ec40

it would appear that Django templates are not safe if shared across
multiple threads, so they have to be loaded anew for each request.
This is the default, but it extracts a moderately large performance
penalty. But caching Template objects to improve performance would be
a no-no, so wouldn't it be worth at least mentioning it somewhere in
the docs?

Also, from Karen Tracey's recent post

http://groups.google.com/group/django-developers/browse_thread/thread/f7d638d1f79ec43c

it appears there may be scenarios where even with default settings,
there might be situations where render() code gets called on the same
node by different threads.

Regards,

Vinay Sajip
Reply all
Reply to author
Forward
0 new messages