Keep template tags grammar (for simple_tag, assignment_tag and inclusion_tag. Extend with block_tag). Ticket 20434 and 20435.

[Jonathan Slenders]

Hi all,

For being able to do AST manupulations on templates we need to keep the grammar of templatetags somewhere.

During the sprints, Russel told me to attach these as a property on the functions that go into Library.tags. (I would attach an instance of a class named TemplateTagMeta.)

For those who wonder what the applications are. They are mainly for template preprocessing, HTML validation, and code completion in editors, when writing Django templates.

For simple_tag, assignment_tag and inclusion tag. The grammar rules are easy.

For any other custom tag it's right now impossible to extract the grammar.

The reason is mainly that it is too complicated to define custom template tags which consist of an open/close tag.

It should certainly not be necessary to touch the (parser, token) tuple.

I propose that we introduce a new, more user-friendly way of defining these tags. I created register.block_tag, but I'm not sure about the API. This works, and it's is all we need for the built-in "if" template tag.

@register.block_tag('if elif* else? endif')

def do_if(parser, blocks):

    conditions_nodelists = []

    for name, bits, nodelist in blocks:

        if name in ('if', 'elif'):

            condition = TemplateIfParser(parser, bits).parse()

        else:

            condition = None

        conditions_nodelists.append( (condition, nodelist) )

    return IfNode(conditions_nodelists)

The advantage is that it's both much more easy to write custom template tags which have one or more "bodies", and that this declaration allows us to extract the grammar rules of templates. (Which can be used for preprocessing, or HTML validation.)

Note that we don't deprecate the old way of writing template tags. It's a public API. But we should discourage custom token/parser manupulation in custom template tags. 

- What do you think about the name "block_tag"? I'm not sure. tag_with_grammar is too long, and the tag function itself cannot be changed anymore.

- What do you think about the syntax of the grammar. The space-separated notation is more often used in the Python core.

- Is the "parser" parameter really necessary. At least we need it in TemplateIfParser. But Why?

- register.block_tag works as well if the templatetags don't have a body. In that case, there's only one iteration through blocks, where nodelist is an empty list. Would this imply we need another name for "block_tag" or not? Otherwise we need something else for -- for instance -- the csrf_token template tag.

I'd also like to add a new management command to Django which prints the AST for templates. But in a human readable way. I think this would help a lot for developers to understand how the template tags of third party libraries work. And further, it would make it easier for people to write autocompletion plug-ins for a lot of editors.

What do you think would be good as a format, maybe the following?

<builtin>

if elif* else? endif

for empty? endfor

...

<admin>

...

Further, the following line does not say anything about the template tag parameters. 

@register.block_tag('if elif* else? endif')

Should we add some syntax to tell which template tags have or don't have any parameters, or leave that to the implementation of block_tag?

Cheers,

Jonathan

[Jonathan Slenders]

Update: NEW PROPOSAL.

After extending and refactoring template.Library, and using block_tag as described above for almost all the built-in tags, I came to the conclusion that we can improve the template tag definitions even more. I would introduce some thing as class based template tags.

Actually, all of the template tags return a Node instance after parsing. And almost all the template tag that we have, can be expressed as a grammar like this: 'if elif* else? endif'. (Exceptions are comment and verbatim, which ignore the inner template tags, and trans/blocktrans which have some backward compatibility code in there.)

There is no reason for not adding some abstraction level to the parser, as long as it's well implemented. The node tree, after parsing will be exactly the same. It's just the compile-functions which can be abstracted.

I propose creating a class TemplateTag, which inherits simply from Node. Actually, it's a node that was created because the template tag was found in the template. TemplateTag.__init__ will receive the parts, like for instance:

class IfTemplateTag(TemplateTag):

    grammar = 'if elif* else? endif'

    def __init__(self, parser, parts):

        self.conditions_nodelist = []

        for p in parts:

            if p.name in ('if', 'elif'):

                condition = TemplateIfParser(parser, p.args).parse()                self.conditions_nodelist.append( (condition, p.nodelist) )

            elif p.name == 'else':

                self.conditions_nodelist.append( (None, p.nodelist) )

library.register(IfTemplateTag)

(Probably, I'll make it TemplateTag.parse, instead of __init__, so that we can save the nodelists to the class and make the nodelists property automatically.)

Register would call IfTemplateTag.create_compile_func(), and add it to library.tags, and this compile function will have a pointer to the Grammar and TemplateTag class for introspection.

It's powerful. The "with"-template tag, will probably be not much more than adding a ContextMixin to the TemplateTag.

If no-one is against, I go on improving and implementing this. I strongly believe that there's a lot of room for improvement here, without loosing backwards compatibility or performance.

QUESTION: Do we consider the classes IfNode, IfEqualNode, etc... part of the public API? They don't appear anywhere in the documentation.

It is because if I implement the new template tag definitions, I'l like to refactor all the built-in template tags to use the new style. (As using more of Django -- inside Django.)

And this can probably break some monkey-patches on django for those who do it.

Cheers,

Jonathan

[Jonathan Slenders]

If someone has some spare time. Take a look at this ticket:

https://code.djangoproject.com/ticket/20434

The implementation is finished and consists of 4 patches. See the ticket for more information. Thanks!