Apply directive in templates and UnicodeDecode Errors

[DLC]

I'm trying to create a template function that turns @mentions or hash tags into their corresponding href links (sample code here: https://gist.github.com/3828280#file_p.py), but I've run into a snag.

It works most of the time, but when I use my own {% apply *function* %} {% end %} I sometimes run into an error like:

 

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

Here is a traceback: https://gist.github.com/3828280#file_gistfile1.pytb

And here's the general form of the template: https://gist.github.com/3828280#file_t.html

If I remove that apply function, I no longer get the error. I'm not quite what causes the error though. I tried using tornado.escape.linkify in it's place, but I still get the error, and so believe I may be doing something wrong, or it may be a bug in the template rendering code.

I noted in the documentation http://www.tornadoweb.org/documentation/template.html#syntax-reference it cautions using apply blocks in for loops with breaks or continues, however I do not use them. I do extend from a base template.

If anyone has any insight, would love to hear!

D

[DLC]

Hmm, I've tried now doing it with a pass through function that just returns the plain text eg {% apply passthorugh %} {% end %} and it works fine, so my guess is that something to do with regex usage is what's causing the problem, as linkify also uses regex and causes the error even though they do escape.xhtml_escape and escape._unicode the output.

[DLC]

Well I had to ship something so I just did this and threw it into a module

{% for word in comment["textContent"].split() %}

{% if word.startswith("@") and len(word) > 1 %}<a href="/{{word[1:]}}">{{word}}</a>{% elif word.startswith("#") and len(word) > 1%}<a href="/tag/{{word[1:]}}">{{word}}</a> {% else %}{{word}}{% end %}

{% end %}

Personally I'm not a fan of the solution, so if any of you figure this one out, would love to hear

[Ben Darnell]

This looks like a bug in the template system. Does it work if you
return utf8(_LINK_RE.sub())? The problem looks like the template
system generally turns unicode strings into utf8 for you, but not for
the output of apply blocks. It'll work as long as you either
pre-encode your strings or everything happens to be ascii-only, but it
will blow up if you get any non-ascii characters in a unicode string.

-Ben

[Dmitri Cherniak]

Yes that did work. Thanks Ben!

I've put in a pull request https://github.com/facebook/tornado/pull/606 which solved my problem.