templates and html escaping

253 views
Skip to first unread message

Michael Radziej

unread,
Mar 2, 2006, 4:16:55 PM3/2/06
to django-d...@googlegroups.com
Hi,

the behaviour of django for fields that contain html markup ("<" ...)
is wrong, at least in version 0.91.

First, the admin pages do not html escape these. There might be
corner cases when this is intended, but it seems very dangerous.

Second, regarding Template, I think the default not to html escape
variable nodes is not the right one. I think the normal case should
be to escape these, unless otherwise noted.

I've written a derived HTMLTemplate class for myself that uses a
different VariableNode to achieve this effect, and attached this if
you're interested.

Now, did I miss something and is this already fixed? Should this be
treated differently? How do other people handle this?

If you're positive about my way I'll polish my stuff up for inclusion
with the contribs.


Michael

htmltemplate.py

Jacob Kaplan-Moss

unread,
Mar 3, 2006, 9:45:24 AM3/3/06
to django-d...@googlegroups.com
On Mar 2, 2006, at 3:16 PM, Michael Radziej wrote:
> Now, did I miss something and is this already fixed? Should this be
> treated differently? How do other people handle this?

The problem in the admin was fixed in [1982]: http://
code.djangoproject.com/changeset/1982; in your own templates you'll
want to use the "escape" filter (http://www.djangoproject.com/
documentation/templates/#escape) on any potentially dangerous entries.

Why not do it for all variables? At times you want to pass chunks of
HTML into a template that get displayed raw. I don't think the
behavior you suggest should be default, but do you have any ideas on
how to make it optional? It certainly could be useful in certain cases.

Jacob

Gábor Farkas

unread,
Mar 3, 2006, 10:27:11 AM3/3/06
to django-d...@googlegroups.com
Jacob Kaplan-Moss wrote:
> On Mar 2, 2006, at 3:16 PM, Michael Radziej wrote:
>> Now, did I miss something and is this already fixed? Should this be
>> treated differently? How do other people handle this?
>
> The problem in the admin was fixed in [1982]: http://
> code.djangoproject.com/changeset/1982; in your own templates you'll
> want to use the "escape" filter (http://www.djangoproject.com/
> documentation/templates/#escape) on any potentially dangerous entries.
>
> Why not do it for all variables? At times you want to pass chunks of
> HTML into a template that get displayed raw. I don't think the
> behavior you suggest should be default,

maybe a stupid question, but why not?

for me it seems that most of time it's fine to escape the text, so for
me it would make sense to have it on by default...and for the cases when
i wan to enter raw data into the page template, we could have an
'unescape' filter..we could call it 'raw' for example..

what do you think?

gabor

Gábor Farkas

unread,
Mar 3, 2006, 10:30:16 AM3/3/06
to django-d...@googlegroups.com
Gábor Farkas wrote:
> Jacob Kaplan-Moss wrote:
>> On Mar 2, 2006, at 3:16 PM, Michael Radziej wrote:
>>> Now, did I miss something and is this already fixed? Should this be
>>> treated differently? How do other people handle this?
>>
>> The problem in the admin was fixed in [1982]: http://
>> code.djangoproject.com/changeset/1982; in your own templates you'll
>> want to use the "escape" filter (http://www.djangoproject.com/
>> documentation/templates/#escape) on any potentially dangerous entries.
>>
>> Why not do it for all variables? At times you want to pass chunks of
>> HTML into a template that get displayed raw. I don't think the
>> behavior you suggest should be default,
>
> maybe a stupid question, but why not?

<snip>

after i sent my response, i've read the original posters mail (i should
have done it at the beginning) and i see that i'm just repeating his
words.i'm sorry.

gabor

Michael Radziej

unread,
Mar 3, 2006, 11:04:50 AM3/3/06
to django-d...@googlegroups.com
Jacob Kaplan-Moss wrote:

> The problem in the admin was fixed in [1982]: http://
> code.djangoproject.com/changeset/1982;

thanks (and sorry--I'll start using the svn then)

> in your own templates you'll
> want to use the "escape" filter (http://www.djangoproject.com/
> documentation/templates/#escape) on any potentially dangerous entries.
>
> Why not do it for all variables? At times you want to pass chunks of
> HTML into a template that get displayed raw. I don't think the
> behavior you suggest should be default, but do you have any ideas on
> how to make it optional? It certainly could be useful in certain
> cases.

Thanks for your input!

The problem I see is that it's potentially dangerous to use the non-
escaped strings, and I'd prefer to have the safe thing as default
(and, isn't it the most common case?)

My idea to make it optional is included (exerimental!) in the
attachment to my original mail. In summary:

- provide an additional template class (I called it HTMLTemplate, but
I'm not sure if this is a good name), together with loader support.

- there's also a subclass of unicode, currently called
HTMLEscapedString. If you have something that is already in html
format and should not be escaped, use this class.

- the template (or rather the associated HTMLVariableNode) will apply
the template filter if the context entry is not already an instance
of HTMLEscapedString


I'd like to see some comments from more seasoned django developers
and whether you feel this is the wrong way or if you are interested
in this some way or the other.

Michael

fre...@pythonware.com

unread,
Mar 3, 2006, 12:05:03 PM3/3/06
to Django developers
Jacob Kaplan-Moss wrote:

> Why not do it for all variables? At times you want to pass chunks of
> HTML into a template that get displayed raw. I don't think the
> behavior you suggest should be default, but do you have any ideas on
> how to make it optional?

the standard solution is to use a wrapper for strings that you
*don't* want escaped. see e.g. the section on "HTML templates"
on this page:


http://www.mems-exchange.org/software/quixote/Quixote-2.2.tar.gz/Quixote-2.2/doc/PTL.html

and the h8 class in Qpy:


http://www.mems-exchange.org/software/qpy/qpy-1.2.tar.gz/qpy-1.2/README.txt

for some bullet-point background, see this presentation:

Eliminating XSS Holes Eliminating XSS Holes
http://arctrix.com/nas/talks/htmltext.pdf

(fwiw, I'm +0 on adding this to Django's template system.)

</F>

Max Battcher

unread,
Mar 3, 2006, 12:25:15 PM3/3/06
to django-d...@googlegroups.com
On 3/3/06, fre...@pythonware.com <fre...@pythonware.com> wrote:
>
> Jacob Kaplan-Moss wrote:
>
> > Why not do it for all variables? At times you want to pass chunks of
> > HTML into a template that get displayed raw. I don't think the
> > behavior you suggest should be default, but do you have any ideas on
> > how to make it optional?
>
> the standard solution is to use a wrapper for strings that you
> *don't* want escaped. see e.g. the section on "HTML templates"
> on this page:

The other problem is that not everyone wants the same escape. For
instance, there are those of us that {% load markup %} and then
|restructuredtext (or |markdown or |textile) just about everything,
and auto-HTML escape could certainly cause havoc here.

--
--Max Battcher--
http://www.worldmaker.net/
All progress is based upon a universal innate desire on the part of
every organism to live beyond its income. --Samuel Butler

gabor

unread,
Mar 3, 2006, 12:45:36 PM3/3/06
to django-d...@googlegroups.com
Max Battcher wrote:
> On 3/3/06, fre...@pythonware.com <fre...@pythonware.com> wrote:
>> Jacob Kaplan-Moss wrote:
>>
>>> Why not do it for all variables? At times you want to pass chunks of
>>> HTML into a template that get displayed raw. I don't think the
>>> behavior you suggest should be default, but do you have any ideas on
>>> how to make it optional?
>> the standard solution is to use a wrapper for strings that you
>> *don't* want escaped. see e.g. the section on "HTML templates"
>> on this page:
>
> The other problem is that not everyone wants the same escape. For
> instance, there are those of us that {% load markup %} and then
> |restructuredtext (or |markdown or |textile) just about everything,
> and auto-HTML escape could certainly cause havoc here.

those filters could first de-html-escape the content.

i understand that it seems silly for this configuration. i only wanted
to demonstrate that even with auto-html-escaping you can still use those
markups fine.


gabor

Jacob Kaplan-Moss

unread,
Mar 3, 2006, 12:53:07 PM3/3/06
to django-d...@googlegroups.com
Hey folks --

I've thought about this a bit more; lots of good points have been
raised. What I'd like to do would be to add a new tag that takes
care of auto-escaping variables used within it. Something like::

{% autoescape %}
Any {{ variables }} used within this {{ block }} will be
automatically {{ escaped }}
{% endautoescape %}

How does that sound?

Jacob

fre...@pythonware.com

unread,
Mar 3, 2006, 1:17:43 PM3/3/06
to Django developers
Max Battcher wrote:

> > the standard solution is to use a wrapper for strings that you
> > *don't* want escaped. see e.g. the section on "HTML templates"
> > on this page:
>
> The other problem is that not everyone wants the same escape. For
> instance, there are those of us that {% load markup %} and then
> |restructuredtext (or |markdown or |textile) just about everything,
> and auto-HTML escape could certainly cause havoc here.

such filters should of course wrap their return values before returning
them.

</F>

Julio Nobrega

unread,
Mar 3, 2006, 1:19:25 PM3/3/06
to django-d...@googlegroups.com
Sounds good, but I just don't like the name. How about "escape", or
"escapeblock"?

And how about...

{% autoescape except "br,p,table" %}

?


--
Julio Nobrega - http://www.inerciasensorial.com.br

Christopher Lenz

unread,
Mar 3, 2006, 1:38:36 PM3/3/06
to django-d...@googlegroups.com

Not quite enough IMHO.

I strongly agree with some of the other posters in this thread that
any kind of output in an HTML-template should be HTML-escaped by
default. Values that contain markup that should not be escaped should
be flagged explicitly.

In Trac 0.9.3 we added exactly this kind of behavior to greatly
eliminate the risk for XSS vulnerabilities. See:

<http://projects.edgewall.com/trac/browser/trunk/trac/util.py#L56>

When the template engine sees a value that is a str/unicode, it gets
escaped. If it gets a value that's an instance of the "Markup" class,
it gets passed through as-is.

Adding such behavior to the Django template processor would be great.
It's *much* easier to forget to escape something than to forget to
flag it as not-needing-escaping.

Cheers,
Chris
--
Christopher Lenz
cmlenz at gmx.de
http://www.cmlenz.net/

Arthur

unread,
Mar 27, 2006, 7:50:06 AM3/27/06
to django-d...@googlegroups.com
> > I've thought about this a bit more; lots of good points have been
> > raised. What I'd like to do would be to add a new tag that takes
> > care of auto-escaping variables used within it. Something like::
> >
> > {% autoescape %}
> > Any {{ variables }} used within this {{ block }} will be
> > automatically {{ escaped }}
> > {% endautoescape %}
> >
> > How does that sound?
>
> Not quite enough IMHO.
>
> I strongly agree with some of the other posters in this thread that
> any kind of output in an HTML-template should be HTML-escaped by
> default. Values that contain markup that should not be escaped should
> be flagged explicitly.
>
> In Trac 0.9.3 we added exactly this kind of behavior to greatly
> eliminate the risk for XSS vulnerabilities. See:
>
> <http://projects.edgewall.com/trac/browser/trunk/trac/util.py#L56>
>
> When the template engine sees a value that is a str/unicode, it gets
> escaped. If it gets a value that's an instance of the "Markup" class,
> it gets passed through as-is.
>
> Adding such behavior to the Django template processor would be great.
> It's *much* easier to forget to escape something than to forget to
> flag it as not-needing-escaping.

Sorry for being late to the discussion. Just wanted to give a +1 on
having the possibility to escape (or not escape) whole blocks and a +0
on escaping everything per default. Is there already an open ticket on
this?

Arthur

Reply all
Reply to author
Forward
0 new messages