[Zope-dev] Content Type Meta tag stripping in zope.pagetemplate
Miano Njoka
I'm a fairly new zope developer, came across a "bug" in my application
that <meta http-equiv="content-type" content="text/html;charset=UTF-8"
/> tags were being stripped out from ZPT templates. Is there a reason
for this? This is done in the _prepare_html function of
zope.pagetemplate.pagetemplatefile.PageTemplateFile. My application
produces XHTML that contains non-ASCII characters that is then used by
other applications so it needs to have the content type set on the
document itself in addition to the HTTP headers.
Secondly, finding and stripping of the meta tag is done using a regular
expression so simply changing the order of the attributes on the
<meta> tag would make the reg-exp not match.
Attached is a patch that uses HTMLParser to find the content type meta
tag instead of a regex. It stops parsing the html as soon as it
encounters the required meta tag.
Miano
Fred Drake
> <meta http-equiv="content-type" content="text/html;charset=UTF-8"
> /> tags were being stripped out from ZPT templates. Is there a reason
> for this?
As I recall, the rationale goes like this:
1. We're sniffing the input encoding from the charset setting.
2. We're storing the content-type on the instance (I hope tihs
is still true).
3. The template/application/publisher is responsible for
delivering the the output with an appropriate content-type
header.
--
Fred L. Drake, Jr. <fred at fdrake.net>
"A storm broke loose in my mind." --Albert Einstein
_______________________________________________
Zope-Dev maillist - Zope...@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Miano Njoka
> On Wed, Feb 22, 2012 at 10:28 AM, Miano Njoka <miano...@gmail.com> wrote:
>> <meta http-equiv="content-type" content="text/html;charset=UTF-8"
>> /> tags were being stripped out from ZPT templates. Is there a reason
>> for this?
>
> As I recall, the rationale goes like this:
>
> 1. We're sniffing the input encoding from the charset setting.
>
> 2. We're storing the content-type on the instance (I hope tihs
> is still true).
>
> 3. The template/application/publisher is responsible for
> delivering the the output with an appropriate content-type
> header.
Yes, this is true, but why strip out the meta tag from the resulting HTML?
Fred Drake
> Yes, this is true, but why strip out the meta tag from the resulting HTML?
Two reasons:
1. It may be incorrect.
2. If multiple templates are used to construct a response, different
values may be included from each template, which may be inconsistent.
Since the meta element is unnecessary, it seemed better to leave it out
of the result, and rely on other components to render the correct values
without requiring them to insert correct values into the rendered template.
(The publisher, for instance, shouldn't need to know how to edit that into
the finished HTML.)
-Fred
--
Fred L. Drake, Jr. <fred at fdrake.net>
"A storm broke loose in my mind." --Albert Einstein
Miano Njoka
> On Thu, Feb 23, 2012 at 2:54 AM, Miano Njoka <miano...@gmail.com> wrote:
>> Yes, this is true, but why strip out the meta tag from the resulting HTML?
>
> Two reasons:
>
> 1. It may be incorrect.
>
> 2. If multiple templates are used to construct a response, different
> values may be included from each template, which may be inconsistent.
>
The code as it is now does not take this into account. It parses the
meta content type tag from all the templates passed to it and the
content type header sent in the response will be that of the last
template processed.
> Since the meta element is unnecessary, it seemed better to leave it out
> of the result,
While it is not essential, it is necessary in some cases where the
finished document will be read from disk or is used by other
applications eg. Deliverance[http://packages.python.org/Deliverance/].
In fact w3c's HTML validator throws a warning that one should declare
the character encoding in the document itself if it is missing.
> and rely on other components to render the correct values
> without requiring them to insert correct values into the rendered template.
Rather than removing the meta tag, I think its less complicated to
leave it in the finished HTML and let the developer fix any
inconsistencies that may arise.
Charlie Clark
> While it is not essential, it is necessary in some cases where the
> finished document will be read from disk or is used by other
> applications eg. Deliverance[http://packages.python.org/Deliverance/].
> In fact w3c's HTML validator throws a warning that one should declare
> the character encoding in the document itself if it is missing.
This is actually what the validator says:
"""
No character encoding information was found within the document, either in
an HTML meta element or an XML declaration. It is often recommended to
declare the character encoding in the document itself, especially if there
is a chance that the document will be read from or saved to disk, CD, etc.
"""
As ZPT produces XHTML the proper place for any encoding declaration is in
the XML declaration, defaulting to UTF-8, which should throw a validation
error if incorrect. Like much of the HTML standard the meta tags were
never really thought through and, because invisible to the user, all too
often copied mindlessly from one project to another: I have customers
today with completely invalid and misleading meta tags of which they and
the rest of the world is blissfully unware. And as a result browsers - the
main consumers of the format were made fault tolerant - after all the user
often had no idea what was causing the problem or how to rectify it. I
have seen many examples of the server saying one think and the meta
something else entirely. I think nearly all browsers believe what the
server says over what's in the meta tag.
According to MAMA, which was instrumental in developing HTML 5 based on
what has actually been written, the charset was set in the
http-headersover 99 % of the time. Unfortunately, it doesn't contain any
stats on discrepancies between the http-header and the meta.
http://dev.opera.com/articles/view/mama
While there is apparently a possible security risk when not declaring the
charset I think the Pythonic principle of "there should be preferably one
obvious way to do something" should apply when within Zope trying to
decide the charset of a file and that should be well documented. I'd
suggest keeping the stripping but implementing a more rigorous approach
such as you suggest.
Charlie
--
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Kronenstr. 27a
Düsseldorf
D- 40217
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
Marius Gedminas
> Am 24.02.2012, 09:47 Uhr, schrieb Miano Njoka <miano...@gmail.com>:
>
> >While it is not essential, it is necessary in some cases where the
> >finished document will be read from disk or is used by other
> >applications eg. Deliverance[http://packages.python.org/Deliverance/].
> >In fact w3c's HTML validator throws a warning that one should declare
> >the character encoding in the document itself if it is missing.
>
> This is actually what the validator says:
>
> """
> No character encoding information was found within the document,
> either in an HTML meta element or an XML declaration. It is often
> recommended to declare the character encoding in the document
> itself, especially if there is a chance that the document will be
> read from or saved to disk, CD, etc.
> """
>
> As ZPT produces XHTML the proper place for any encoding declaration
> is in the XML declaration, defaulting to UTF-8, which should throw a
> validation error if incorrect.
A strong -1 for zope.pagetemplate adding <?xml ... ?> declarations
automatically.
> Like much of the HTML standard the
> meta tags were never really thought through and, because invisible
> to the user, all too often copied mindlessly from one project to
> another: I have customers today with completely invalid and
> misleading meta tags of which they and the rest of the world is
> blissfully unware. And as a result browsers - the main consumers of
> the format were made fault tolerant - after all the user often had
> no idea what was causing the problem or how to rectify it. I have
> seen many examples of the server saying one think and the meta
> something else entirely. I think nearly all browsers believe what
> the server says over what's in the meta tag.
The HTML spec requires that:
"To sum up, conforming user agents must observe the following
priorities when determining a document's character encoding (from
highest priority to lowest):
1. An HTTP "charset" parameter in a "Content-Type" field.
2. A META declaration with "http-equiv" set to "Content-Type" and a
value set for "charset".
3. The charset attribute set on an element that designates an
external resource."
-- http://www.w3.org/TR/html4/charset.html#h-5.2.2
(Aside: The rationale for this ordering, IIRC, is that it allows HTTP
servers to do on-the-fly charset conversion from one 8-bit charset to a
different one, without having to parse HTML and modify the charset name
in the <meta> declaration.)
> According to MAMA, which was instrumental in developing HTML 5 based
> on what has actually been written, the charset was set in the
> http-headersover 99 % of the time. Unfortunately, it doesn't contain
> any stats on discrepancies between the http-header and the meta.
>
> http://dev.opera.com/articles/view/mama
>
> While there is apparently a possible security risk when not
> declaring the charset I think the Pythonic principle of "there
> should be preferably one obvious way to do something" should apply
> when within Zope trying to decide the charset of a file and that
> should be well documented. I'd suggest keeping the stripping but
> implementing a more rigorous approach such as you suggest.
I'm not a big fan of the stripping.
Consider people using wget to mirror websites (or some equivalent way --
hitting Save As in a browser and selecting "Web Page (original)" instead
of "Web Page (complete)"). The Content-Type header is not going to be
saved on disk.
Why should zope.pagetemplate forbid programmers from duplicating the
charset information in the <meta> element, at least as long as that
information is correct (i.e. matches the content type)?
Marius Gedminas
--
http://pov.lt/ -- Zope 3/BlueBream consulting and development
Charlie Clark
> The HTML spec requires that:
> "To sum up, conforming user agents must observe the following
> priorities when determining a document's character encoding (from
> highest priority to lowest):
> 1. An HTTP "charset" parameter in a "Content-Type" field.
> 2. A META declaration with "http-equiv" set to "Content-Type" and a
> value set for "charset".
> 3. The charset attribute set on an element that designates an
> external resource."
> -- http://www.w3.org/TR/html4/charset.html#h-5.2.2
> (Aside: The rationale for this ordering, IIRC, is that it allows HTTP
> servers to do on-the-fly charset conversion from one 8-bit charset to a
> different one, without having to parse HTML and modify the charset name
> in the <meta> declaration.)
As a follow up to this it's worth noting that as from Opera 12 the
practice will be:
* BOM sniffing
* http header
* meta declaration
In that order and inline with Webkit and IE:
"""
It is better to encode your Web pages in UTF-8, and serve them as such. In
HTTP, the HTTP header has priority, then the meta name contained in HTML.
Some Web pages have specific encoding. It happens often on the Web that
the Web page encoding is different from the one specified in the file
and/or the one specified in HTTP headers. It creates issues for users who
receive unreadable characters on their screens. So the browsers have to
fix the encoding on the fly. We had bug reports about Web sites sending
BOM different from the HTTP header. We decided to make the BOM
authoritative like webkit and IE, because there are more chances for it to
be exact than the HTTP headers.
"""
http://my.opera.com/ODIN/blog/2012/03/26/whats-new-in-opera-development-snapshots-march-26-2012