HTML5 and XHTML5 documents

[Nils Fredrik Gjerull]

I write this because the latest release version of Django (2.1) broke my
site.

On my site I serve my html documents with the unusual, but perfectly
valid, mimetype application/xhtml+xml. That is I tell the browser to
parse my html documents as XML. I do this because I prefer the stricter
parsing and error messages for invalid documents.

It is perfectly valid to serve HTML5 as application/xhtml+xml, this is
called XHTML5. Some might thinks that HTML5 have replaced XHTML so we
should not care any more. But HTML5 has not replaced XHTML (as an XML
version of HTML), it has replaced the XHTML-Transitional and Strict
standards. You can think of XHTML5 as HTML5 serialized as XML.

As Django is a framework I think it should output HTML that can be
served as both text/html and application/xhtml+xml, and leave the
decision of what to use to the developer.

As valid XHTML5 is also valid HTML5, but not vice versa, all HTML
generated by the framework need to be written as XHTML5.

I reopened ticket #29038, but there is more that need to be fixed. I am
happy to provide a pull-request for this.

See:
- https://www.w3.org/TR/html/introduction.html#html-vs-xhtml
- https://www.w3.org/TR/html/the-xhtml-syntax.html

Regards

--
Nils Fredrik Gjerull
-----------------------------
"Ministry of Eternal Affairs"
Computer Department
( Not an official title :) )

[Adam Johnson]

You didn't pose an exact question in your message, but I guess it's "Can Django officially support XHTML5?"

As it's a stricter subset of HTML5, it seems a valid idea. But at the same time, we need to be able to support it going forwards, with test coverage and knowledge that this is what we aim for on every PR that adds HTML. Also I doubt you'll find many third party packages support XHTML5 too.

You say there's more to be fixed that just reversing #29038. What else is there? Would you be willing to do all the work?

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/eb10d68f-706f-4bc3-4ee3-c440723aa0b4%40gjerull.net.
For more options, visit https://groups.google.com/d/optout.

--

Adam

[Nils Fredrik Gjerull]

Den 12. aug. 2018 12:20, skrev Adam Johnson:

You didn't pose an exact question in your message, but I guess it's "Can Django officially support XHTML5?"

Django has supported XHTML for years, so it came as a surprise to me that Django 2.1 broke it. So it is more like "Can Django continue to support XHTML, even for HTML5?". But the principle is the same.

As it's a stricter subset of HTML5, it seems a valid idea. But at the same time, we need to be able to support it going forwards, with test coverage and knowledge that this is what we aim for on every PR that adds HTML. Also I doubt you'll find many third party packages support XHTML5 too.

I use a number of third party libraries, for my site that uses XHTML5. I had to patch a few of them, but only small changes.

You say there's more to be fixed that just reversing #29038. What else is there? Would you be willing to do all the work?

I am willing to do the work. It is not much that need to be done. Just some query-replace to change the generated HTML into valid XML.

[Jon Dufresne]

Django started adopting HTML5 features since before 2.1. I found changes as early as 1.11. From the release notes:

https://docs.djangoproject.com/en/dev/releases/1.11/#miscellaneous

> The checked and selected attribute rendered by form widgets now uses HTML5 boolean syntax rather than XHTML’s checked='checked' and selected='selected'.

FWIW, I prefer the concise syntax of HTML5 over XHTML and seems to be the de facto preferred style across much of the web. Style guides such as Google's recommend it. From https://google.github.io/styleguide/htmlcssguide.html#HTML:

> Document Type
>
> Use HTML5.
>
> HTML5 (HTML syntax) is preferred for all HTML documents: <!DOCTYPE html>.
>
> (It’s recommended to use HTML, as text/html. Do not use XHTML. XHTML, as application/xhtml+xml, lacks both browser and infrastructure support and offers less room for optimization than HTML.)
>
> Although fine with HTML, do not close void elements, i.e. write <br>, not <br />.

Cheers,

Jon

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d0645f61-836c-c908-98a9-b99dec62d5f6%40gjerull.net.

[Nils Fredrik Gjerull]

Den 12. aug. 2018 20:06, skrev Jon Dufresne:
> Django started adopting HTML5 features since before 2.1. I found
> changes as early as 1.11. From the release notes:
>
> https://docs.djangoproject.com/en/dev/releases/1.11/#miscellaneous

Perhaps, but I have run my project on Django 2.0 without being affected
by it.

>
> > The checked and selected attribute rendered by form widgets now uses
> HTML5 boolean syntax rather than XHTML’s checked='checked' and
> selected='selected'.
>
> FWIW, I prefer the concise syntax of HTML5 over XHTML and seems to be
> the de facto preferred style across much of the web. Style guides such
> as Google's recommend it. From
> https://google.github.io/styleguide/htmlcssguide.html#HTML:

HTML5 is not a replacement for XHTML as a XML serialization. In XML
empty attributes are not allowed, so the value has to be something (even
if just en empty string). In place of using true/false the standard says
that is must be either the same as the attribute name or an empty string
(for XHTML5). This is what we are stuck with for XML serialization of HTML5.

I think a framework should rate compatibility over conciseness. And
adding a value to a boolean attribute I think is a very minor hassle. In
our own application code we are free to choose.

>
> > Document Type
> >
> > Use HTML5.
> >
> > HTML5 (HTML syntax) is preferred for all HTML documents: <!DOCTYPE
> html>.
> >
> > (It’s recommended to use HTML, as text/html. Do not use XHTML.
> XHTML, as application/xhtml+xml, lacks both browser and infrastructure
> support and offers less room for optimization than HTML.)
> >
> > Although fine with HTML, do not close void elements, i.e. write
> <br>, not <br />.

I will say that this time is the best time ever to serve pages with
application/xhtml+xml, because all browsers supports it. IE was the road
blocker for year, but since IE9 it supports it as well. The points about
infrastructure and optimization are so vague I can hardly guess what the
writer thought about.

Se a list of the default value of the Accept header for various browsers
from Mozilla:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation/List_of_default_Accept_values

All except IE8 supports application/xhtml+xml.

[Tim Graham]

Another discussion about HTML vs XHTML is https://groups.google.com/d/topic/django-developers/EdwwxxqcKVU/discussion.

I think if you want to use XHTML, Django should make is possible, but as it seems the majority of projects use HTML5, I would stick to that as the default.

Have you tried creating custom widget templates for XHTML?
https://docs.djangoproject.com/en/stable/ref/forms/renderers/

[Nils Fredrik Gjerull]

Den 13. aug. 2018 14:52, skrev Tim Graham:
> Another discussion about HTML vs XHTML
> is https://groups.google.com/d/topic/django-developers/EdwwxxqcKVU/discussion.
>
> I think if you want to use XHTML, Django should make is possible, but
> as it seems the majority of projects use HTML5, I would stick to that
> as the default.
>
> Have you tried creating custom widget templates for XHTML?
> https://docs.djangoproject.com/en/stable/ref/forms/renderers/

By making simple changes to the HTML that Django, by default, uses we
can make it work for both XHTML5 and HTML5. When there is a choice
between making in work for one case and making it work for both cases,
why make it work for only one?

I have created a pull-request for this, and an issue (#29681), which was
prematurely closed as wontfix.

[Carlton Gibson]

Hi Nils. 

> ... and an issue (#29681), which was 

prematurely closed as wontfix. 

As per the when you already re-opened #29038 on this issue, there now needs to be a consensus here before 

we can (or will) consider a new ticket for this. That's not "premature" — it's just how the project management of 

Django works. The mailing list allows a wider discussion on issues that have previously been resolved. 

I ask you to respect that. 

The PR: https://github.com/django/django/pull/10308

Only half-joking, the diff here makes me want to weep. For me, 150 files and 1803 line changes is just too much to 

enforce something that is of minority appeal. 

On XHTML5 generally, I have no problem with properly closing tags, and I guess `/>` if you must but I look at `checked="checked"` and my personal response is that I just don't want that. 

I understand the benefits of XML but I think trying to enforce it in a web framework in 2018 and beyond is skating to where the puck was, so to speak. Developers expect HTML5 and it we don't go with that as a default every PR that comes in will need "correcting" for the XML syntax, and we'll end up with a 10:1 increase in new issues asking why we're not taking advantage the new, more concise, syntax. 

You want to serve the pages you generate with XHTML. Fine. (Beyond custom widget templates what do you need?) But (from the PR) why do we need to serve (e.g.) the Admin so? Or have the examples in the docs (and code comments) be XHTML compliant? Or the template we use to test the email sending functionality? (I appreciate you probably scripted these changes.)

Does it really matter if framework provided pages use HTML5? Why? (If it does matter can you not warp a middleware around HTML Tidy, or similar, to do the conversion for you?)

If there are barriers to you creating XHTML pages, we can look at those, but I'd be -1 on bringing it back framework wide. 

Kind Regards,

Carlton

[Nils Fredrik Gjerull]

Den 17. aug. 2018 10:04, skrev Carlton Gibson:
> Only half-joking, the diff here makes me want to weep. For me, 150
> files and 1803 line changes is just too much to 
> enforce something that is of minority appeal.

The changes are simple. Use '/>' at end of self-closing tags and give
all attributes values. And most of it is test-code. It is only the
'attrs.html'-files that are the real change (when it comes to giving an
attribute a value).

> On XHTML5 generally, I have no problem with properly closing tags, and
> I guess `/>` if you must but I look at `checked="checked"` and my
> personal response is that I just don't want that.
> I understand the benefits of XML but I think trying to enforce it in a
> web framework in 2018 and beyond is skating to where the puck was, so
> to speak. Developers expect HTML5 and it we don't go with that as a
> default every PR that comes in will need "correcting" for the XML
> syntax, and we'll end up with a 10:1 increase in new issues asking why
> we're not taking advantage the new, more concise, syntax.

Unfortunately web-development has been and are plagued by fashion waves.
First everything should be XML, now everything should be JSON. We first
had web services and XML-RPC, then REST and now perhaps GraphQL. I agree
that `checked="checked"` is not cool, but that's beside the point. It is
the standard and frameworks and library need to be extra careful to
support them. It is possible to write `checked=""`, if that helps. I
think, however, that it is more clear if we give it a value. By the way,
the syntax is not new it is the old syntax from HTML4. I have spent
quite some time cleaning up ill-formed HTML4.

>
> You want to serve the pages you generate with XHTML. Fine. (Beyond
> custom widget templates what do you need?) But (from the PR) why do we
> need to serve (e.g.) the Admin so? Or have the examples in the docs
> (and code comments) be XHTML compliant? Or the template we use to test
> the email sending functionality? (I appreciate you probably scripted
> these changes.)

I avoided updating the docs myself. The revert of a previous commit has
changed it. The comments can be changed to the HTML (SGML) style if it
helps. I did not add the XML style of boolean attributes to the docs,
but I can remove the '/>' if it helps.

>
> Does it really matter if framework provided pages use HTML5? Why? (If
> it does matter can you not warp a middleware around HTML Tidy, or
> similar, to do the conversion for you?)
>
> If there are barriers to you creating XHTML pages, we can look at
> those, but I'd be -1 on bringing it back framework wide. 

I still would like a technical answer to why not support both standards?
And again XHTML5 is HTML5 with valid XML syntax. So valid XHTML5 is
valid HTML5, so there is no problem for a framework to provide HTML5 it
should just be done in the most compatible manner.

[Curtis Maloney]

On 08/17/2018 06:50 PM, Nils Fredrik Gjerull wrote:
> think, however, that it is more clear if we give it a value. By the way,
> the syntax is not new it is the old syntax from HTML4. I have spent
> quite some time cleaning up ill-formed HTML4.

One of the more significant differences between the advent of XHTML and
now, is that HTML5 introduced standard rules for how to deal with
"invalid" markup, meaning its handling in browsers became consistent.

XHTML was a great move to allow a shift to browsers only accepting valid
markup, but it never happened -- in part because IE just wouldn't play
along.

So a softer solution was found - moving to HTML5 with defined failure
modes and handling.

Currently, ISTM the only thing standing in the way of you using Django
for generating valid XHTML is the form widgets, in which case I suspect
writing your own widget templates would be far less work for everyone
involved.

You could also provide them, quite trivially, as a 3rd party app for
other people facing the same issues as you.

--
Curtis

[Nils Fredrik Gjerull]

Den 17. aug. 2018 11:01, skrev Curtis Maloney:
> One of the more significant differences between the advent of XHTML
> and now, is that HTML5 introduced standard rules for how to deal with
> "invalid" markup, meaning its handling in browsers became consistent.
>
> XHTML was a great move to allow a shift to browsers only accepting
> valid markup, but it never happened -- in part because IE just
> wouldn't play along.

IE now supports XHTML, so now it seams to me the biggest stumbling block
is fashion. I cannot shake of the feeling that the arguments are "I
don't like giving attributes a value and ending self-closing tags with
'/>'" or perhaps "some engineer at Google says so".

> So a softer solution was found - moving to HTML5 with defined failure
> modes and handling.
>
> Currently, ISTM the only thing standing in the way of you using Django
> for generating valid XHTML is the form widgets, in which case I
> suspect writing your own widget templates would be far less work for
> everyone involved.
>
> You could also provide them, quite trivially, as a 3rd party app for
> other people facing the same issues as you.

I suppose I could, but why? When giving attributes values and ending
self-closing tags with '/>' makes it work for both cases? I really like
Django with it excellent documentation and promotion of good practices.
I newcomer can learn a lot from how Django is made. Django has provided
valid XHTML markup for years, as far as I can tell.

I can update the pull-request and concentrate the changes on the actual
HTML generated by Django, and leave out the documentation and comments
if it helps to make it more palatable. However, I think it would be a
good thing to learn that there is a thing called XHTML5. Perhaps I can
update the documentation to make it clear, but give the examples using
the SGML syntax of HTML5?

[James Bennett]

On Fri, Aug 17, 2018 at 1:50 AM, Nils Fredrik Gjerull <ni...@gjerull.net> wrote:

I still would like a technical answer to why not support both standards?
And again XHTML5 is HTML5 with valid XML syntax. So valid XHTML5 is
valid HTML5, so there is no problem for a framework to provide HTML5 it
should just be done in the most compatible manner.

You probably won't get a "technical answer", because what you're proposing is not a technical change.

So, look. I was someone who, in the 2000s, passed Evan Goer's "XHTML 100" test, even if he didn't test my site as part of it. I bought into it. But I gradually learned it's more of a headache than it's worth, and should be discouraged as a default. People who do still have use cases requiring XML toolchains are free to implement what's necessary to support that (for Django, that's mostly templating changes, and I believe we make it easy enough to supply your own overrides that you could do it as a 100% third-party effort for anyone else who shares your desire for XHTML).

But requiring Django to maintain XML-serialization-of-HTML5 compatibility is a non-starter. It commits everyone else on earth to keeping that supported, to ensuring we never do anything that's not XHTML-compatible, etc., in order to serve a very small subset of developers, and I don't think you realize how much specialized domain knowledge is actually necessary to do that (it's more complex than "/>" and specifying attribute values).

And I suppose if you really want technical arguments:

JavaScript is not portable between XML and HTML contexts, full stop. This isn't just about the CDATA dance you have to do with inline script elements; it's about the fact that the HTML and XML DOMs are different. And not just with respect to namespaced versus non-namespaced DOM methods: XML's rules literally do produce a different parse tree from the same DOM when compared to HTML's.

XHTML5 itself is neither well-specified nor robust; the new elements of HTML5 are more or less dumped by fiat into the old 1999/XHTML namespace, but who's to say parsers will actually be aware of that? Especially validating parsers? Can you point to a standardized XML DTD, XSD, RELAX NG or, well, anything that's broadly accepted as defining XHTML5? W3C doesn't publish one that I'm aware of, and the Editor's Draft for Polyglot does not provide guidance.

And speaking of validating parsers: HTML5 has a large set of named entities. Validating XML parsers are required, as always, to accept only the base five of XML, and can fatal-error on anything else. For non-validating parsers, or for those which choose to accept more than the base five named entities, XHTML and HTML historically did not agree on the set (XHTML defined 253 named entities, HTML historically defined at most 252). More fun times there. XML-ENTITY-NAMES tried to alleviate this by defining and exposing entity sets documents could refer to, but see again: validating parsers don't have to accept them. Are we forbidden ever to use an HTML5 entity, or anything outside the base five XML entities, in Django, in order to make sure serving Django as XHTML works? I hope not.

While we're on the topic of characters: HTML5's character encoding rules are not compatible with XML's. Compliant HTML5 processors *must* use the 8.2.2.2 encoding sniffing algorithm in their first pass, and this can produce a different determined encoding than XML's rules (and let's not get into the quagmire of default encoding varying with Content-Type headers).

And, well, I could go on for a while here because my rants on this topic are pretty well-developed at this point, but I'll just leave it alone here. I'm strongly against any attempt to have Django try to produce XML-serializable polyglot HTML5.

[Nils Fredrik Gjerull]

Den 17. aug. 2018 12:07, skrev James Bennett:
>
> XHTML5 itself is neither well-specified nor robust; the new elements
> of HTML5 are more or less dumped by fiat into the old 1999/XHTML
> namespace, but who's to say parsers will actually be aware of that?
> Especially validating parsers? Can you point to a standardized XML
> DTD, XSD, RELAX NG or, well, anything that's broadly accepted as
> defining XHTML5? W3C doesn't publish one that I'm aware of, and the
> Editor's Draft for Polyglot does not provide guidance.
>
> And speaking of validating parsers: HTML5 has a large set of named
> entities. Validating XML parsers are required, as always, to accept
> only the base five of XML, and can fatal-error on anything else. For
> non-validating parsers, or for those which choose to accept more than
> the base five named entities, XHTML and HTML historically did not
> agree on the set (XHTML defined 253 named entities, HTML historically
> defined at most 252). More fun times there. XML-ENTITY-NAMES tried to
> alleviate this by defining and exposing entity sets documents could
> refer to, but see again: validating parsers don't have to accept them.
> Are we forbidden ever to use an HTML5 entity, or anything outside the
> base five XML entities, in Django, in order to make sure serving
> Django as XHTML works? I hope not.

I am talking about being able to serve pages as application/xhtml+xml,
this is defined by browser support as is the SGML version of HTML5. I
hardly think XML version of HML5 is more ill-defined than the SGML
version. I am not talking about supporting validators, as the browsers
are the validators. No need to use a validator if the browser makes it
clear when it is not. No need to limit what entities to use as long at
it is supported by the browsers.

As for Javascript and the difference between a XML-dom and and HTML-dom.
I have not run into this. I use Javascript to manipulate the DOM, but
not heavily. Do the browsers generate different DOM when using
application/xhtml+xml to server the pages? That is an argument to serve
Django Admin as 'text/html', and let other apps choose for themselves.

In my case it is only the use '/>' and giving attributes a value that
stands in the way.

[James Bennett]

On Fri, Aug 17, 2018 at 3:41 AM, Nils Fredrik Gjerull <ni...@gjerull.net> wrote:

I am talking about being able to serve pages as application/xhtml+xml,
this is defined by browser support as is the SGML version of HTML5. I
hardly think XML version of HML5 is more ill-defined than the SGML
version. I am not talking about supporting validators, as the browsers
are the validators. No need to use a validator if the browser makes it
clear when it is not. No need to limit what entities to use as long at
it is supported by the browsers.

If you're basing your understanding on browser support, you're not doing XML/XHTML. You're doing "a thing that looks like XHTML and works in my browser".

I'm even *more* against having Django try to do that, since it's a moving target.

[Collin Anderson]

> serve Django Admin as 'text/html', and let other apps choose for themselves.

Yes, DEFAULT_CONTENT_TYPE is deprecated, so the admin (soon) should always return text/html, and other apps can choose for themselves. (see ticket #23908)

I think django should be encouraging html instead of xhtml, but it shouldn't be too hard to use django with xhtml.

Forms are probably the next biggest issue after the admin, right? I agree a 3rd-party package would be best for that. Templates could probably handle most if not all of it. I'd be fine changing that one line in django/forms/forms.py as_table from <br> to <br />. maybe render_css too. (or maybe those need to get moved to templates?) The 3rd party package could provide xhtml versions of template tags and filters where needed.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAL13Cg9UEtSBEPnc0_OJWy4HHALSA-XOLhgKBnnF5Fq7%3Dr811Q%40mail.gmail.com.

[Nils Fredrik Gjerull]

Den 17. aug. 2018 22:07, skrev James Bennett:
>
> If you're basing your understanding on browser support, you're not
> doing XML/XHTML. You're doing "a thing that looks like XHTML and works
> in my browser".

Webstandards has and probably always will be defined by browser support.
That's how the web works. I am sure you know it. XHTML5 is a XML
materialization of HTML5, and there is also the SGML-inspired version.
It is possible have valid xml markup without a schema, and it is
possible to make a schema more or less tolerant. The reason there is no
official schema for the XML version is the same as the reason there is
no official SGML DTD. Browser support is a moving target. Any official
schema also need to take custom elements into consideration. So a schema
can be strict about all the wrapping tags and header tags, but it need
to be more "free-form" when it comes to what kind of tags can be inside
the body.

It not not only work in "my browser" it work for every major browser out
there. Which is the "de facto" definition of when browser technology can
be used.

[Nils Fredrik Gjerull]

Den 20. aug. 2018 11:32, skrev Nils Fredrik Gjerull:
> XML
> materialization of HTML5, and there is also the SGML-inspired version.

I intended to write XML serialization :)

[Nils Fredrik Gjerull]

How can we move forward on supporting XML serialization of HTML5 (XHTML5)?

When I created the pull-request I did not think it would be very
controversial. After all, the XML style of writing HTML has been used
for years. I guess many are not aware that HTML5 comes in two flavors
and it creates confusion.

My proposal is.
- Use the SGML serialization for the admin and the docs.
- Add some information about the two serializations of HTML5 to the docs.
- Make the Django core output XML compatible HTML as both flavors of
HTML5 supports it. This is mainly the forms library, I think.
- Add some tests to check that both serializations are supported.

[Carlton Gibson]

Hi Nils. 

Tim's original advice still stands: 

> Have you tried creating custom widget templates for XHTML?
> https://docs.djangoproject.com/en/stable/ref/forms/renderers/

Once you have widget templates in place, if there are particular issues you are facing we can consider them 

but the summary here seems to be that there's little desire to do anything further. 

Kind Regards,

Carlton

[Aymeric Augustin]

Hello Nils,

On 23 Aug 2018, at 19:33, Nils Fredrik Gjerull <ni...@gjerull.net> wrote:

How can we move forward on supporting XML serialization of HTML5 (XHTML5)?

When I created the pull-request I did not think it would be very
controversial. After all, the XML style of writing HTML has been used
for years. I guess many are not aware that HTML5 comes in two flavors
and it creates confusion.

You may have missed that Django made the opposite move over the last couple years. You don't appear to be aware of the corresponding discussions.

You should review these discussions, compile a list of the reasons why these changes were made, and brining counter-arguments if you want to reverse course.

Best regards,

-- 

Aymeric.