[Django] #37212: djangodocs Sphinx extension incompatible with other themes

[Django]

#37212: djangodocs Sphinx extension incompatible with other themes
-------------------------------------+-------------------------------------
Reporter: Mike | Owner: Mike Edmunds
Edmunds |
Type: | Status: assigned
Cleanup/optimization |
Component: | Version: 6.0
Documentation | Keywords: sphinx, djangodocs,
Severity: Normal | djangoproject.com, sphinx-extension
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
The `docs/_ext/djangodocs.py` catchall Sphinx extension has three problems
that make it difficult or impossible to use with other Sphinx themes.
These need to be fixed before the `docs/_theme/djangodocs` Sphinx
''theme'' can be replaced with a third-party, well-maintained theme as
proposed in #37138.

== 1. Outdated workaround to remove table borders crashes other themes

The `DjangoHTMLTranslator.visit_table()` and `depart_table()` are working
around `<table border=1>` created by the docutils html4 writer. But Sphinx
has used the ''html5'' writer (which doesn't emit the obsolete border
attribute) since 2.0. The workaround depends on undocumented Sphinx
internal attrs and causes an `IndexError` when used with (e.g.,) sphinx-
pydata-theme.

It can and should be removed.

== 2. console-tabs implementation creates asymmetric DOM

The console-tabs renders structurally different html for the Unix/macOS
tab:

{{{#!text
<section class="c-content-unix" id="c-content-0-unix">
<div class="highlight-console notranslate">
<div class="highlight">...</div>
</div>
</section>
}}}

… compared to the Windows tab:

{{{#!text
<section class="c-content-win" id="c-content-0-win">
<div class="highlight">...</div>
</section>
}}}

Notice the expected wrapper `<div class="highlight-doscon notranslate">`)
is missing on the Windows tab. Although this doesn't matter for the
djangodocs and djangoproject.com themes (or they have specific
workarounds?), it affects spacing in themes like furo and pydata-sphinx-
theme. (It would also break a Pygments style that had special rules for
the `doscon` lexer, and it might result in translation tools treating the
Windows command shell syntax as English text to be translated.)

One solution is to borrow Sphinx's own machinery to render the Windows
literal block, rather than calling the Pygments highlighter directly as
now. This would also also simplify the code a bit.

== 3. console-tabs extension is tightly coupled to djangodocs theme
templates

The console-tabs extension relies on a `console-tabs.css` file that is
conditionally included in `docs/_theme/djangodocs/layout.html` based on
context provided by the extension. That doesn't work for third-party
themes with their own layout templates. (Or requires maintaining a custom
theme inheriting from the third party.)

We should extract console-tabs.css from the djangodocs ''theme'' and make
it available in the djangodocs ''extension'' for any theme. Options:
1. Move console-tabs.css to the project Sphinx static dir (`docs/_static)`
and add it to ''all pages'' in `docs/conf.py`, whether or not they contain
console tabs. (Note that docs.djangoproject.com essentially already does
this with its customized console-tabs.scss.)
2. Move console-tabs.css to `docs/_static` but inject it conditionally
only in pages that need it in the djangodocs ''extension''. (Extensions
can call `app.add_css_file()` in page-specific hooks since Sphinx 3.5.)
3. Move console-tabs.css to `docs/_ext/static` and make the djangodocs
''extension'' responsible for copying it into the static output. (Requires
Sphinx 9.1 or later for `app.add_static_dir()` support in extensions.)

Also, as part of any rework here, we should eliminate the vendored fa-
brands font in the Django docs source (it's used only for console-tabs
icons) and update the console-tabs rendering for improved accessibility
(see #35874 stage 1)

== djangoproject.com considerations

Items 1 and 2 should not affect docs.djangoproject.com.

docs.djangoproject.com will likely want to track any changes to console-
tabs.css in its custom console-tabs.scss. That file is used for ''all''
recently-rendered versions on docs.djangoproject.com, and older versions
are rendered with the older djangodocs Sphinx extension (unless this gets
backported). If the rendered DOM for the tabs is changing significantly,
we should make it possible for djangoproject.com styles to differentiate
docs built with the older tabs from the new tabs (e.g., by changing `class
="console-tabs"` to `class="dj-console-tabs"`).
--
Ticket URL: <https://code.djangoproject.com/ticket/37212>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#37212: djangodocs Sphinx extension incompatible with other themes
-------------------------------------+-------------------------------------

Reporter: Mike Edmunds | Owner: Mike
Type: | Edmunds
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 6.0
Severity: Normal | Resolution:
Keywords: sphinx, djangodocs, | Triage Stage: Accepted
djangoproject.com, sphinx- |
extension |
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Natalia Bidart):

* stage: Unreviewed => Accepted

Comment:

Thank you Mike, this all makes sense and matches what I saw when working
on a PoC for #37138. My thoughts on some the questions:

1. Agreed
2. For this I was using a custom CSS rule in console-tabs.css to cope with
the different spacing, I guess that's one solution but not robust and does
not scale well. I'm not familiar with "Sphinx's own machinery to render
the Windows" but sounds like a sensible approach.
3. I think option 1 is the simplest and is acceptable.
--
Ticket URL: <https://code.djangoproject.com/ticket/37212#comment:1>

[Django]

#37212: djangodocs Sphinx extension incompatible with other themes
-------------------------------------+-------------------------------------
Reporter: Mike Edmunds | Owner: Mike
Type: | Edmunds
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 6.0
Severity: Normal | Resolution:
Keywords: sphinx, djangodocs, | Triage Stage: Accepted
djangoproject.com, sphinx- |
extension |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Description changed by Mike Edmunds:

Old description:

> `class="console-tabs"` to `class="dj-console-tabs"`).

New description:

The `docs/_ext/djangodocs.py` catchall Sphinx extension has three problems
that make it difficult or impossible to use with other Sphinx themes.
These need to be fixed before the `docs/_theme/djangodocs` Sphinx
''theme'' can be replaced with a third-party, well-maintained theme as
proposed in #37138.

== 1. Outdated workaround to remove table borders crashes other themes

The `DjangoHTMLTranslator.visit_table()` and `depart_table()` overrides

are working around `<table border=1>` created by the docutils html4
writer. But Sphinx has used the ''html5'' writer (which doesn't emit the
obsolete border attribute) since 2.0. The workaround depends on
undocumented Sphinx internal attrs and causes an `IndexError` when used
with (e.g.,) sphinx-pydata-theme.

It can and should be removed.

Edit: similar problem with
`DjangoHTMLTranslator.visit_desc_parameterlist()`, which was meant to
prevent `<big>(</big>` in parameter lists. Sphinx hasn't done that in
ages, and the override regularly breaks with new Sphinx versions (and
prevents use of Sphinx's `multi_line_parameter_list` option).

--
Ticket URL: <https://code.djangoproject.com/ticket/37212#comment:2>

[Django]

#37212: djangodocs Sphinx extension incompatible with other themes
-------------------------------------+-------------------------------------
Reporter: Mike Edmunds | Owner: Mike
Type: | Edmunds
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 6.0
Severity: Normal | Resolution:
Keywords: sphinx, djangodocs, | Triage Stage: Accepted
djangoproject.com, sphinx- |
extension |

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mike Edmunds):

* has_patch: 0 => 1

Comment:

https://github.com/django/django/pull/21613
--
Ticket URL: <https://code.djangoproject.com/ticket/37212#comment:3>

[Django]

#37212: djangodocs Sphinx extension incompatible with other themes
-------------------------------------+-------------------------------------
Reporter: Mike Edmunds | Owner: Mike
Type: | Edmunds
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 6.0
Severity: Normal | Resolution:
Keywords: sphinx, djangodocs, | Triage Stage: Accepted
djangoproject.com, sphinx- |
extension |
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mike Edmunds):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/37212#comment:4>

[Django]

#37212: djangodocs Sphinx extension incompatible with other themes
-------------------------------------+-------------------------------------
Reporter: Mike Edmunds | Owner: Mike
Type: | Edmunds
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 6.0
Severity: Normal | Resolution:
Keywords: sphinx, djangodocs, | Triage Stage: Accepted
djangoproject.com, sphinx- |
extension |
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mike Edmunds):

* needs_better_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/37212#comment:5>

[Django]

#37212: djangodocs Sphinx extension incompatible with other themes
-------------------------------------+-------------------------------------
Reporter: Mike Edmunds | Owner: Mike
Type: | Edmunds
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 6.0
Severity: Normal | Resolution:
Keywords: sphinx, djangodocs, | Triage Stage: Accepted
djangoproject.com, sphinx- |
extension |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mike Edmunds):

Replying to [comment:4 Mike Edmunds]:
> a. Show "Django N.M" rather than the generic "version N.M" in these
messages.
> b. Show "Added in Django Development version" rather than "Added in
Django N.M" for the current in-development version (as set by conf.py
`django_next_version`).

It's worth asking how important these behaviors are. I suspect if someone
proposed adding this functionality today, it would be rejected as too
complicated for too little value.

An alternative would be to remove these customizations and use Sphinx's
built-in behavior:

* "Added in version 6.0" (rather than "Added in Django 6.0")
* "Changed in version 6.2" (rather than "Changed in Django development
version" during 6.2 development)
* "Deprecated since version 6.2" (no change)

Of the two, (a) is considerably more complicated/fragile than (b) to
maintain.
--
Ticket URL: <https://code.djangoproject.com/ticket/37212#comment:6>