Results of the SymPy Documentation Theme Survey

[Jeremy Monat]

Hello SymPy community,

SymPy ran a user survey about its documentation theme from February 5-19, 2022. The primary purpose of the survey was to guide the selection of a Sphinx theme for the SymPy Documentation at https://docs.sympy.org. We thank everyone who took and shared the survey.

Even though the survey is no longer open, we still welcome feedback on SymPy's documentation. Feel free to reach out to us on the mailing list, or in the Github issue to change the Sphinx theme.

I have written up an analysis of the results at
https://www.sympy.org/sympy-docs-survey/2022-theme-survey.html (thanks to Aaron Meurer for some analysis code, and posting the analysis there). The source code for the
Jupyter notebook can be found at https://github.com/sympy/sympy-docs-survey. I
have included a summary of this analysis here.

A total of 22 people responded. The survey was done on Google Surveys and was shared on the SymPy public mailing list, the @SymPy Twitter account, and a SymPy discussion on GitHub. The survey consisted of 14 questions, all of which were optional. The results of these responses are summarized here. We would like to thank everyone who took and shared the survey.

At a high level, there are three main takeaways from the results.

1. The themes can be divided into three ratings categories, where the rating scale was 1 (Not very useful) to 4 (Very useful):

   1. Highest: Furo at 2.95.
   2. Middle: PyData and Book, nearly tied at 2.85 and 2.86, respectively.
   3. Lowest: Read the Docs (RTD) at 2.47.

2. Most comments about themes, both likes and dislikes, were about formatting, look and feel, and navigation.

3. We should proceed with the Furo theme, customizing it to address respondents' dislikes about its formatting. We can keep the PyData and Book themes in mind as backup options.

Again, I would like to thank everyone who took the time to fill out this survey. It really helps us to have your feedback.

Jeremy Monat

[Aaron Meurer]

Thank you for all the work you've done here Jeremy.

It sounds like the Furo theme is the winner. I also like it the best,
because I think it does the best job with the sidebars (there are some
styling aspects of it that I don't like, but those are easy to change
in the CSS).

I've started a pull request proposing the Furo theme with several CSS
modifications to make it styled better (green colors, and so on) here
https://github.com/sympy/sympy/pull/23159. There is a demo site built
at https://www.asmeurer.com/sympy-furo-demo/

Any comments on my styling fixes are welcome. I'm not really a CSS
expert so please let me know if you see something that is difficult to
read or should otherwise be improved.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/ae6935a1-d638-4265-a094-bd59f1dfc643n%40googlegroups.com.

[Jason Moore]

Thanks for doing this! I read through all the comments.

Couple of points:

- With 22 respondents and large standard deviation, the numbers don't really mean anything. Basically all themes are rated the same.

- The written comments are most useful and I get the impression that almost any of the themes could work, but each requires some tweaking to fit for SymPy.

I would recommend choosing based on which theme has the most configuration options and energy behind it because we want to easily tweak things and we automatically benefit from upstream improvements. If we do pydata, we join with our counterparts Numpy, scipy, pandas, etc. and it keeps us connected nicely to that community and when people jump around the scipy ecosystem docs they get the same (or similar) experience. RTD theme, by far, is the most used because it is the default theme on their service and there is a company that spends a lot of dev time on it. RTD is quite valuable and gives a uniform experience across a large set of python projects. Furo and book are likely used the least and have the smallest dev communities. Furo, as I understand, is essentially a one man show. It looks nice now, but may not be a good long term solution.

Jermey and Aaron concluded that Furo was the best choice, but I hope these other aspects are considered too. We're a big project and even if Furo currently has the best looking design of the four, there are other non-design factors that are also quite important and, IMO, outweigh the 0.1 point rating differences in the comparison of the designs.

Jason

moorepants.info
+01 530-601-9791

--

[Jason Moore]

I also just realized that the book theme is the pydata theme with some tweaks.

Jason

moorepants.info
+01 530-601-9791

[Aaron Meurer]

On Sat, Feb 26, 2022 at 1:11 AM Jason Moore <moore...@gmail.com> wrote:
>
> Thanks for doing this! I read through all the comments.
>
> Couple of points:
>
> - With 22 respondents and large standard deviation, the numbers don't really mean anything. Basically all themes are rated the same.
> - The written comments are most useful and I get the impression that almost any of the themes could work, but each requires some tweaking to fit for SymPy.
>
> I would recommend choosing based on which theme has the most configuration options and energy behind it because we want to easily tweak things and we automatically benefit from upstream improvements. If we do pydata, we join with our counterparts Numpy, scipy, pandas, etc. and it keeps us connected nicely to that community and when people jump around the scipy ecosystem docs they get the same (or similar) experience. RTD theme, by far, is the most used because it is the default theme on their service and there is a company that spends a lot of dev time on it. RTD is quite valuable and gives a uniform experience across a large set of python projects. Furo and book are likely used the least and have the smallest dev communities. Furo, as I understand, is essentially a one man show. It looks nice now, but may not be a good long term solution.

I agree that the bus factor is a downside to Furo. However, I'm not
too worried about it given that it's not all that hard to change the
Sphinx theme. Any customizations would have to be redone, but it took
me about a day of work to restyle Furo (and honestly someone more
familiar with CSS could have done it much faster). And there are ways
that Furo could have made restyling easier than it was, so potentially
restyling a hypothetical future theme could be done even easier.

The styling (colors, font choices, very basic CSS changes) are easy to
make. What's hard to do is to change how the theme works at a
fundamental level. That's why one of the primary things we looked at
was the behavior of the sidebars in the different themes. This is not
something we can "fix" ourselves with some CSS. We are really just
stuck with however the theme handles things. Here Furo had the best
behavior: for instance, the right sidebar always being expanded, which
was noted in the survey as a plus. I would like to avoid things like
custom Javascript on the docs site, as it becomes unmaintainable given
that most SymPy developers are not frontend developers.

In general, the Furo theme seems to have had a finer attention to
detail than the other themes. We have a lot of docs and they exercise
a lot of corner cases that the other themes don't seem to have been
designed around, but Furo handles them correctly. As an example, look
at how the different themes' sidebars handle the very long section
names on the active deprecations page. Book and Pydata add a
horizontal scrollbar to the sidebar:

https://bertiewooster.github.io/sympy-doc/book/explanation/active-deprecations.html#sympy-stats-discretemarkovchain-absorbing-probabilites
https://bertiewooster.github.io/sympy-doc/pydata/explanation/active-deprecations.html#sympy-stats-discretemarkovchain-absorbing-probabilites

Readthedocs just truncates the long names:

https://bertiewooster.github.io/sympy-doc/rtd/explanation/active-deprecations.html#sympy-stats-discretemarkovchain-absorbing-probabilites

Furo word wraps the text:

https://bertiewooster.github.io/sympy-doc/furo/explanation/active-deprecations.html#sympy-stats-discretemarkovchain-absorbing-probabilites

The Furo behavior is clearly the best, and it suggests to me that the
other themes were not ever tested on this sort of thing.

>
> Jermey and Aaron concluded that Furo was the best choice, but I hope these other aspects are considered too. We're a big project and even if Furo currently has the best looking design of the four, there are other non-design factors that are also quite important and, IMO, outweigh the 0.1 point rating differences in the comparison of the designs.

The decision to use Furo isn't completely final yet. So if you want to
make the case for one of the other themes, you still can.

Aaron Meurer

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1Aidk%3DK_hNciDukWPUM4V6hUNxZXDMkQm-%3DMnR5WcnNspA%40mail.gmail.com.

[Jeremy Monat]

While PyData or Book would be fine choices, I believe Furo is currently the best theme for SymPy, and the best choice overall, for the reasons above and below.

For Furo, while there is one primary contributor, there are at least half a dozen other contributors. Furo is also the customized theme suggested and shown in the Sphinx tutorial. So the risk of Furo development ceasing seems lower than we've believed. There is also no guarantee that the other scipy packages will continue to use PyData--they might change themes at some point.

Furo had 13 releases in the last six months, PyData had 4, and Book had 9. So Furo is likely more responsive to feedback. The author joins our discussions on their own initiative to answer questions, which hasn't happened for the other themes we've discussed, and even suggests improvements to our implementation.

Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2BUOV6t%3DgYFqsSxpOs3XuhrtskhFJ0WC_SBMHm2-tvobQ%40mail.gmail.com.

[Jason Moore]

Furo looks good. If you think the bus factor is not a big deal, that's fine then. It's not as important as an actual dependency of sympy.

> The decision to use Furo isn't completely final yet. So if you want to make the case for one of the other themes, you still can.

My vote in the survey was RTD. I explained it in the survey my reasoning. But that's all I have to offer for the case.

Jason

moorepants.info
+01 530-601-9791

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2BUOV6t%3DgYFqsSxpOs3XuhrtskhFJ0WC_SBMHm2-tvobQ%40mail.gmail.com.

[Chris Smith]

For anyone else not familiar yet with the "bus factor", I learned from wikipedia that "The bus factor is a measurement of the risk resulting from information and capabilities not being shared among team members, derived from the phrase "in case they get hit by a bus."

/c

[Aaron Meurer]

An update on this: the Furo theme pull request is now ready for a
final review. The demo site is at
https://www.asmeurer.com/sympy-furo-demo/dev/index.html. I've done
several modifications to the base Furo theme, mostly changing colors
and a few small font tweaks, so please let me know if you see anything
that should be improved style-wise. If you can, please also test the
dark mode (click the sun icon at the top), and on mobile, and try to
look at different types of documentation pages. If something looks
off, there's a good chance I messed up the CSS for it somehow or just
didn't notice it, so please let me know.

The pull request is https://github.com/sympy/sympy/pull/23159/. Also
if any frontend experts can critique my terrible CSS/Javascript
skills, that would be helpful.

Note that this does remove the SymPy Live extension from the
documentation, as it's not compatible with Furo. If we can get a
similar extension implemented that uses pyiodide, preferably one that
is maintained by the broader community, that would be great.

Aaron Meurer

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/94987327-7337-4386-bd6f-112dd5713d97n%40googlegroups.com.

[Jason Moore]

Aaron,

I browsed around the demo site. It is looking quite nice! Great job.

Jason

moorepants.info
+01 530-601-9791

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6Kctcx7wCuDyrHuwnpR2rD6eSe95cd_0BgLVen-TqEJPg%40mail.gmail.com.

[Nicolas Guarin]

I agree that the demo site is looking great. Although, I think that sharing a theme with other projects might be something desirable.

[Jason Moore]

Can you make a sphinx theme be a child of a parent theme, such that we can easily update furo without having to fork and customize furo? If all we are doing is css overrides, I suspect that is possible.

Jason

moorepants.info
+01 530-601-9791

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/524b24d0-416f-477f-a3a8-22eab1141ee4n%40googlegroups.com.

[Aaron Meurer]

I'm not sure I'm following you exactly. We are not forking Furo. My
pull request only does three things:

- Modifies some of the CSS variables provided by Furo (the changes in conf.py).
- Adds a custom.css to make some more complicated changes which Furo
does not provide as CSS variables (e.g., tweaking the size of the
"examples" headers).
- Adding a versions.html to the sidebar, with some custom Javascript
to make the "latest" and "dev" versions links (see
https://pradyunsg.me/furo/customisation/sidebar/).

All three of these just build on top of Furo in supported ways.

The only possible concern is that while the custom.css modifications
are supported, they are considered "unstable" (see
https://pradyunsg.me/furo/customisation/injecting/). That means they
could break if a future version of Furo changes how the classes in the
HTML are laid out. If this is a concern we can pin the Furo version
(also recommended by the Furo docs, see
https://pradyunsg.me/furo/stability/). However, if this happens we can
also just fix the CSS for the newer version.

Are you asking if we can turn our modifications into an actual Sphinx
theme, so that it is maintained separately from the repo? I believe
this is possible, although I'm not really sure what benefit it would
bring. At present the Sphinx docs site is the only site that uses this
theme, so there would be no reuse. If we ever had another SymPy
project that wanted to reuse the same theme with the same styling, it
might make sense to do this. For projects that are not directly part
of the SymPy organization I would prefer if they don't use the same
styling, so that the "SymPy green" branding stays unique to the SymPy
documentation.

Aaron Meurer

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AjMfCmd-%3D3PiM9Er%3D%2BCMeSSY%3DXsVjOfKZieNMz8D_GX3g%40mail.gmail.com.

[Jason Moore]

> All three of these just build on top of Furo in supported ways.

Great. I just guessed wrongly, thinking the changes you were making were not supported.

Jason

moorepants.info
+01 530-601-9791

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6KyN9LqTStJgFhVafELNO_tZZf2vN1iyCCSkdDQLFa%2B-A%40mail.gmail.com.