Graphics files in Sage documentation

[Marc Culler]

At John Palmieri's suggestion, I am duplicating this message that I posted on sage-release on this list:

Details really do matter.  The devil is in them.  Here is one which makes a major difference to my efforts to distribute Sage.  I hope that I can convince people to pay attention to it.

DETAIL: An svg image can be compressed very efficiently and will look good at all resolutions.  A png image can not be compressed, being already compressed, and pixelates at high resolutions.

BACK STORY: I am working on adding documentation to the Sage_macOS distribution.  Size is important.  For 10.1beta3 the English documentation occupied 631MB, and compressed down to 104MB.  For 10.1beta4 the uncompressed size increased by over 100MB due to the addition of 100MB of png images.  Now it only compresses to 207MB, doubling the compressed size.

There is no need for this wastefulness.  Here is an example:

sage: D = graphs.DodecahedralGraph()
sage: p = D.plot()
sage: p.save_image('/tmp/dodec.svg')
sage: p.save_image('/tmp/dodec.png')

The result:

% ls -l /tmp/dodec*
-rw-r--r--@ 1 culler  wheel  42735 Jun 28 08:48 /tmp/dodec.png
-rw-r--r--  1 culler  wheel  18524 Jun 28 08:48 /tmp/dodec.svg
% gzip /tmp/dodec*
% ls -l /tmp/dodec*
-rw-r--r--@ 1 culler  wheel  39473 Jun 28 08:48 /tmp/dodec.png.gz
-rw-r--r--  1 culler  wheel   3815 Jun 28 08:48 /tmp/dodec.svg.gz

BOTTOM LINE: we get more than a 90% reduction in size simply by choosing to use the .svg extension when saving the plot instead of the .png extension.

I am not allowed to upload an svg file to google groups, so you will have to verify for yourself that the svg looks better than the png, but it does.

- Marc

[Michael Orlitzky]

On Wed, 2023-06-28 at 10:07 -0700, Marc Culler wrote:
>
> BOTTOM LINE: we get more than a 90% reduction in size simply by choosing to
> use the .svg extension when saving the plot instead of the .png extension.
>

SVG is the right choice for most graphics, but there are some practical
problems:

* The documentation needs to know how big to display an SVG graphic.
With a PNG, the default is to use the image's height/width in 
pixels, but with SVG, there's no such obvious default.

* gzipped SVG doesn't work over the file:/// protocol in my firefox.
This hasn't been a big enough problem for me yet to diagnose it,
so I can't say how serious a problem it is. (I'll play around later
today.)

* Browser support in firefox/chrome alternatives still isn't great,
although I think webkit is getting a new SVG renderer "soon." This
is actually relevant more today than it was ten years ago, because 
adding rust to firefox made it less portable, meaning you're
more likely to be stuck with one of those alternatives.

* Somebody's got to go through and look at 100MB of images to make 
sure they still look right if we change 'em.

[Marc Culler]

I am not proposing using gzipped svg for a normal sage build.  I only want to do it for the Sage_macOS app, which will have a simple self-contained server running on the loopback interface that provides gzipped content (with the Content-Encoding header set to "gzip").  Of course I would be happy to work on incorporating the same mechanism in a standard build of Sage, and that would reduce the footprint of the documentation considerably.  But in the short term, for the macOS binary, I need to be able to find the magic switch which makes sage_docbuild use svg instead of png.

I don't know how much of a problem the image size will pose.  There are some svg images already in the documentation, which seem to work, but I hadn't gotten to the stage of worrying about that yet.

- Marc

[John H Palmieri]

I tried making this change (which would affect each ".. PLOT:" directive) but the documentation failed to build:

diff --git a/src/sage_docbuild/conf.py b/src/sage_docbuild/conf.py
index 9b6b37480b..db33c4b1d6 100644
--- a/src/sage_docbuild/conf.py
+++ b/src/sage_docbuild/conf.py
@@ -115,7 +115,7 @@ def sphinx_plot(graphics, **kwds):
             mpl.rcParams['figure.figsize'] = [8.0, 6.0]
             mpl.rcParams['figure.dpi'] = 80
             mpl.rcParams['savefig.dpi'] = 100
-            fn = tmp_filename(ext=".png")
+            fn = tmp_filename(ext=".svg")
             graphics.save(fn)
             img = mpimg.imread(fn)
             plt.imshow(img)

The error: ValueError: filetype .svg not supported by save()

[Marc Culler]

I think I may have found a resolution of this problem, that involves doing almost nothing.  It appears that sage_docbuild oftern builds image in all three formats - svg, png and pdf.  All three types get stored in the html directory and the svg is duplicated:

find local/share/doc/sage/html/en -name 'chart-12*'
local/share/doc/sage/html/en/reference/manifolds/_images/chart-12.svg.gz
local/share/doc/sage/html/en/reference/manifolds/sage/manifolds/chart-12.svg.gz
local/share/doc/sage/html/en/reference/manifolds/sage/manifolds/chart-12.png
local/share/doc/sage/html/en/reference/manifolds/sage/manifolds/chart-12.pdf

The actual html code seems to only uses the svg files stored in _images.  Hopefully I can just remove the others without affecting the html documentation.

grep -r chart-12 local/share/doc/sage/html/en/reference/manifolds
local/share/doc/sage/html/en/reference/manifolds/sage/manifolds/chart.html:<img alt="../../_images/chart-12.svg" class="plot-directive" src="../../_images/chart-12.svg" /></figure>

Since this seems to be a change between 10.1beta3 and 10.1beta4, I would have to guess that it might be considered a bug.  So maybe I should move this back to sage-release?

- Marc

[Marc Culler]

Note: I had already compressed the .svg files, hence the .gz extension.

Before compression I get:

% find local/share/doc/sage/html/en -name 'chart-12*'
local/share/doc/sage/html/en/reference/manifolds/_images/chart-12.svg
local/share/doc/sage/html/en/reference/manifolds/sage/manifolds/chart-12.svg
local/share/doc/sage/html/en/reference/manifolds/sage/manifolds/chart-12.png
local/share/doc/sage/html/en/reference/manifolds/sage/manifolds/chart-12.pdf

- Marc

[John H Palmieri]

This change should produce only the .svg files:

diff --git a/src/sage_docbuild/conf.py b/src/sage_docbuild/conf.py
index 9b6b37480b..b5d6815c76 100644
--- a/src/sage_docbuild/conf.py
+++ b/src/sage_docbuild/conf.py
@@ -128,7 +128,7 @@ from sage.all_cmdline import *
 """
 
 plot_html_show_formats = False
-plot_formats = ['svg', 'pdf', 'png']
+plot_formats = ['svg']
 
 # We do *not* fully initialize intersphinx since we call it by hand
 # in find_sage_dangling_links.

(I'm not suggesting this as a general change, just for your particular situation.)

[John H Palmieri]

By the way, on my machine, local/share/doc/sage are almost identical sizes in 10.1.beta3 and 10.1.beta4. I don't know what explains the difference you're seeing.

[Marc Culler]

Thank you!  Probably it is my bug, or it is somehow triggered by my wierd setup.  That would probably mean some environment difference in the sage or sage-env scripts, since those are basically the only things I have tinkered with.

Can you verify that you do not see any image files in the top level of, say:

local/share/doc/sage/html/en/reference/manifolds/sage/manifolds ?

- Marc

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/a5t3mFJNMDI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/95aeb60f-5705-4d9e-a17b-2c26f118e724n%40googlegroups.com.

[Michael Orlitzky]

On Wed, 2023-06-28 at 13:50 -0400, Michael Orlitzky wrote:
>
> * gzipped SVG doesn't work over the file:/// protocol in my firefox.
> This hasn't been a big enough problem for me yet to diagnose it,
> so I can't say how serious a problem it is. (I'll play around later
> today.)

Firefox bug opened 23 years ago:

https://bugzilla.mozilla.org/show_bug.cgi?id=52282



If they were going to fix it, it'd be fixed, so I'd suggest doing the
comparison uncompressed. A lot of people (statistics all made up) read
the docs locally with firefox.

[Dima Pasechnik]

One can always start a web server on localhost, instead of using file:/// :P

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/28b1bd234f8a344062c0bd103dd53760ff24c693.camel%40orlitzky.com.

[Marc Culler]

Right. That is what I will be doing.

- Marc

You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/a5t3mFJNMDI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq1Y5_EK0NhjmuzDkC8hvNC5Rg7zchiYAB7kMi2r2i_RkA%40mail.gmail.com.

[John H Palmieri]

I confirm that I do not see any image files in local/share/doc/sage/html/en/reference/manifolds/sage/manifolds. All of the images (and all are in svg format) are in local/share/doc/sage/html/en/reference/manifolds/_images/. These are copies of files in local/share/doc/sage/inventory/en/reference/manifolds/sage/manifolds/ and its subdirectories. If your system can handle symlinks, and if the inventory directory is intact, you could create some symlinks to save more space. (The inventory build is where in the default build I see png, svg, and pdf files, but only svg files seem to be copied to the html directory.)

--

John

[Michael Orlitzky]

On 2023-06-28 23:27:22, Dima Pasechnik wrote:
> One can always start a web server on localhost, instead of using file:/// :P
>

I know you're mostly joking, but that's not as easy as it sounds. To
start the local web server as a non-root user, you have to run it on
an unprivileged (i.e. not the default of 80) port. Then you have to
teach it about mime types and gzip, and probably add a line or two
about image/svg+xml and the svgz extension.

Then you have to secure it somehow. Other local users can hit that TCP
port, and it will be running with your desktop user's privileges, so
it can see all of your files. Your server might support path
restrictions, but does it protect against the past 30 years' worth of
path traversal hacks and stupid symlink tricks? Maybe if you're using
apache or nginx, but if you're using "python -m http.server", I
wouldn't count on it.

[Dima Pasechnik]

On Thu, Jun 29, 2023 at 3:24 AM Michael Orlitzky <mic...@orlitzky.com> wrote:
>
> On 2023-06-28 23:27:22, Dima Pasechnik wrote:
> > One can always start a web server on localhost, instead of using file:/// :P
> >
>
> I know you're mostly joking, but that's not as easy as it sounds.

no, why? If you run ./sage -n you'll see something like

To access the notebook, open this file in a browser:
file:///home/dima/.local/share/jupyter/runtime/nbserver-15031-open.html
Or copy and paste one of these URLs:
http://localhost:8888/?token=067f4b0aab24836fd21c349e9cfa2e78c29b183444e6b44d
or http://127.0.0.1:8888/?token=067f4b0aab24836fd21c349e9cfa2e78c29b183444e6b44d

- so jupyter offers you an alternative like this. And it's very useful
if e.g. your Sage session is run in a VM-like
environment, e.g. in a Crostini container (an LXC container) so that
the browser can't directly access files, but
HTTP has a tunnel set up.

> To
> start the local web server as a non-root user, you have to run it on
> an unprivileged (i.e. not the default of 80) port. Then you have to
> teach it about mime types and gzip, and probably add a line or two
> about image/svg+xml and the svgz extension.
>
> Then you have to secure it somehow. Other local users can hit that TCP
> port, and it will be running with your desktop user's privileges, so
> it can see all of your files. Your server might support path
> restrictions, but does it protect against the past 30 years' worth of
> path traversal hacks and stupid symlink tricks? Maybe if you're using
> apache or nginx, but if you're using "python -m http.server", I
> wouldn't count on it.
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/ZJzrXxtT1lnmuywr%40stitch.

[Michael Orlitzky]

On 2023-06-29 23:30:57, Dima Pasechnik wrote:
> On Thu, Jun 29, 2023 at 3:24 AM Michael Orlitzky <mic...@orlitzky.com> wrote:
> >
> > On 2023-06-28 23:27:22, Dima Pasechnik wrote:
> > > One can always start a web server on localhost, instead of using file:/// :P
> > >
> >
> > I know you're mostly joking, but that's not as easy as it sounds.
>
> no, why? If you run ./sage -n you'll see something like
>

Oh, sorry then. You could probably guess that if I run sage -n,
nothing happens, because I don't have the notebook installed. First
I'd have to re./configure sage and pull down a few hundred megabytes
of random crap from pypi.

Once you have all that installed, Jupyter uses a custom web server
with token-based authentication to solve the permission problem. But
in trade,

* Every user needs to run his own server;

* You've upped the ante from "read my files" to "execute
arbitrary code" if there's a bug in Jupyter.

Given all that I'd still say it's more complicated than "just launch a
local web server" makes it sound.