generic float (numbered figure) directive

[Achab Bachadiensis]

Hi everybody,

I need to generate reports with numbered (floating) plots produced via matplotlib's plot directive (it only creates in-place images, not floats). I posted a question on that http://stackoverflow.com/questions/30374317/sphinx-floating-numbered-matplotlib-plot without getting any answer, and googling around, I found no extension which implements something similar.

I see there are two ways forward:

1. implement a figplot (or whatever) directive which will wrap plot as figure.

2. implement a generic float (or whatever else) directive which can had arbitrary body and arbitrary caption.

I can approximate no. 2 by using this (fig/pixel.png is 1x1 transparent image):


.. figure:: fig/pixel.png
 
   This is a caption

   .. plot::

      # matplotlib code here

but it has its drawbacks, namely that the caption is above the plot (in both HTML and LaTeX). SInce I primarily target ATM, I fix those by dirty

   re.sub(r'^\\includegraphics{pixel.png}\n?(?P<cap>^\\caption{.*?}){\\small\s*(?P<img>.*?)}(?P<end>\\end{figure})',r'\g<img>\g<cap>\g<end>',f,flags=re.MULTILINE|re.DOTALL)

which puts things as they should be.

I am at this moment not capable of delving into docutils and implementing proper extension (be it (1) or (2)) myself. So I would like to offer a financial bounty to someone who could contribute this (contact me by PM or here if you are interested).

If there is some cleaner workaround to have numbered plots, I will be grateful for any pointers.

Cheers, Václav

[Guenter Milde]

On 2015-05-22, Achab Bachadiensis wrote:

> I need to generate reports with numbered (floating) plots produced via
> matplotlib's plot directive (it only creates in-place images, not floats).
> I posted a question on that
> http://stackoverflow.com/questions/30374317/sphinx-floating-numbered-matplotlib-plot
> without getting any answer, and googling around, I found no extension which
> implements something similar.

> I see there are two ways forward:

> 1. implement a figplot (or whatever) directive which will wrap plot as
> figure.

> 2. implement a generic float (or whatever else) directive which can had
> arbitrary body and arbitrary caption.

This is a long-discussed approach for Docutils (cf.
http://docutils.sourceforge.net/docs/dev/todo.html#object-numbering-and-object-references)
However, due to open issues and missing manpower, there is no movement on
this front currently.

...

> If there is some cleaner workaround to have numbered plots, I will be
> grateful for any pointers.

3. Create the plots before the Docutils/Sphinx run (e.g. via a wrapper
script or makefile) and use the figure directive to include the
graphic files.

Hope this helps,

Günter

[Achab Bachadiensis]

3. Create the plots before the Docutils/Sphinx run (e.g. via a wrapper
   script or makefile) and use the figure directive to include the
   graphic files.

Thanks.

I would have to duplicate the whole machinery of updating the plots, and, more importantly, the plot parameters would not be in the same file (unless I parse the RST with special comments, write it back and things like that)

Fortunately I found out that sphinxtr's figtable can include any content (including images, plots, etc) and make it float; it just does not support multi-paragraph captions. For now I stay with the regex thing, but hope to switch to that once multi-paragraph captions are implemented and merged.

Which leads me to ask (though I am sure it was discussed before) if there is a chance of including some of the sphinxtr's extensions either in sphinx core or as somewhat more official extensions -- at least figtable and subfig come to mind.

Cheers, václav

[Ignacio Fernández Galván]

I implemented something for generic floats in sphinx, and submitted a pull request that is still waiting:

https://github.com/sphinx-doc/sphinx/pull/1858

.. float::
   :type: figure
   :caption: My caption

   Arbitrary content

[Achab Bachadiensis]

Wow, great job, almost exactly what I need. Almost: can it do multi-paragraph captions? They need some care when exported to LaTeX (e.g. as \caption[par1]{par1 \par par2 \par par3}). v.

[Ignacio Fernández Galván]

I haven't tried, but I haven't considered it either, so they probably won't work fine. The LaTeX writer has too many problems for my needs, anyway, so I'll probably have to write something specific for my project.

[Bram Geron]

What problems are you having?

I'm currently applying some customisation to write out a .tex for a conference submission, and things look promising. I couldn't do some things I needed (e.g. I did need to modify writers.latex.HEADER), but I'm monkeypatching that with an extension.

I'd love to know of any future problems I might encounter.

Cheers, Bram

[Ignacio Fernández Galván]

I'd like code blocks to use the listings packages.

I need tables more customizable, ideally using booktabs, more compact and without vertical lines.

No indent in "compound-middle" paragraphs (this is probably configurable now).

Working footnotes, at least within tables.

Some other details are probably customizable now too, but I haven't looked further yet, not until I have the tables and code blocks fixed. I'm working on porting a user's manual written originally in LaTeX into sphinx, so keeping the quality of the LaTeX-PDF output is important.

[Guenter Milde]

Hi Ignacio,

On 2015-06-23, Ignacio Fernández Galván wrote:

> I'd like code blocks to use the listings packages.

> I need tables more customizable, ideally using booktabs, more compact and
> without vertical lines.

This is both possible with the Docutils "latex" writer.

> No indent in "compound-middle" paragraphs (this is probably configurable
> now).

I am not sure about this one, but it could be made possible.

> Working footnotes, at least within tables.

This is possible with the Docutils "latex" writer, too.

The problem is, that Sphinx handles LaTeX output differently: while the
parser, the transfomrs and the HTML writer inherit from Docutils, the Sphinx
LaTeX writer is a fork of a very outdated Docutils version and the
customized to the narrow task of working for the "sphinx" LaTeX
package/documentclass.
Until now, no one took on the task to rebase it on a current version of the
upstream writer (merging the best of the two worlds). Rather, some elements
were taken over or reinvented triggered by feature requests.

If there is someone from the Sphinx side interested in bringing the LaTeX
writers of Docutils and Sphinx together, I am ready to work on this project
from the Docutils side.

Günter

[Bram Geron]

Interesting! You seem have a grasp on how the two have diverged; can you shed some light on that? Is it just details like what specific strings to put where? Is the architecture for overriding strings ('latex_elements') different? Is the Docutils LaTeX writer even based on walkabout?

Cheers, Bram

[Guenter Milde]

On 2015-06-23, Bram Geron wrote:

> [-- Type: text/plain, Encoding: quoted-printable --]

> Interesting! You seem have a grasp on how the two have diverged; can you
> shed some light on that?

In the beginning, there was just Docutils.

When Sphinx was created araound 2007, the latex builder and writer were
created as independent, Sphinx-specific modules, with inspiration from
the Docutils LaTeX writer Release 0.5 (2008-06-25). I don't know the
details that led to this decision, but imagine that problems with the
upstream writer and the different requirements for the PDF version of the
Python documentation were the main reason.

The Docutils LaTeX writer went through some major development and
improvement cycles since then (including the addition of a "xetex" writer
inheriting from the latex2e writer). See the Docutils' HISTORY__ file
and the repository at http://sourceforge.net/p/docutils/code for details.
There is also a git mirror at http://repo.or.cz/docutils.git

__ http://docutils.sourceforge.net/HISTORY.html

> Is it just details like what specific strings to put where?

No, it was a newinplementation from the beginning and independent
development for 8 years. Even with some "cross pollunation", there are
plenty of details differing.

OTOH, the main task: "take a Docutils document tree and convert it into LaTeX
code" is similar and the Sphinx writer imports and uses parts of Docutils:

from docutils import nodes, writers
from docutils.writers.latex2e import Babel

> Is the architecture for overriding strings ('latex_elements')
> different?

Yes.

> Is the Docutils LaTeX writer even based on walkabout?

Yes, the use of the "visitor pattern" is a core element of Docutils from the
very beginning (see `PEP 0258 -- Docutils Design Specification`__ from 2001)

walkabout() is provided by Docutils' nodes.py and used in both
systems.

__https://www.python.org/dev/peps/pep-0258



Hope this helps,

Günter

[Bram Geron]

On Wednesday, 24 June 2015 11:35:07 UTC+1, Guenter Milde wrote:

On 2015-06-23, Bram Geron wrote:

> Interesting! You seem have a grasp on how the two have diverged; can you
> shed some light on that?

In the beginning, there was just Docutils. 

 

[snip]

Thanks for the background and pointers. I checked out the Docutils LaTeX writer, and it seems at least a lot better documented, and more extensible too. 

It's not important for me now to bring the two together, but it seems a right thing to do, and it might probably help both projects to be more coordinated. However, I'm not sure about backwards compatibility: there must be a number of people who use Docutils writer or the Sphinx writer, and have configured them each to their needs; if we were to unify the writers I don't see how a significant number of existing configurations might stay functional. Thoughts?

 

Cheers, Bram

[Guenter Milde]

On 2015-06-29, Bram Geron wrote:
> On Wednesday, 24 June 2015 11:35:07 UTC+1, Guenter Milde wrote:
>> On 2015-06-23, Bram Geron wrote:

> Thanks for the background and pointers. I checked out the Docutils LaTeX
> writer, and it seems at least a lot better documented, and more extensible
> too.

> It's not important for me now to bring the two together, but it seems a
> right thing to do, and it might probably help both projects to be more
> coordinated. However, I'm not sure about backwards compatibility: there
> must be a number of people who use Docutils writer or the Sphinx writer,
> and have configured them each to their needs;

Up to now, using Docutils (to convert a stand-alone document) means using
the Docutils LaTeX writer, using Sphinx (to convert a "project") means using
the Sphinx LaTeX builder and writer.

Configuration of Docutils is done via command line arguments and
configuration files (see
http://docutils.sourceforge.net/docs/user/config.html)
configuration of Sphinx is done in a Python file.
(BTW: as Sphinx uses the Docutils HTML writer, HTML output can also be
configured via Docutils configuration files)

> if we were to unify the writers I don't see how a significant number of
> existing configurations might stay functional. Thoughts?

For Docutils, I would continue the existing practice: make backwards
compatibility a priority - with exceptions announced in the RELEASE-NOTES
before actually implemented.

I suppose that most changes to the Docutils writer required for better
working with Sphinx will not affect the "standard use case".

For Sphinx, I see two options:

a) Gradual development of the existing writer by porting recent
developments/improvements. This may also include import of certain
features/functions/auxiliary classes (as already done for the Babel class).

b) Provide a new LaTeX writer inheriting from Docutils "latex2e" writer.
This could be done parallel to the existing one, so that existing setups
continue using the "traditional sphinx-latex-writer" while new projects
may select the alternative.

Günter