ifconfig and toctree

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Does anyone have any guidance on how to get Sphinx to play nicely with
ifconfig and toctree? What I want to write is something like this (but
quite a bit more complicated) where the ifconfig choices are mutually
exclusive (only one will ever be true).

.. ifconfig:: modeA

.. toctree::

a
b
f
g

.. ifconfig:: modeB

.. toctree::

f
a
g


What I found is that HTML generation only includes the content from the
relevant toctree, but the Table Of Contents includes *every* toctree.

I tried making separate files (index-modeA, index-modeB) and select in the
conf.py but then sphinx whines about "WARNING: document isn't included in
any toctree" for the one not being used even though it is in exclude_patterns.

I did see one example where someone split the toctree into multiple
fragments with an ifconfig before each one and tried to replicate it, but
quickly gave up as I actually have 3 different modes, 12 documents (so
far) and the file was becoming lengthy and unintelligible.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk/AeXcACgkQmOOfHg372QRA1QCdEJb5ueGjhWaEBuM018P/qN0O
ryMAnRSWt8BQt677CrnEyAdw64eAnY3w
=A7aN
-----END PGP SIGNATURE-----

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 25/05/12 23:34, Roger Binns wrote:
> Does anyone have any guidance on how to get Sphinx to play nicely with
> ifconfig and toctree?

I'm encountering yet more issues. This generates a warning about
undefined label abc when debug is False.

normal

.. ifconfig:: debug

This doesn't show up. :ref:`abc`

The warning is nonsensical - the section is not included and I have
verified the text is not present in the output.

It seems to me that ifconfig is virtually useless only working in the most
trivial of circumstances.

It looks like the only solution will be a preprocessor that copies the
source files and evaluates the ifconfig itself completely omitting or
deindenting as appropriate.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk/DGL4ACgkQmOOfHg372QT7GQCePRkIRnb7CFgo9HAd+PVF/Yzb
PRwAnR6QmCqa5uqOQiejVN2Fd2mNrbvc
=o8pw
-----END PGP SIGNATURE-----

[Viktor Haag]

On Saturday, 26 May 2012 02:34:31 UTC-4, Roger Binns wrote:

Does anyone have any guidance on how to get Sphinx to play nicely with 
ifconfig and toctree?  What I want to write is something like this (but 
quite a bit more complicated) where the ifconfig choices are mutually 
exclusive (only one will ever be true). 

 

Supporting two different build sets with a common set of source files seems to be a common problem with Sphinx.

You might want to investigate the "only::" directive as well as, or instead of, "ifconfig::". 'Only' supports build tags given on the command line to make and passes them through into the running Sphinx app... I /believe/ that "ifconfig" depends on values set in conf.py, and I haven't met with much success trying to pass "configuration values" through from the build process into the running application.

The 'tags' that are used as switches in the "only::" directive are accessible in the conf.py file with the 'tags' object, so you might want to investigate using build command line tags in conjunction with "only::" in your sources, plus a conf.py construction that looks something like this:

if tags.has('modeA'):

   # exclude patterns should throw out any modeB-only files

   exclude_patterns = exclude_patterns + ['modeB_toc.rst']

   master_doc = 'modeA_toc.rst'

else if tags.has('modeB'):

   # exclude patterns should throw out any modeA-only files

   exclude_patterns = exclude_patterns + ['modeA_toc.rst']

   master_doc = 'modeB_toc.rst'

else:

   # some default config you're happy to have as default

I do something similar to this, but not precisely like it, so I'm not sure if this method will work, or if it will bring more heartache than helpfulness, but it seems like it would work in theory.

--

Viktor

[Viktor Haag]

On Monday, 28 May 2012 10:20:50 UTC-4, Viktor Haag wrote:

else if tags.has('modeB'):

Except of course that should read as if it were actually Python... 8P

elif tags.has('modeB'): 

Apologies,

--

Viktor 

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 28/05/12 07:20, Viktor Haag wrote:
> I /believe/ that "ifconfig" depends on values set in conf.py, and I
> haven't met with much success trying to pass "configuration values"
> through from the build process into the running application.

I have no problem setting config values for ifconfig and that side of
things works fine. The problem is that Sphinx doesn't do the sensible
thing and instead generates some stuff (HTML toc, ref warnings, excludes)
ignoring ifconfig while generating others after processing the ifconfig.

The only directive doesn't take an expression which is what I need.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk/DjkAACgkQmOOfHg372QRJOACgkwNmtvWgpnxXoBvVddGKar3n
wX4An15L0uo9w45LZ7zHBdRyDF+gRqqi
=qfjS
-----END PGP SIGNATURE-----

[Viktor Haag]

On Monday, 28 May 2012 02:18:38 UTC-4, Roger Binns wrote:

On 25/05/12 23:34, Roger Binns wrote:
> Does anyone have any guidance on how to get Sphinx to play nicely with
> ifconfig and toctree?

I'm encountering yet more issues.  This generates a warning about
undefined label abc when debug is False.

  normal

  .. ifconfig:: debug

     This doesn't show up.  :ref:`abc`

The warning is nonsensical - the section is not included and I have
verified the text is not present in the output.

So, here's what I /think/ is going on with ifconfig, and only, and other conditional text inclusions. And once I made this assumption/realization, the reason it "doesn't work" makes a lot of sense to me.

Sphinx's build obviously uses a multi-pass  process in order to do its work. I'm pretty convinced that what happens is Sphinx first parses/assembles the /entire/ document that it can reach using only the exclusions and rules about what files to look at, and the 'toctree' contents directives it can find to assemble a massive tree of content nodes.

/Then/ as it process that tree of nodes, it will leave out things that should get left out of a particular build by logic contained within the nodes themselves. (Such as 'ifconfig' and 'only' and etc.)

Annoyingly, the processing for certain /kinds/ of things using this massive tree doesn't all happen at once. For example, I /believe/ that Sphinx uses the tree to process/assemble an index, large table of contents structure, and so on, /separately/ from processing the nodes to produce output text (in HTML, TeX, whatever).

What this means is that you might have hiccups if you try to use 'ifconfig', 'only' or similar types of conditional inclusion structures /around/ structural markup that would appear in tables of contents, indices, tables of references and so on, because those bits are parsed separately, and the conditional inclusion directives really only affect the /text output pass/.

Once I made this assumption, I said "damn! these conditional inclusion directives are not nearly as useful as I thought", but it also let me suddenly figure out that to do certain kinds of conditional builds I had to rely on other techniques to do it.

Given that Sphinx' operation works the way it does, I think that properly implementing "conditional inclusion" that operates as you would /hope/ it to is actually a very, very difficult problem (or potentially very difficult).

You /might/ get more mileage out of using some /other/ preprocessor on your RST sources /before/ you actually use Sphinx to do processing on them, if you really have situations where you need to absolutely chop out certain blocks of text before you let Sphinx even do anything with them.

I hope this explanation is helpful to you -- I'm sorry if it's a bit long and/or rambly. (Also, it's possible it could be not entirely correct. 8( )

--

Viktor

[Viktor Haag]

On Monday, 28 May 2012 10:40:00 UTC-4, Roger Binns wrote:

On 28/05/12 07:20, Viktor Haag wrote:
> I /believe/ that "ifconfig" depends on values set in conf.py, and I
> haven't met with much success trying to pass "configuration values"
> through from the build process into the running application.

I have no problem setting config values for ifconfig and that side of
things works fine.  The problem is that Sphinx doesn't do the sensible
thing and instead generates some stuff (HTML toc, ref warnings, excludes)
ignoring ifconfig while generating others after processing the ifconfig.

Well, it does the thing it was designed to do. It's just that maybe that design wasn't communicated effectively when the features were described to end-users. 'ifconfig' and 'only' don't exclude/include stuff entirely as one might expect: that is certainly the case. 8)

 

The only directive doesn't take an expression which is what I need.

Only takes simple boolean expressions with tags passed with -t to the build process. You can use at least AND and OR. I'm not sure if you can use NOT (I haven't tried, and the docs don't say).

If you need more complex expressions than that, agree, 'only' won't work for you.

--

Viktor

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 28/05/12 07:51, Viktor Haag wrote:
> It's just that maybe that design wasn't communicated effectively when
> the features were described to end-users. 'ifconfig' and 'only' don't
> exclude/include stuff entirely as one might expect:

http://sphinx.pocoo.org/ext/ifconfig.html

Include content of the directive only if the Python expression given
as an argument is True ...

That text is completely unambiguous. An accurate description would appear
to be:

Always include the content for some processing (eg excludes, toctree,
refs) but exclude when the expression evaluates to False for other
processing. The behaviour is unpredictable and you won't know exactly
what happens for non-trivial cases until you try it.

The next bit says:

The second argument is the default value, the third should always be True
for such values (it selects if Sphinx re-reads the documents if the value
changes).

And should add:

Some processing occurs before the document is re-read and some is done
after the document is re-read. Good luck.

I'm busy writing code now that reads the document, processes the ifconfig
and either omits it completely or dedents it if True. That is really the
effect ifconfig should have had in the first place.

Roger




-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk/D1K8ACgkQmOOfHg372QSOawCfSuHc1pPGBeHZu4zOliujvvGn
/q8An1osSAW3x//eKlddutJJvBFgJaIV
=q7TH
-----END PGP SIGNATURE-----

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 28/05/12 07:47, Viktor Haag wrote:
> Sphinx's build obviously uses a multi-pass process in order to do its
> work. I'm pretty convinced that what happens is Sphinx first
> parses/assembles the /entire/ document that it can reach using only the
> exclusions and rules about what files to look at, and the 'toctree'
> contents directives it can find to assemble a massive tree of content
> nodes.

It is far more subtle than that. For example all files (completely
ignoring excludes) are read and you'll get errors for any files that have
toctree even if they have been explicitly excluded.

> Annoyingly, the processing for certain /kinds/ of things using this
> massive tree doesn't all happen at once. For example, I /believe/ that
> Sphinx uses the tree to process/assemble an index, large table of
> contents structure, and so on, /separately/ from processing the nodes
> to produce output text (in HTML, TeX, whatever).

The PDF (via rst2pdf) TOC doesn't include the contents of all toctrees,
but HTML does.

> What this means is that you might have hiccups if you try to use
> 'ifconfig', 'only' or similar types of conditional inclusion structures
> /around/ structural markup that would appear in tables of contents,
> indices, tables of references and so on, because those bits are parsed
> separately, and the conditional inclusion directives really only affect
> the /text output pass/.

The references are particularly painful. References aren't structural in
any way, just a reference to somewhere else. Not being able to use
references in ifconfig sections is really painful.

Even not being able to do headers was getting annoying since I wanted to
use different wording in them depending on the context.

> Given that Sphinx' operation works the way it does, I think that
> properly implementing "conditional inclusion" that operates as you
> would /hope/ it to is actually a very, very difficult problem (or
> potentially very difficult).

If Sphinx did only/ifconfig as the very first pass then things would work
perfectly. The problem is it appears to do it in the middle which is what
leads to all the problems.

> You /might/ get more mileage out of using some /other/ preprocessor on
> your RST sources /before/ you actually use Sphinx to do processing on
> them, if you really have situations where you need to absolutely chop
> out certain blocks of text before you let Sphinx even do anything with
> them.

All I'm doing in the preprocessor that I am writing is doing ifconfig as a
first pass operation.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk/D38sACgkQmOOfHg372QQFqACfZf1r6/lGIkVWoBuuk+Bz+uj7
Ya8AoIf2fHoRnMm3v11YbO1j1tM/5/fB
=9Zyf
-----END PGP SIGNATURE-----

[Viktor Haag]

On Monday, 28 May 2012 16:27:55 UTC-4, Roger Binns wrote:

> Given that Sphinx' operation works the way it does, I think that
> properly implementing "conditional inclusion" that operates as you
> would /hope/ it to is actually a very, very difficult problem (or
> potentially very difficult).

If Sphinx did only/ifconfig as the very first pass then things would work
perfectly.  The problem is it appears to do it in the middle which is what
leads to all the problems.

That would seem to be the right thing to do, yes. It might have follow-on implications, but at least it would act as a complete conditional to chop out blocks of text in complete ignorance of what those blocks include.

Frankly, I'd like to see a conditional of both types: one that gets done like that, as the first operation (as in the C-preprocessor) and one that gets done after the Sphinx app has some intelligence about the actual contents of the document.

--

V.