sphinx build painfully slow

[Olaf]

Hi,

We have documentation for hardware modules that is written in rst and translated to HTML with sphinx. 

We are translating about ~300000 lines of rst. Which is not small but also not absolutely huge:

olaf@pc:~/ee/doc$ find . -name '*.rst' | xargs wc -l

[...]

335911 total 

You can take a look at the documentation here: http://www.tinkerforge.com/en/doc/index.html

The building process takes about 10 minutes if i don't change anything after the last build (i.e. everything that can be cached should be cached):

olaf@pc:~/ee/doc$ time make html

[...]

real 10m27.748s

user 9m30.036s

sys 0m6.932s

We will be adding lots of new hardware modules soon, which will double the amount of documentation...

Adding stuff to the documentation has become really painful since it takes so long before one can review it. So the question is: How can we speed this up?

For us it would also be helpfull if we were able to only build a part of the documentation. Is that somehow possible?

Thanks!

Olaf

[shirou]

Hi Olaf,

How about using -j option to parallel build?
http://sphinx-doc.org/latest/invocation.html

This option is added at version 1.2b1 and currently experimental. But
it may help.

Thank you
WAKAYAMA Shirou


2013/7/17 Olaf <ol...@tinkerforge.com>:

> --
> You received this message because you are subscribed to the Google Groups
> "sphinx-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sphinx-users...@googlegroups.com.
> To post to this group, send email to sphinx...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sphinx-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Olaf]

I updated to 1.2b1 (from 1.1.3) and the build is 25% faster now, without the -j option!

But with -j N (i triead N=1-4) the build is not faster, in fact it is a little bit slower :).

[shirou]

> I updated to 1.2b1 (from 1.1.3) and the build is 25% faster now, without the -j option!

I'm happy to hear that.

> But with -j N (i triead N=1-4) the build is not faster, in fact it is a little bit slower :).

On my understanding, -j option makes parallel only writing files. Not
fit your environment at this time.

Regards,

[Sam Kleinman]

There are some minor fixes in the latest HEAD with parallel that might
improve your overall experiences. Nothing should change timing very
much.

The -j option only speeds the write-phase of the build. I'm not
*totally* sure about all of the particulars, but I've noticed the
following:

- The amount of speed up is more closely related to the number of files
than the total length of the text.

This more conjectural, but I expect that Sphinx will run builds with
small numbers of files in parallel more effectively *if* the files are
roughly the same size.

- You'll see greater improvements if your document require less
processing on the part of Sphinx: your sphinx extensions are still a
singly-threaded bottleneck, any processing of input (includes, cross
referencing, indexing, etc,) has to happen before multiprocessing.

- If you're running singlhtml or latex/man/texinfo with a small number
of outputs, the parallel builder won't help at all.

I hope these conclusions will be pretty self evident. And please feel
free to correct me if I'm wrong.

Cheers,
sam/tychoish

--
Sam Kleinman (tychoish):
- ga...@tychoish.com
- tychoish <http://tychoish.com/>
"don't get it right, get it written" -- james thurber

[purnank h g]

Hi Olaf,

I am worried about:

   ``The building process takes about 10 minutes if i don't change anything after the last build (i.e. everything that can be cached should be cached):``

I tried to reproduce similar behaviour with my biggest book (Bible, both old and new testaments, http://www.purnank.in/books/samples/theBibleOnNTsKJV/, Lines: 112178) and reproduced similar anomaly.  Incremental builds are also take long time for me.  I tried to dig deeper into the issue and found the culprit at my end.

For me, Nested :numbered: directives always think that the files have changed during incremental builds.

(contents of root.rst)

   .. toctree:

       :numbered:    <<<<<<<<<<<<<<<<<<<<<<<<< culprit >>>>>>>>>>>>>>>>>>>>>>>>>

       

       old-testaments

       new-testaments

(contents of old-testaments.rst)

   .. toctree:

       :numbered:

       :glob:

       

       old-testaments/*

(contents of new-testaments.rst)

   .. toctree:

       :numbered:

       :glob:

       

       new-testaments/*

If i change my root.rst like this. (:numbered: removed)

(new contents of root.rst)

   .. toctree:

       

       old-testaments

       new-testaments

The incremental builds finish within seconds. I hope your build time issue is due to a similar anomaly...

OTOH: I have also found that a very big global rst-pre-include / post include has a strongly negative impact on overall build times.  Try to include only necessary files in each individual document.

--

Purnank

[Michael Schlenker]

Am 16.07.2013 17:34, schrieb Olaf:
> Hi,
>
> We have documentation for hardware modules that is written in rst and
> translated to HTML with sphinx.
>

[snip]

> The building process takes about 10 minutes if i don't change anything
> after the last build (i.e. everything that can be cached should be cached):
> olaf@pc:~/ee/doc$ time make html
> [...]
> real10m27.748s
> user9m30.036s
> sys0m6.932s
>
> We will be adding lots of new hardware modules soon, which will double
> the amount of documentation...
>
> Adding stuff to the documentation has become really painful since it
> takes so long before one can review it. So the question is: How can we
> speed this up?

One way to speed it up a bit is skipping the build of the search index.
For my setup, this cut build times by roughly 30% (needs a patch
though), and as we didn't use the built in html search it was very useful.

Removing most of the contents from the prefix/postfix block also helped
a lot. It really hurts to have lots of small .rst files and one huge
file with substitutions defined, as doctools/sphinx reparses them over
and over again, no 'precompiled headers' style parsing. So if you have a
large (and even 500 lines can be large...), include inside your prefix,
try to shrink it, or remove it for a test.

Michael

--
Michael Schlenker
Software Architect

CONTACT Software GmbH Tel.: +49 (421) 20153-80
Wiener Stra�e 1-3 Fax: +49 (421) 20153-41
28359 Bremen
http://www.contact.de/ E-Mail: m...@contact.de

Sitz der Gesellschaft: Bremen
Gesch�ftsf�hrer: Karl Heinz Zachries, Ralf Holtgrefe
Eingetragen im Handelsregister des Amtsgerichts Bremen unter HRB 13215