Multi-CPU compilation
Theo Markettos
is very much a single-threaded program, and will probably always remain so.
But even on fast machines, some big documents can take minutes to compile.
This gets worse with math-intensive packages like TiKZ.
So here's a slightly mad idea. Suppose you could split up your document
into sections (chapters most likely). Is there a way you could fire each of
these off at a different CPU, and then recombine them afterwards?
I know there are various packages like subfiles and subdocs for doing
related things, but would they be bendable to compiling sections to be
concatenated afterwards? References would presumably be a problem - would
there be some way to patch up the .aux files so that the references came out
right?
To me it wouldn't matter so much if the final version didn't come out
perfectly - I'm quite happy to run the final copy through normal Latex.
Parallel compiling would just help in the writing/debugging phase.
So can anyone tell me the hundred reasons why it won't work? ;-)
Thanks
Theo
Bob Tennent
If you're on a Linux or Unix machine, prepare a Makefile
that specifies separate build steps for parts of the document.
Then do a make -j n where n is the number of processors. If the
build steps are independent, they'll be done in parallel.
Patricia Shanahan
make -j n also does parallel make on MS-Windows systems if done under
Cygwin.
Patricia
Theo Markettos
> If you're on a Linux or Unix machine, prepare a Makefile
> that specifies separate build steps for parts of the document.
> Then do a make -j n where n is the number of processors. If the
> build steps are independent, they'll be done in parallel.
Indeed, that bit's easy. But since this is comp.text.tex I was wondering if
there's a sensible way to split up the document, build each bit separately
and merge them together afterwards. Taking care of things like references
along the way. Is that possible?
Theo
Will Robertson
<theom...@chiark.greenend.org.uk> said:
My brains hurts trying to resolve the dependencies, so I'd say "no" in
general. You could get a first approximation with a scheme as you
suggest, but consider -- material at the end of the document will be
affecting material at the beginning of the document (order of the
bibliography, other cross references, changing the size of the ToC).
As far as I know (which isn't very much), the only way to resolve such
problems are multiple, linear passes over the document.
Will
Tim Arnold
news:Kiv*Tz...@news.chiark.greenend.org.uk...
What I do is not exactly what you're asking, but fwiw:
I compile a book composed of several chapters first and copy the *.aux files
to a networked disk location. Then I execute all the chapter builds in
multiple processes. I use a set up where each chapter has its own
subdirectory, some python scripting and wonderful \includeonly LaTeX
directive. The cross-chapter references come out fine since each chapter is
looking at the master aux file on the network.
At the end I've got the book pdf and separate chapter-level pdfs.
--Tim Arnold
smith...@sbcglobal.clothes.net
>
> What I do is not exactly what you're asking, but fwiw:
> I compile a book composed of several chapters first and copy the *.aux files
> to a networked disk location. Then I execute all the chapter builds in
> multiple processes. I use a set up where each chapter has its own
> subdirectory, some python scripting and wonderful \includeonly LaTeX
> directive. The cross-chapter references come out fine since each chapter is
> looking at the master aux file on the network.
> At the end I've got the book pdf and separate chapter-level pdfs.
>
> --Tim Arnold
>
. . .and now I'm thinking about distributing different chapters to
different machines and doing the build that way. I'd like more details
on your approach. Thanks.
--
J'm Sm'th
Joris
Will Robertson
<smith...@sbcglobal.clothes.net> said:
> Tim Arnold wrote:
>
>> What I do is not exactly what you're asking, but fwiw:
>> I compile a book composed of several chapters first and copy the *.aux files
>> to a networked disk location. Then I execute all the chapter builds in
>> multiple processes. I use a set up where each chapter has its own
>> subdirectory, some python scripting and wonderful \includeonly LaTeX
>> directive. The cross-chapter references come out fine since each chapter is
>> looking at the master aux file on the network.
>> At the end I've got the book pdf and separate chapter-level pdfs.
>
> . . .and now I'm thinking about distributing different chapters to
> different machines and doing the build that way.
Having thought about this further, it doesn't seem as crazy as it first
sounded. (Be warned, though: I'm sleep deprived and likely to be
thinking crooked.)
Imagine you've got two halves of a document. One with pages 1-10, the
other with pages 11-20. The first half as a phrase "see section 5.5 on
page 15" and the second half has a phrase "see section 1.1 on page 2".
Let's compile them in parallel. Say it takes 5 secs per chapter, and
assume we can glue them together instantly to create the final document.
- The first time you compile the two in parallel, they have no idea
about page numbers or crossreferences. First half is numbered 1-10; so
is the second half. Five seconds gone.
- Second run in parallel, page numbers are now correct, and so are
section number cross-references. But because the page numbers weren't
correct at the start of this run, the page number cross-references are
still wrong. Ten seconds gone total.
- Third run in parallel: finally the page number cross-references
resolve. Fifteen seconds in total.
Contrast that to running the whole document linearly twice: that would
take twenty seconds. And having more cores with more chapters would
improve the ratio. The difficult part would be synchronising the .aux
files, but it shouldn't be impossible.
Will
Tim Arnold
news:hdcno1$2qgp$1...@adenine.netfront.net...
The main thing is to build the book first, that's why I don't think it's
what you're looking for.
Anyway, once the book has finished, I copy all the *.aux files to a central
location.
Then for each chapter, I start a process on a remote machine that:
(1) adds that aux directory to the TEXINPUTS environment variable
(2) writes a mock book.tex file that uses includeonly for that particular
chapter
(3) runs latex on that book.tex file.
Say the book takes 5 minutes to compile with bibtexing, makeindex, etc.
The chapters takes only a few seconds, so at the end (5 minutes and some
seconds), you've got the book pdf and all the chapter-level pdfs finished.
If you don't need chapter-level pdfs, this method doesn't buy you anything
since in the first step the entire book is compiled.
--Tim Arnold
smith...@sbcglobal.clothes.net
>
> The main thing is to build the book first, that's why I don't think it's
> what you're looking for.
>
> Anyway, once the book has finished, I copy all the *.aux files to a central
> location.
> Then for each chapter, I start a process on a remote machine that:
> (1) adds that aux directory to the TEXINPUTS environment variable
> (2) writes a mock book.tex file that uses includeonly for that particular
> chapter
> (3) runs latex on that book.tex file.
>
> Say the book takes 5 minutes to compile with bibtexing, makeindex, etc.
> The chapters takes only a few seconds, so at the end (5 minutes and some
> seconds), you've got the book pdf and all the chapter-level pdfs finished.
>
> If you don't need chapter-level pdfs, this method doesn't buy you anything
> since in the first step the entire book is compiled.
That's quite helpful. as that is exactly what I'm attempting to do.
thanks!
Theo Markettos
> Say the book takes 5 minutes to compile with bibtexing, makeindex, etc.
> The chapters takes only a few seconds, so at the end (5 minutes and some
> seconds), you've got the book pdf and all the chapter-level pdfs finished.
>
> If you don't need chapter-level pdfs, this method doesn't buy you anything
> since in the first step the entire book is compiled.
The reason I came up with the idea was more for drafting purposes.
You can always produce the final 'shipping' version by building on a single
machine in linear fashion (and making a cup of tea/leaving it for a
hour/whatever). But before you ship, you might want to do proof copies. As
you fix things, you'll want to respin the proofs on a regular basis which
takes time. So if the proofs have everything except references a page or
two out, that can probably be lived with if they build in 1 minute not 10
minutes.
Theo
Will Robertson
<theom...@chiark.greenend.org.uk> said:
> So if the proofs have everything except references a page or
> two out, that can probably be lived with if they build in 1 minute not 10
> minutes.
But isn't that exactly what \includeonly is for?
pdflatex "\includeonly{chapterone}\input{mydocument}"
Will
Tim Arnold
Hi,
I see your situation now. One other thing that you might find useful is to
set up a cron job to run the books during the night. Then each chapter build
during the day references the aux files created in the previous night's run.
If you're on windows, you can 'Add Scheduled Task' instead of using cron.
--Tim
Theo Markettos
If making local changes (fix a typo, resize a figure) then you can use
\includeonly. But you might make global changes. Like tweaking the
typeface, or a global macro, or the margins, or pagination or...
This might be of particular interest if the whole document has some
constraint (eg book must be smaller than XXX pages otherwise the publisher
can't bind it, or will cost more)
Theo