Performance issue?

[at]

So, I managed to install a render server

I've tested it with a single page, and the pdf renders and downloads
just fine.

Then I try to add a category containing 55 articles.

The render page shows up, hovering at 1% and doesn't seem to ever
finish.

After a while the render server seems to suffer also, and becomes
slower than frozen molasses.

Any hints on what to look at? Do I need better server hardware?

[at]

It took an entire hour to finish the process, but all that was
included in the pdf was two pages. Something is very wrong here.

[Johannes Beigel]

Am 30.09.2008 um 19:02 schrieb at:
> It took an entire hour to finish the process, but all that was
> included in the pdf was two pages. Something is very wrong here.

What's the log output from mw-render? This logfile should be named "mw-
render.log.rl" and can be found inside the directory named after the
collection ID (as seen in the URL or in the log from mw-serve) inside
your mw-serve --cache-dir.

If there is a file "collection.zip" in this collection directory, what
happens if you call "mw-render -w rl -o test.pdf -c collection.zip"?

Regards,
Johannes Beigel

[at]

It seems that the errors start at "FETCHING TEMPLATES",

Then I start getting things like

mwlib.utils.error >> <urlopen error timed out> - while fetching

and

mwapidb.warn >> Long request: HTTP request took 10.028341 s

and

mwapidb.error >> Got no data from api.php


extracting the zip gets the same result: an empty pdf, pretty much.

[Johannes Beigel]

Am 13.10.2008 um 07:52 schrieb at:
> It seems that the errors start at "FETCHING TEMPLATES",
>
> Then I start getting things like
>
> mwlib.utils.error >> <urlopen error timed out> - while fetching
>
> and
>
> mwapidb.warn >> Long request: HTTP request took 10.028341 s

What wiki are you using? It seems to be overloaded.

You can try to minimize the load caused by mwlib by using the option --
no-threads (or a lower number for --num-threads) for mw-zip and mw-
render. But be prepared that the ZIP file generation will take more
time in this case.

Note, that we're already trying to fetch some resources three times in
a row because we don't want to give up too early because of failing
HTTP requests.

> extracting the zip gets the same result: an empty pdf, pretty much.

One could argue that the program should return with an error in these
cases, but sometimes we are in a situation (e.g. with broken links in
articles) that *some* resources can't be fetched, but the majority of
resouces can, so we do want to produce a ZIP file. In your case, where
essentially nothing could be fetched, this will result in an (almost)
empty ZIP file.

Regards,
Johannes

P.S.: If this happended to you, while you tried fetching articles from
the Wikipedia in the near past, see Brion Vibber's blog post http://leuksman.com/log/2008/10/12/database-borkage/