release notes building time ...
Gerfried Fuchs
Again I feel the need to raise my voice about the building time of the
release notes. I just had to kill it once again because it was running
_way_ too long. I am thinking of moving the release note building from
the often cronjob (every four hours) to once a day to avoid not being
able to not have the rest of the site delayed because of a still running
build.
Sorry for the inconvenience, and if you are able to work on optimizing
the building of the release notes please do so now, I don't really see
much gain in having it hold up everything else.
Thanks for your attention,
Rhonda
W. Martin Borgert
> Again I feel the need to raise my voice about the building time of the
> release notes. I just had to kill it once again because it was running
> _way_ too long. I am thinking of moving the release note building from
> the often cronjob (every four hours) to once a day to avoid not being
> able to not have the rest of the site delayed because of a still running
> build.
My impression is, that the time is mostly spend in po4a-translate.
AFAIK, the release-notes are not the only ones using po4a. So I
wonder: What are we doing wrong, that the release notes take so
long? (Maybe my impression is wrong and po4a is not the "culprit",
I can't do any tests before 20:00 UTC.)
--
To UNSUBSCRIBE, email to debian-i1...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Nicolas François
On Tue, Feb 10, 2009 at 01:21:35PM +0100, W. Martin Borgert wrote:
>
> My impression is, that the time is mostly spend in po4a-translate.
> AFAIK, the release-notes are not the only ones using po4a. So I
> wonder: What are we doing wrong, that the release notes take so
> long? (Maybe my impression is wrong and po4a is not the "culprit",
> I can't do any tests before 20:00 UTC.)
I recently improved the performance of the Xml po4a module.
I also think it was too slow. (even more when a lot of tags were added in
the Docbook module)
This does not really provide a solution, but it might point the culprit.
I think using po4a with a config file (instead of po4a-updatepo and
po4a-translate) is faster.
I don't know if the Makefile has the right dependencies to avoid
rebuilding every time if no changes appeared in the English version and in
a given PO file (I don't even know if this situation happens).
Best Regards,
--
Nekral
W. Martin Borgert
> Again I feel the need to raise my voice about the building time of the
> release notes. I just had to kill it once again because it was running
> _way_ too long. I am thinking of moving the release note building from
> the often cronjob (every four hours) to once a day to avoid not being
> able to not have the rest of the site delayed because of a still running
> build.
Btw: Is there anything between every 4 hours and every 24 hours?
How about e.g. every 12 hours?
Gerfried Fuchs
* W. Martin Borgert <deb...@debian.org> [2009-02-10 13:21:35 CET]:
> Zitat von "Gerfried Fuchs" <rho...@deb.at>:
> > Again I feel the need to raise my voice about the building time of the
> >release notes. I just had to kill it once again because it was running
> >_way_ too long. I am thinking of moving the release note building from
> >the often cronjob (every four hours) to once a day to avoid not being
> >able to not have the rest of the site delayed because of a still running
> >build.
Alright, this is getting ridiculous. Did anyone actually try the build
on an etch system before it was dumped on us? What times did it take? It
is now building since over 8 hours and still not finished.
> My impression is, that the time is mostly spend in po4a-translate.
My impression is that the time is currently spent in trying to get
responses from external servers:
#v+
debwww@klecker:~$ ps -u debwww fu
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
[....]
debwww 5415 0.0 0.0 4468 2136 ? S 07:08 0:00 xsltproc --xinclude -o /tmp/tmpkekuRz/listings.xml --param current.dir '/org/www.debian.org/release-notes/len
debwww@klecker:~$ strace -p 5415
Process 5415 attached - interrupt to quit
select(7, [6], NULL, NULL, {21, 781777}^C <unfinished ...>
Process 5415 detached
debwww@klecker:~$ cd /proc/5415/fd
debwww@klecker:/proc/5415/fd$ ls -lh 6
lrwx------ 1 debwww debwww 64 2009-02-11 07:10 6 -> socket:[19255815]
debwww@klecker:/proc/5415/fd$ lsof -n | grep 19255815
xsltproc 5415 debwww 6u IPv4 19255815 TCP 194.109.137.218:40450->66.151.234.59:www (ESTABLISHED)
debwww@klecker:/proc/5415/fd$
#v-
I've noticed the same thing several times, and it hangs in that state
for quite a bit. The question is: Why does it (need to) query external
websites?
> AFAIK, the release-notes are not the only ones using po4a. So I
> wonder: What are we doing wrong, that the release notes take so
> long? (Maybe my impression is wrong and po4a is not the "culprit",
> I can't do any tests before 20:00 UTC.)
I would like to know wether someone really tried it and what their
timing for stuff is. And why the build (has to) query external websites.
* W. Martin Borgert <deb...@debian.org> [2009-02-10 21:00:35 CET]:
> Btw: Is there anything between every 4 hours and every 24 hours?
> How about e.g. every 12 hours?
I don't think that this will improve the situation, at all, sorry, and
that there is a real need for it. The build process does look too
problematic to me.
I don't kill the current running process to actually see how long it
will really need - but I fear that we will have to switch to manual
triggerd building instead of having it in cron at all. The current
situation is far from pleasing, to say the least.
So long,
Rhonda
Raphael Hertzog
> I've noticed the same thing several times, and it hangs in that state
> for quite a bit. The question is: Why does it (need to) query external
> websites?
It's probably downloading DTD. xsltproc has this option:
--nonet
Do not use the Internet to fetch DTDs, entities or documents.
Having DTD installed locally and the proper header should also avoid the need
for the network IIRC.
Cheers,
--
Raphaël Hertzog
Le best-seller français mis à jour pour Debian Etch :
http://www.ouaza.com/livre/admin-debian/
Luk Claes
> On Wed, 11 Feb 2009, Gerfried Fuchs wrote:
>> I've noticed the same thing several times, and it hangs in that state
>> for quite a bit. The question is: Why does it (need to) query external
>> websites?
>
> It's probably downloading DTD. xsltproc has this option:
> --nonet
> Do not use the Internet to fetch DTDs, entities or documents.
>
> Having DTD installed locally and the proper header should also avoid the need
> for the network IIRC.
Please request debian...@l.d.o the right package to be installed, TIA.
Cheers
Luk
Gerfried Fuchs
> Raphael Hertzog wrote:
> > On Wed, 11 Feb 2009, Gerfried Fuchs wrote:
> >> I've noticed the same thing several times, and it hangs in that state
> >> for quite a bit. The question is: Why does it (need to) query external
> >> websites?
> >
> > It's probably downloading DTD. xsltproc has this option:
> > --nonet
> > Do not use the Internet to fetch DTDs, entities or documents.
> >
> > Having DTD installed locally and the proper header should also avoid the need
> > for the network IIRC.
>
> Please request debian...@l.d.o the right package to be installed, TIA.
To be honest, in addition to that I would very much like to have the
--nonet option to be used. Just to be on the save side, I have no idea
if all the DTDs are packaged, and even if they are the option should be
used as safeguard, IMHO. Or wouldn't it use the local DTDs then neither?
So long,
Rhonda
Jens Seidel
> * Luk Claes <l...@debian.org> [2009-02-11 08:36:26 CET]:
> > Raphael Hertzog wrote:
> > > It's probably downloading DTD. xsltproc has this option:
> > > --nonet
> > > Do not use the Internet to fetch DTDs, entities or documents.
> To be honest, in addition to that I would very much like to have the
> --nonet option to be used. Just to be on the save side, I have no idea
> if all the DTDs are packaged, and even if they are the option should be
> used as safeguard, IMHO. Or wouldn't it use the local DTDs then neither?
--nonet is already used in Makefile ...
Jens
W. Martin Borgert
> Alright, this is getting ridiculous. Did anyone actually try the build
> on an etch system before it was dumped on us? What times did it take? It
> is now building since over 8 hours and still not finished.
> My impression is that the time is currently spent in trying to get
> responses from external servers:
Just for explanation:
1. No external nor internal DTDs are ever needed to build the
lenny release notes. (DTDs are needed, if you want to
validate XML, but not for compilation.) No need to install
docbook-xml, no harm either.
Locally, via the XML catalog system, are accessed the XSL
stylesheets of the docbook-xsl package. Is this package
installed (standard etch version, no backport required).
2. xsltproc is always called with the --nonet option in the
lenny release notes Makefile and always has been since we
started to use xsltproc. It shows an error message
(IO error), but one can - to my knowledge - ignore that error
message safely.
3. Unfortunately, I forgot that dblatex internally also calls
xsltproc. So I now pass this option also to dblatex. Now I
discovered, that there is a bug in dblatex: In one case it
"forgets" to pass the options to xsltproc. Patch attached
for review.
4. I did not have problems building the release notes on etch,
because I built with network.
HTH.
W. Martin Borgert
On 2009-02-11 08:22, Gerfried Fuchs wrote:
> Alright, this is getting ridiculous. Did anyone actually try the build
> on an etch system before it was dumped on us? What times did it take? It
> is now building since over 8 hours and still not finished.
...
> My impression is that the time is currently spent in trying to get
> responses from external servers:
Just for explanation:
Matt Kraai
> Locally, via the XML catalog system, are accessed the XSL
> stylesheets of the docbook-xsl package. Is this package
> installed (standard etch version, no backport required).
Yes.
--
Matt http://ftbfs.org/
Nicolas François
On Thu, Feb 12, 2009 at 06:45:28PM +0000, W. Martin Borgert wrote:
>
> I think, we can/should optimize po4a. It was already slow (with
> slow = "po4a-translate takes a significant part of the
> compilation time"), but got even a little bit slower in lenny.
That's not something that should be changed now, but after Lenny is
released, you should try copying /usr/share/perl5/Locale/Po4a/Xml.pm to
lib/Locale/Po4a/Xml.pm and applying
https://alioth.debian.org/plugins/scmcvs/cvsweb.php/po4a/lib/Locale/Po4a/Xml.pm.diff?r1=1.84;r2=1.85;cvsroot=po4a;f=h
This improved a lot (x10?) the speed of the Xml and derivative modules in
the upstream tree.
It would be preferable to get the Docbook.pm and Wml.pm from the same
po4a version.
Best Regards,
--
Nekral
--
To UNSUBSCRIBE, email to debian-rele...@lists.debian.org
W. Martin Borgert
> That's not something that should be changed now, but after Lenny is
> released, you should try
That sounds awesome. I'll try that after release. Thanks!