Bug#900749: [www.debian.org] webwml build takes too long

[Laura Arjona Reina]

Package: www.debian.org
Severity: normal
User: debia...@lists.debian.org
Usertags: script

Since we migrated the website from cvs to git, the build jobs take too long.
The bottleneck part is the 4webwml script

The logs are here:
https://www-master.debian.org/build-logs/webwml

I've looked at the last 6 build jobs using CVS and they were taking around 48min
- 1h10min.

Now with git, we had a couple of jobs that cannot be taken as reference because
they were building almost the whole website, but the last one is building
respect these git changes:

From https://salsa.debian.org/webmaster-team/webwml
156e62e0ff8..d5c74615a1a master -> origin/master
Updating 156e62e0ff8..d5c74615a1a
Fast-forward
Makefile.common | 9 +-
chinese/Bugs/Reporting.wml | 13 +-
chinese/index.wml | 2 +-
chinese/ports/amd64/index.wml | 10 +-
chinese/security/2018/dsa-4205.wml | 2 +-
dutch/News/2018/20180601.wml | 4 +-
dutch/po/templates.nl.po | 17 ++-
english/devel/website/desc.wml | 10 +-
english/devel/website/examples.wml | 38 ++---
english/devel/website/htmlediting.wml | 4 +-
english/devel/website/index.wml | 1 +
english/devel/website/using_git.wml | 193 ++++++++++++++++++++++++++
english/devel/website/working.wml | 12 +-
english/ports/amd64/index.wml | 7 +-
english/security/2018/dsa-4216.data | 13 ++
english/security/2018/dsa-4216.wml | 27 ++++
english/security/2018/dsa-4217.data | 13 ++
english/security/2018/dsa-4217.wml | 23 +++
english/template/debian/translation-check.wml | 122 ++++++++--------
french/security/2018/dsa-4215.wml | 26 ++++
french/security/2018/dsa-4216.wml | 30 ++++
korean/News/2018/20180601.wml | 69 +++++++++
korean/devel/website/using_git.wml | 149 ++++++++++++++++++++
russian/News/2018/20180601.wml | 69 +++++++++
russian/security/2018/dsa-4214.wml | 28 ++++
russian/security/2018/dsa-4215.wml | 25 ++++
spanish/security/2018/dsa-4201.wml | 48 +++++++
spanish/security/2018/dsa-4203.wml | 31 +++++
swedish/ports/amd64/index.wml | 6 +-
swedish/security/2018/dsa-4216.wml | 27 ++++
swedish/security/2018/dsa-4217.wml | 23 +++
swedish/security/copyadvisory.pl | 2 +
32 files changed, 929 insertions(+), 124 deletions(-)
create mode 100644 english/devel/website/using_git.wml
create mode 100644 english/security/2018/dsa-4216.data
create mode 100644 english/security/2018/dsa-4216.wml
create mode 100644 english/security/2018/dsa-4217.data
create mode 100644 english/security/2018/dsa-4217.wml
create mode 100644 french/security/2018/dsa-4215.wml
create mode 100644 french/security/2018/dsa-4216.wml
create mode 100644 korean/News/2018/20180601.wml
create mode 100644 korean/devel/website/using_git.wml
create mode 100644 russian/News/2018/20180601.wml
create mode 100644 russian/security/2018/dsa-4214.wml
create mode 100644 russian/security/2018/dsa-4215.wml
create mode 100644 spanish/security/2018/dsa-4201.wml
create mode 100644 spanish/security/2018/dsa-4203.wml
create mode 100644 swedish/security/2018/dsa-4216.wml
create mode 100644 swedish/security/2018/dsa-4217.wml
0.66user 0.47system 0:08.20elapsed 13%CPU (0avgtext+0avgdata 63036maxresident)k
114840inputs+14376outputs (406major+19309minor)pagefaults 0swaps

And it's taking more than 13h

Maybe it's because of the change in Makefile.common, but I'm filing the bug for
the case the situation does not improve in the next builds, and to see if we can
improve the "make" recipes for the release points and massive changes.

Cheers

--
Laura Arjona Reina
https://wiki.debian.org/LauraArjona

[Laura Arjona Reina]

Hello all

I'm providing here some info about the current situation.

The last build job was done with no updates in the git repo.
This time the webwml job took 1 hour approx:

3stattrans finished (at Tue Jun 5 07:27:30 UTC 2018)
run-parts: executing /srv/www.debian.org/cron/parts/4webwml
4webwml finished (at Tue Jun 5 08:29:51 UTC 2018)

The logs of this job can be found in:

https://www-master.debian.org/build-logs/webwml/often.no-changes-git.log
https://www-master.debian.org/build-logs/webwml/git_update.no-changes-git.log
https://www-master.debian.org/build-logs/webwml/stattrans.no-changes-git.log
https://www-master.debian.org/build-logs/webwml/wml_run.no-changes-git.log

So I think this is the "bare minimum" that we can aim without changes to the
makefiles or scripts.

[Laura Arjona Reina]

Package: www.debian.org
Severity: normal
User: debia...@lists.debian.org
Usertags: script

We need to throw out /english/mirror/Mirrors.masterlist and update it from https://salsa.debian.org/mirror-team/masterlist at build time.

Filing a bug to not forget (and maybe somebody wants to submit a patch?)

Cheers
--
Laura Arjona Reina
https://wiki.debian.org/LauraArjona

Sent with K-9 mail

[Steve McIntyre]

(Sharing here for posterity...)

And I pushed more changes to the scripts and Makefiles last night to
add a shared on-disk cache of git revisions that should make a big
improvement for performance - see commits

533db96be6c Major Makefile overhauls
f035d5bf508 Move locking into the cache build utility
c9d8b4e7e2e Merge branch 'cache-files'
06a8db57de5 Tell git to ignore our on-disk cache
45fc485b91c Add a simple wrapper script to (re)generate the on-disk cache
da40997da40 Update test wrapper to test saving the cache to disk
3912e3865cb Add support for a simple on-disk cache of git revs

There's now a new order-only prerequisite on "$(VCSREVCACHE)" that
I've added in a lot of places in Makefiles to check/regenerate the
cache when running make rules to generate html files from wml
source. That seems to work well locally for me; please report any
problems you see.

I also updated ../cron/4webwml to explicitly check/build the cache
before we start running make, as that will be slightly faster.

At this point, I *think* I'm done with sciprt updates for the git
conversion itself, although obviously I'll investigate bugs as people
find/report them. :-)

There are a few other things I'm going to look at in the build scripts
- more on that separately...

--
Steve McIntyre, Cambridge, UK. st...@einval.com
"C++ ate my sanity" -- Jon Rabone

[Laura Arjona Reina]

Hi
The situation has improved a lot with the VCS cache and changes in the
makefiles (thanks Steve!).

Here are logs of a recent build after a change in the footer (hence,
rebuilding most of the pages):

https://www-master.debian.org/build-logs/webwml/often.change_in_footer.log

4webwml took 8h 30 min approx. Here is the detailed log:

https://www-master.debian.org/build-logs/webwml/wml_run.change_in_footer.log

(with CVS such builds took more than 24h).

While I know there is still room for improvement, I think the current
status of things is acceptable, so I propose to close the bug.