[pkg-discuss] Code review for usage of pkgdepend resolve -e in X gate
Alan Coopersmith
feature to provide a limited list of packages from the reference image?
https://cr.opensolaris.org/action/browse/x-cons/alanc/7155045/7155045/
This transforms the developer/opensolaris/X package manifest into a simple
package list using pkgmogrify and then passes it as the -e flag to
pkgdepend resolve.
Besides the performance improvement, it also caught that I'd missed adding
the gnutls package to our developer package list when we upgraded to the
TigerVNC release that added encryption support via gnutls.
As for performance, I ran the X gate "make_release_packages" script on several
hosts. This script completely removes the existing package repo in the gate,
runs make clean in the pkg directory, and then does a full package publication,
pkglint, and validate_pkg run across all 184 live packages, and 127 legacy
packages in the X gate.
- On my Ultra 27, with 1369 packages installed, this cut the average runtime
down from 5 minutes 15 seconds to 3:01, and average user CPU time from 656
seconds to 522 seconds.
- On our Sun Fire X4100 build server, with 684 packages installed, this cut
the average measured user cpu time down from 717 seconds to 673 seconds,
and the average wall clock time from an average of 11 minutes to 10 minutes
(though another build was running on the machine at a time).
- On our SPARC T2 build server, I gave up on averaging, after a single run to
establish the "before" time took 56 minutes on an otherwise idle machine.
The "after" time was reduced to 44:40 though. ("pkg list | wc" = 798)
BTW, since the X pkg/Makefile is based on ON's, it was easy to make the
equivalent change to it, but it's developer package needed a *lot* more
change:
http://kodiak.us.oracle.com/builds/alanc/on11u1/pkgdepend-e/
On a handy T4 build server (2 cpus x 8 cores x 8 threads, 617 pkgs installed),
I ran 5 times each, before and after, on a fully built ON gate:
rm -rf $PKGARCHIVE ; dmake clean ; time dmake -j 16 install > make.out
Average before: 450.76 user cpu time ; 3 minutes, 33 seconds wall clock
Average after: 401.15 user cpu time ; 2 minutes, 42 seconds wall clock
But unlike X, ON is not yet ready to use the S11U1 pkgdepend, so they'll have
to wait for it. (Did I hear that someone is already working on cleaning up
ON's missing developer depends? If so, I'll happily hand this off to them
to follow up with.)
--
-Alan Coopersmith- alan.coo...@oracle.com
Oracle Solaris Engineering - http://blogs.oracle.com/alanc
_______________________________________________
pkg-discuss mailing list
pkg-d...@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
Alan Coopersmith
> I'm no makefile expert, but the use of -e seems to be right to me. The only
> thing that I spent awhile thinking about was whether using the developer package
> as the source of external dependencies was the right thing to do. I'd be a
> little curious whether pkgdepend -E -e reported any unused external
> dependencies. For example, while developer/gcc-3 is probably needed to build X,
> I'd be surprised if it's needed when using X. Similar thoughts with mdb, make,
> onbld .... You have made me wonder if the developer package should be a superset
> of the external dependencies though, and whether perhaps we should leverage that
> in the IPS gate. While I can imagine a package that might be needed to run a
> program, but not build it, if a developer's testing at all, it seems like they'd
> need at least all the external dependencies.
Right - it seemed reasonable to me as the runtime dependencies have to be a
subset of the build time dependencies for pkgdepend to be able to resolve them.
I had thought about adding a local attribute to the dependency file to be able
to filter those out in pkgmogrify, but for X that seemed like overkill, since
the win is reducing from hundreds of packages down to 62, and taking it down to
what I figured would be 50-something didn't seem worth the additional effort and
ongoing maintenance.
To check with -E, I had to remove build-i386/developer-opensolaris-X.dep from
the list of packages to resolve, or it found them all in there and reported
none were extra. That actually reported 44 of the 62 were excess to
requirements, far more than I expected, so I went ahead and added the
attribute and filtering to it. Revised webrev:
https://cr.opensolaris.org/action/browse/x-cons/alanc/7155045/webrev-v2/
> Just to double check, the performance improvements you listed were for the
> entire publication process, not just the change in the time to do pkgdep
> resolve, correct?
Yes - they include the pkglint, pkgmogrify, pkgsend, etc. steps.
For just the pkgdepend resolve step, on my Ultra 27 the difference was:
Without -e (resolving against 1369 pkgs): 144.95u 2.12s 2:27.36 99.8%
First attempt at -e (66 pkgs): 10.56u 0.18s 0:10.83 99.1%
Revised attempt at -e (18 pkgs): 7.24u 0.14s 0:07.48 98.6%
But the real test is on our T2 build server, where the numbers from
just pkgdepend resolve were:
Without -e (resolving against 798 pkgs): 669.62u 10.06s 11:22.11 99.6%
First attempt at -e (66 pkgs): 117.16u 1.66s 2:02.55 96.9%
Revised attempt at -e (18 pkgs): 81.71u 0.82s 1:22.73 99.7%