Google Groups

Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds)


David James Jul 27, 2010 6:05 AM
Posted in group: Chromium OS dev
+chromiumos-dev

On Mon, Jul 26, 2010 at 10:06 PM, Brian Harring <ferr...@gmail.com> wrote:
> Hola.  This posting caught my eye- namely since this obviously is an
> issue we have in gentoo, and there have been some proposals for how to
> solve it but nothing has been implemented.
>
> If I may ask, is there a reason you didn't look at an EAPI with
> (essentially) ABI slots?  Such a route, while it requires specifying a
> bit more info (or tweaking the manager to generate that info), would
> give fine grained control to the resolver for doing rebuilds of the
> sort mentioned above.

Long term, I think that that's exactly what we want. Right now, it's
hard to tell which packages require dependencies to be rebuilt, and
which packages do not have this requirement. If there were an
automated way to tell which packages need their dependencies to be
rebuilt, that would be great.

The ABI slot solution sounds good, as long as we don't have too many
packages where the maintainers specify their rebuild requirements
incorrectly (e.g., by breaking binary compatibility without realizing
it). If we found that this was a regular problem, we would probably
want to go back to just rebuilding all dependencies whenever a package
changes.

> Aside from that, I'd be very interested if there is an ML, irc
> channel, or any area people are doing discussions of this sort- I'm
> the author of pkgcore, long term hacker of portage
> (refactoring/cleanup- trying to combat the mess rather than creating
> it), and have been working on issues similar to what y'all have been
> rolling out.  What's been a bit problematic is that there isn't
> exactly a lot of people doing the level of binpkg work y'all are- thus
> finding stakeholders to bug for info has been pretty sparse.

Personally, I'd be quite interested in this problem. The
chromium-os-dev list is fine for this type of discussion, but if you
have another mailing list that would be better, I'd be happy to join.

> At least from the pkgcore angle, I'd very much like to be tracking
> what y'alls reqs are, hacks you're using to work around the manager,
> etc, so I can either address it in my own manager or at least plan for
> addressing it down the line- whether it be from the PM implementation
> angle, or from the standpoint of trying to address y'alls needs in a
> new EAPI.

Here are a few other suggestions about things to address in regular emerge:
 1. To address package rebuilds completely correctly, we'll also need
to differentiate between host and target build dependencies. There's a
patch for this, but it looks like it got closed as a duplicate:
  http://bugs.gentoo.org/show_bug.cgi?id=317337
 2. Strangely, Portage often favors the dependencies (and masking) of
built packages over the ebuilds you have in your package dir. I would
rather that Portage look at the ebuilds, because developers often
change the ebuilds and want their new ebuild to be prioritized over
the one that was already built. In parallel_emerge, we work around
this by setting --usepkg=n for dependency calculation and then convert
the source package to a binary package at a later stage.
 3. Does Portage have public APIs for calculating dependencies and
merging packages? Right now, parallel_emerge uses private APIs but I
would like to convert it to public APIs if they exist.
 4. Portage is a bit aggressive about locking directories, so this
makes parallel merging slow. Whenever a package is being merged,
Portage locks the entire package directory. This means essentially
that package merges do not occur in parallel at all. Is it possible to
reduce the scope of the locking so that it does not have such a
significant effect on performance?
 5. Some packages will make global system changes in their postinstall
scripts, but these scripts are not parallel-safe. Can we make these
scripts parallel-safe? Currently, this is not a big problem for emerge
because it just locks for the entire merge, including postinstall (see
#4), but once we get rid of that lock it becomes a problem.

Cheers,

David