On Mon, Jul 26, 2010 at 10:06 PM, Brian Harring <ferr...@gmail.com> wrote: > Hola. This posting caught my eye- namely since this obviously is an > issue we have in gentoo, and there have been some proposals for how to > solve it but nothing has been implemented. > > If I may ask, is there a reason you didn't look at an EAPI with > (essentially) ABI slots? Such a route, while it requires specifying a > bit more info (or tweaking the manager to generate that info), would > give fine grained control to the resolver for doing rebuilds of the > sort mentioned above.
Long term, I think that that's exactly what we want. Right now, it's hard to tell which packages require dependencies to be rebuilt, and which packages do not have this requirement. If there were an automated way to tell which packages need their dependencies to be rebuilt, that would be great.
The ABI slot solution sounds good, as long as we don't have too many packages where the maintainers specify their rebuild requirements incorrectly (e.g., by breaking binary compatibility without realizing it). If we found that this was a regular problem, we would probably want to go back to just rebuilding all dependencies whenever a package changes.
> Aside from that, I'd be very interested if there is an ML, irc > channel, or any area people are doing discussions of this sort- I'm > the author of pkgcore, long term hacker of portage > (refactoring/cleanup- trying to combat the mess rather than creating > it), and have been working on issues similar to what y'all have been > rolling out. What's been a bit problematic is that there isn't > exactly a lot of people doing the level of binpkg work y'all are- thus > finding stakeholders to bug for info has been pretty sparse.
Personally, I'd be quite interested in this problem. The chromium-os-dev list is fine for this type of discussion, but if you have another mailing list that would be better, I'd be happy to join.
> At least from the pkgcore angle, I'd very much like to be tracking > what y'alls reqs are, hacks you're using to work around the manager, > etc, so I can either address it in my own manager or at least plan for > addressing it down the line- whether it be from the PM implementation > angle, or from the standpoint of trying to address y'alls needs in a > new EAPI.
Here are a few other suggestions about things to address in regular emerge: 1. To address package rebuilds completely correctly, we'll also need to differentiate between host and target build dependencies. There's a patch for this, but it looks like it got closed as a duplicate: http://bugs.gentoo.org/show_bug.cgi?id=317337 2. Strangely, Portage often favors the dependencies (and masking) of built packages over the ebuilds you have in your package dir. I would rather that Portage look at the ebuilds, because developers often change the ebuilds and want their new ebuild to be prioritized over the one that was already built. In parallel_emerge, we work around this by setting --usepkg=n for dependency calculation and then convert the source package to a binary package at a later stage. 3. Does Portage have public APIs for calculating dependencies and merging packages? Right now, parallel_emerge uses private APIs but I would like to convert it to public APIs if they exist. 4. Portage is a bit aggressive about locking directories, so this makes parallel merging slow. Whenever a package is being merged, Portage locks the entire package directory. This means essentially that package merges do not occur in parallel at all. Is it possible to reduce the scope of the locking so that it does not have such a significant effect on performance? 5. Some packages will make global system changes in their postinstall scripts, but these scripts are not parallel-safe. Can we make these scripts parallel-safe? Currently, this is not a big problem for emerge because it just locks for the entire merge, including postinstall (see #4), but once we get rid of that lock it becomes a problem.