On Mon, Jul 26, 2010 at 10:06 PM, Brian Harring <ferri...@gmail.com> wrote: > Hola. This posting caught my eye- namely since this obviously is an > issue we have in gentoo, and there have been some proposals for how to > solve it but nothing has been implemented.
> If I may ask, is there a reason you didn't look at an EAPI with > (essentially) ABI slots? Such a route, while it requires specifying a > bit more info (or tweaking the manager to generate that info), would > give fine grained control to the resolver for doing rebuilds of the > sort mentioned above.
Long term, I think that that's exactly what we want. Right now, it's hard to tell which packages require dependencies to be rebuilt, and which packages do not have this requirement. If there were an automated way to tell which packages need their dependencies to be rebuilt, that would be great.
The ABI slot solution sounds good, as long as we don't have too many packages where the maintainers specify their rebuild requirements incorrectly (e.g., by breaking binary compatibility without realizing it). If we found that this was a regular problem, we would probably want to go back to just rebuilding all dependencies whenever a package changes.
> Aside from that, I'd be very interested if there is an ML, irc > channel, or any area people are doing discussions of this sort- I'm > the author of pkgcore, long term hacker of portage > (refactoring/cleanup- trying to combat the mess rather than creating > it), and have been working on issues similar to what y'all have been > rolling out. What's been a bit problematic is that there isn't > exactly a lot of people doing the level of binpkg work y'all are- thus > finding stakeholders to bug for info has been pretty sparse.
Personally, I'd be quite interested in this problem. The chromium-os-dev list is fine for this type of discussion, but if you have another mailing list that would be better, I'd be happy to join.
> At least from the pkgcore angle, I'd very much like to be tracking > what y'alls reqs are, hacks you're using to work around the manager, > etc, so I can either address it in my own manager or at least plan for > addressing it down the line- whether it be from the PM implementation > angle, or from the standpoint of trying to address y'alls needs in a > new EAPI.
Here are a few other suggestions about things to address in regular emerge: 1. To address package rebuilds completely correctly, we'll also need to differentiate between host and target build dependencies. There's a patch for this, but it looks like it got closed as a duplicate: http://bugs.gentoo.org/show_bug.cgi?id=317337 2. Strangely, Portage often favors the dependencies (and masking) of built packages over the ebuilds you have in your package dir. I would rather that Portage look at the ebuilds, because developers often change the ebuilds and want their new ebuild to be prioritized over the one that was already built. In parallel_emerge, we work around this by setting --usepkg=n for dependency calculation and then convert the source package to a binary package at a later stage. 3. Does Portage have public APIs for calculating dependencies and merging packages? Right now, parallel_emerge uses private APIs but I would like to convert it to public APIs if they exist. 4. Portage is a bit aggressive about locking directories, so this makes parallel merging slow. Whenever a package is being merged, Portage locks the entire package directory. This means essentially that package merges do not occur in parallel at all. Is it possible to reduce the scope of the locking so that it does not have such a significant effect on performance? 5. Some packages will make global system changes in their postinstall scripts, but these scripts are not parallel-safe. Can we make these scripts parallel-safe? Currently, this is not a big problem for emerge because it just locks for the entire merge, including postinstall (see #4), but once we get rid of that lock it becomes a problem.
On Tue, Jul 27, 2010 at 09:05:28AM -0400, David James wrote: > On Mon, Jul 26, 2010 at 10:06 PM, Brian Harring <ferri...@gmail.com> wrote: > > If I may ask, is there a reason you didn't look at an EAPI with > > (essentially) ABI slots? Such a route, while it requires specifying a > > bit more info (or tweaking the manager to generate that info), would > > give fine grained control to the resolver for doing rebuilds of the > > sort mentioned above.
> Long term, I think that that's exactly what we want. Right now, it's > hard to tell which packages require dependencies to be rebuilt, and > which packages do not have this requirement. If there were an > automated way to tell which packages need their dependencies to be > rebuilt, that would be great.
> The ABI slot solution sounds good, as long as we don't have too many > packages where the maintainers specify their rebuild requirements > incorrectly (e.g., by breaking binary compatibility without realizing > it). If we found that this was a regular problem, we would probably > want to go back to just rebuilding all dependencies whenever a package > changes.
Bad packaging metadata we can't really do much about realistically, beyond threaten to wedige the packaging dev- that said a DT_NEEDED validation QA check to ensure the targets are within stated rdeps might be useful.
Doesn't do anything for dlopen or static, but it was one of the more useful features of the old EBD/saviour branch of portage- iirc I named that feature 'verify-linkage'. Implementation was ldd based (ick), but it was a start.
> > Aside from that, I'd be very interested if there is an ML, irc > > channel, or any area people are doing discussions of this sort- I'm > > the author of pkgcore, long term hacker of portage > > (refactoring/cleanup- trying to combat the mess rather than creating > > it), and have been working on issues similar to what y'all have been > > rolling out. What's been a bit problematic is that there isn't > > exactly a lot of people doing the level of binpkg work y'all are- thus > > finding stakeholders to bug for info has been pretty sparse.
> Personally, I'd be quite interested in this problem. The > chromium-os-dev list is fine for this type of discussion, but if you > have another mailing list that would be better, I'd be happy to join.
Unfortunately I'm not particularly aware of any. Typically I track down people in irc and pick their brains directly; that said from an irc standpoint #gentoo-portage on freenode is probably the best spot to catch the appropriate people (zac medico/zmedico, sebastian luther/zac, solar, myself) who would have interest/comments.
> Here are a few other suggestions about things to address in regular emerge: > 1. To address package rebuilds completely correctly, we'll also need > to differentiate between host and target build dependencies. There's a > patch for this, but it looks like it got closed as a duplicate: > http://bugs.gentoo.org/show_bug.cgi?id=317337
This was proposed as "bdepends" at one point- build time depends, specifically what is executed during building the package (bash and sed for example).
The problem w/ a new dependency type is mainly getting developer buy in- building tools to help developers track/validate these dependencies. At least in the past, this sort of proposal died out since their wasn't a sufficient carrot (nor was it easy to modify portage for such a capability).
> 2. Strangely, Portage often favors the dependencies (and masking) of > built packages over the ebuilds you have in your package dir. I would > rather that Portage look at the ebuilds, because developers often > change the ebuilds and want their new ebuild to be prioritized over > the one that was already built. In parallel_emerge, we work around > this by setting --usepkg=n for dependency calculation and then convert > the source package to a binary package at a later stage.
Essentially you want binpkgs to be utilized strictly as a cache, right? I suspect this isn't a particularly hard mod to implement for the folk who know the innards of _emerge.* reasonably well.
> 3. Does Portage have public APIs for calculating dependencies and > merging packages? Right now, parallel_emerge uses private APIs but I > would like to convert it to public APIs if they exist.
CC'ing the folk who could best answer this one- at this point, there isn't perse a public api for the most part. I know they're currently trying to derive one for packagekit/porthole/SoC, but I'm not aware of it's status.
> 4. Portage is a bit aggressive about locking directories, so this > makes parallel merging slow. Whenever a package is being merged, > Portage locks the entire package directory. This means essentially > that package merges do not occur in parallel at all. Is it possible to > reduce the scope of the locking so that it does not have such a > significant effect on performance?
Heh. This is a contentious issue. Roughly, everything after pkg_setup when it comes to merging is expected to be the only thing modifying the FS during this- the reasoning is purely due to pkg_preinst/pkg_postinst being essentially uncontrolled access to the livefs. That critical locking unfortunately does need to exist if those phases are defined. I'd expect in addition that the treewalk bits (the merger) wouldn't particularly like having a run in with another active merge at the same time.
That said, the scope of the locking definitely should be reducable, and parallelization increased w/in the atomic locked merge. Locking of the binpkgs repository in particular seems unneeded since effectively it's an untarring to ${D}, and unpacking the xpak (that annoying trailing data after EOS on tbzs) to ${T} w/in the pkgs build env. After that that point, there shouldn't need to be any further locking on binpkgs- further creating a new binpkg doesn't require a write lock until you actually try to make it live w/in the binpkg repo (the final atomic rename), so at least the compression steps should be able to be done in parallel.
> 5. Some packages will make global system changes in their postinstall > scripts, but these scripts are not parallel-safe. Can we make these > scripts parallel-safe? Currently, this is not a big problem for emerge > because it just locks for the entire merge, including postinstall (see > #4), but once we get rid of that lock it becomes a problem.
I'm definitely open to suggestions of how to make them safe, but by pkg_(pre|post)(inst|rm)'s nature, they're free form root level execution. From a format standpoint, for better or worse they technically can do whatever the hell they want, bounded only by QA standards of that particularly repository.
I could see adding a PROPERTIES hint of some sort indicating that the locking isn't needed for that pkg offhand, although that's a bit of a hack.
> On Tue, Jul 27, 2010 at 09:05:28AM -0400, David James wrote: >> On Mon, Jul 26, 2010 at 10:06 PM, Brian Harring <ferri...@gmail.com> wrote: >>> If I may ask, is there a reason you didn't look at an EAPI with >>> (essentially) ABI slots? Such a route, while it requires specifying a >>> bit more info (or tweaking the manager to generate that info), would >>> give fine grained control to the resolver for doing rebuilds of the >>> sort mentioned above.
>> Long term, I think that that's exactly what we want. Right now, it's >> hard to tell which packages require dependencies to be rebuilt, and >> which packages do not have this requirement. If there were an >> automated way to tell which packages need their dependencies to be >> rebuilt, that would be great.
>> The ABI slot solution sounds good, as long as we don't have too many >> packages where the maintainers specify their rebuild requirements >> incorrectly (e.g., by breaking binary compatibility without realizing >> it). If we found that this was a regular problem, we would probably >> want to go back to just rebuilding all dependencies whenever a package >> changes.
> Bad packaging metadata we can't really do much about realistically, > beyond threaten to wedige the packaging dev- that said a DT_NEEDED > validation QA check to ensure the targets are within stated rdeps > might be useful.
> Doesn't do anything for dlopen or static, but it was one of the more > useful features of the old EBD/saviour branch of portage- iirc I named > that feature 'verify-linkage'. Implementation was ldd based (ick), > but it was a start.
A notable advantage of giving the resolver the metadata (ABI slots) to account for rebuilds in advance is that it allows for more informed decisions with respect to aggressive --jobs parallelization. Considering --fast chromium-os builds, this seems like an area of interest.
>>> Aside from that, I'd be very interested if there is an ML, irc >>> channel, or any area people are doing discussions of this sort- I'm >>> the author of pkgcore, long term hacker of portage >>> (refactoring/cleanup- trying to combat the mess rather than creating >>> it), and have been working on issues similar to what y'all have been >>> rolling out. What's been a bit problematic is that there isn't >>> exactly a lot of people doing the level of binpkg work y'all are- thus >>> finding stakeholders to bug for info has been pretty sparse.
>> Personally, I'd be quite interested in this problem. The >> chromium-os-dev list is fine for this type of discussion, but if you >> have another mailing list that would be better, I'd be happy to join.
> Unfortunately I'm not particularly aware of any. Typically I track > down people in irc and pick their brains directly; that said from an > irc standpoint #gentoo-portage on freenode is probably the best spot > to catch the appropriate people (zac medico/zmedico, sebastian > luther/zac, solar, myself) who would have interest/comments.
The gentoo-portage-...@gentoo.org list seems applicable.
>> Here are a few other suggestions about things to address in regular emerge: >> 1. To address package rebuilds completely correctly, we'll also need >> to differentiate between host and target build dependencies. There's a >> patch for this, but it looks like it got closed as a duplicate: >> http://bugs.gentoo.org/show_bug.cgi?id=317337
> This was proposed as "bdepends" at one point- build time depends, > specifically what is executed during building the package (bash and > sed for example).
> The problem w/ a new dependency type is mainly getting developer buy > in- building tools to help developers track/validate these > dependencies. At least in the past, this sort of proposal died out > since their wasn't a sufficient carrot (nor was it easy to modify > portage for such a capability).
Nowadays, it would be pretty trivial to add a BDEPEND variable or something like that. Adding a new *DEPEND variable like this would be a lot less invasive that trying to jump to a DEPENDENCIES variable that unifies *DEPEND.
>> 2. Strangely, Portage often favors the dependencies (and masking) of >> built packages over the ebuilds you have in your package dir. I would >> rather that Portage look at the ebuilds, because developers often >> change the ebuilds and want their new ebuild to be prioritized over >> the one that was already built. In parallel_emerge, we work around >> this by setting --usepkg=n for dependency calculation and then convert >> the source package to a binary package at a later stage.
> Essentially you want binpkgs to be utilized strictly as a cache, > right? I suspect this isn't a particularly hard mod to implement for > the folk who know the innards of _emerge.* reasonably well.
I think maybe he's talking about updating the *DEPEND metadata of the binary packages, so that they match the ebuilds. I have an old script that does it:
You could update the metadata of all binary packages with this command:
vdb-tools.py --transfer --pkgtype=binary
>> 3. Does Portage have public APIs for calculating dependencies and >> merging packages? Right now, parallel_emerge uses private APIs but I >> would like to convert it to public APIs if they exist.
> CC'ing the folk who could best answer this one- at this point, there > isn't perse a public api for the most part. I know they're currently > trying to derive one for packagekit/porthole/SoC, but I'm not aware of > it's status.
Brian Dolbeck (porthole developer) has been working on this, but I'm unsure of progress. I've CC'd him in case he wants to chime in.
>> 4. Portage is a bit aggressive about locking directories, so this >> makes parallel merging slow. Whenever a package is being merged, >> Portage locks the entire package directory. This means essentially >> that package merges do not occur in parallel at all. Is it possible to >> reduce the scope of the locking so that it does not have such a >> significant effect on performance?
> Heh. This is a contentious issue. Roughly, everything after > pkg_setup when it comes to merging is expected to be the only thing > modifying the FS during this- the reasoning is purely due to > pkg_preinst/pkg_postinst being essentially uncontrolled access to the > livefs. That critical locking unfortunately does need to exist if > those phases are defined. I'd expect in addition that the treewalk > bits (the merger) wouldn't particularly like having a run in with > another active merge at the same time.
> That said, the scope of the locking definitely should be reducable, > and parallelization increased w/in the atomic locked merge. Locking > of the binpkgs repository in particular seems unneeded since > effectively it's an untarring to ${D}, and unpacking the xpak (that > annoying trailing data after EOS on tbzs) to ${T} w/in the pkgs build > env. After that that point, there shouldn't need to be any further > locking on binpkgs- further creating a new binpkg doesn't require a > write lock until you actually try to make it live w/in the binpkg repo > (the final atomic rename), so at least the compression steps should be > able to be done in parallel.
Of course it can be done. Again, additional metadata for ABI slots will be quite helpful for aggressive --jobs parallelization like this.
>> 5. Some packages will make global system changes in their postinstall >> scripts, but these scripts are not parallel-safe. Can we make these >> scripts parallel-safe? Currently, this is not a big problem for emerge >> because it just locks for the entire merge, including postinstall (see >> #4), but once we get rid of that lock it becomes a problem.
> I'm definitely open to suggestions of how to make them safe, but by > pkg_(pre|post)(inst|rm)'s nature, they're free form root level > execution. From a format standpoint, for better or worse they > technically can do whatever the hell they want, bounded only by QA > standards of that particularly repository.
> I could see adding a PROPERTIES hint of some sort indicating that the > locking isn't needed for that pkg offhand, although that's a bit of a > hack.
A PROPERTIES or RESTRICT flag sounds pretty reasonable to me. This leads to the question of whether it should be opt-in or opt-out. We could conceivably make the opt-in/opt-out behavior conditional on EAPI, but maybe that won't be necessary.
We could also add helpers for locking/unlocking resources. This could be implemented in an eclass, but it might be nicer to had the helpers on the package manager side (in a new EAPI). -- Thanks, Zac
> Zac, > Here is a patch Tien-Ren came up with for HDEPEND. CC'ing him too. He > was planning on revising the patch. Just fyi.
> Thanks > Anush
> On Wed, Jul 28, 2010 at 6:46 PM, Zac Medico <zmed...@gentoo.org> wrote:
>> On 07/28/2010 04:22 AM, Brian Harring wrote: >> > On Tue, Jul 27, 2010 at 09:05:28AM -0400, David James wrote: >> >> On Mon, Jul 26, 2010 at 10:06 PM, Brian Harring <ferri...@gmail.com> >> wrote: >> >>> If I may ask, is there a reason you didn't look at an EAPI with >> >>> (essentially) ABI slots? Such a route, while it requires specifying a >> >>> bit more info (or tweaking the manager to generate that info), would >> >>> give fine grained control to the resolver for doing rebuilds of the >> >>> sort mentioned above.
>> >> Long term, I think that that's exactly what we want. Right now, it's >> >> hard to tell which packages require dependencies to be rebuilt, and >> >> which packages do not have this requirement. If there were an >> >> automated way to tell which packages need their dependencies to be >> >> rebuilt, that would be great.
>> >> The ABI slot solution sounds good, as long as we don't have too many >> >> packages where the maintainers specify their rebuild requirements >> >> incorrectly (e.g., by breaking binary compatibility without realizing >> >> it). If we found that this was a regular problem, we would probably >> >> want to go back to just rebuilding all dependencies whenever a package >> >> changes.
>> > Bad packaging metadata we can't really do much about realistically, >> > beyond threaten to wedige the packaging dev- that said a DT_NEEDED >> > validation QA check to ensure the targets are within stated rdeps >> > might be useful.
>> > Doesn't do anything for dlopen or static, but it was one of the more >> > useful features of the old EBD/saviour branch of portage- iirc I named >> > that feature 'verify-linkage'. Implementation was ldd based (ick), >> > but it was a start.
>> A notable advantage of giving the resolver the metadata (ABI slots) >> to account for rebuilds in advance is that it allows for more >> informed decisions with respect to aggressive --jobs >> parallelization. Considering --fast chromium-os builds, this seems >> like an area of interest.
>> >>> Aside from that, I'd be very interested if there is an ML, irc >> >>> channel, or any area people are doing discussions of this sort- I'm >> >>> the author of pkgcore, long term hacker of portage >> >>> (refactoring/cleanup- trying to combat the mess rather than creating >> >>> it), and have been working on issues similar to what y'all have been >> >>> rolling out. What's been a bit problematic is that there isn't >> >>> exactly a lot of people doing the level of binpkg work y'all are- thus >> >>> finding stakeholders to bug for info has been pretty sparse.
>> >> Personally, I'd be quite interested in this problem. The >> >> chromium-os-dev list is fine for this type of discussion, but if you >> >> have another mailing list that would be better, I'd be happy to join.
>> > Unfortunately I'm not particularly aware of any. Typically I track >> > down people in irc and pick their brains directly; that said from an >> > irc standpoint #gentoo-portage on freenode is probably the best spot >> > to catch the appropriate people (zac medico/zmedico, sebastian >> > luther/zac, solar, myself) who would have interest/comments.
>> The gentoo-portage-...@gentoo.org list seems applicable.
>> >> Here are a few other suggestions about things to address in regular >> emerge: >> >> 1. To address package rebuilds completely correctly, we'll also need >> >> to differentiate between host and target build dependencies. There's a >> >> patch for this, but it looks like it got closed as a duplicate: >> >> http://bugs.gentoo.org/show_bug.cgi?id=317337
>> > This was proposed as "bdepends" at one point- build time depends, >> > specifically what is executed during building the package (bash and >> > sed for example).
>> > The problem w/ a new dependency type is mainly getting developer buy >> > in- building tools to help developers track/validate these >> > dependencies. At least in the past, this sort of proposal died out >> > since their wasn't a sufficient carrot (nor was it easy to modify >> > portage for such a capability).
>> Nowadays, it would be pretty trivial to add a BDEPEND variable or >> something like that. Adding a new *DEPEND variable like this would >> be a lot less invasive that trying to jump to a DEPENDENCIES >> variable that unifies *DEPEND.
>> >> 2. Strangely, Portage often favors the dependencies (and masking) of >> >> built packages over the ebuilds you have in your package dir. I would >> >> rather that Portage look at the ebuilds, because developers often >> >> change the ebuilds and want their new ebuild to be prioritized over >> >> the one that was already built. In parallel_emerge, we work around >> >> this by setting --usepkg=n for dependency calculation and then convert >> >> the source package to a binary package at a later stage.
>> > Essentially you want binpkgs to be utilized strictly as a cache, >> > right? I suspect this isn't a particularly hard mod to implement for >> > the folk who know the innards of _emerge.* reasonably well.
>> I think maybe he's talking about updating the *DEPEND metadata of >> the binary packages, so that they match the ebuilds. I have an old >> script that does it:
>> You could update the metadata of all binary packages with this command:
>> vdb-tools.py --transfer --pkgtype=binary
>> >> 3. Does Portage have public APIs for calculating dependencies and >> >> merging packages? Right now, parallel_emerge uses private APIs but I >> >> would like to convert it to public APIs if they exist.
>> > CC'ing the folk who could best answer this one- at this point, there >> > isn't perse a public api for the most part. I know they're currently >> > trying to derive one for packagekit/porthole/SoC, but I'm not aware of >> > it's status.
>> Brian Dolbeck (porthole developer) has been working on this, but I'm >> unsure of progress. I've CC'd him in case he wants to chime in.
>> >> 4. Portage is a bit aggressive about locking directories, so this >> >> makes parallel merging slow. Whenever a package is being merged, >> >> Portage locks the entire package directory. This means essentially >> >> that package merges do not occur in parallel at all. Is it possible to >> >> reduce the scope of the locking so that it does not have such a >> >> significant effect on performance?
>> > Heh. This is a contentious issue. Roughly, everything after >> > pkg_setup when it comes to merging is expected to be the only thing >> > modifying the FS during this- the reasoning is purely due to >> > pkg_preinst/pkg_postinst being essentially uncontrolled access to the >> > livefs. That critical locking unfortunately does need to exist if >> > those phases are defined. I'd expect in addition that the treewalk >> > bits (the merger) wouldn't particularly like having a run in with >> > another active merge at the same time.
>> > That said, the scope of the locking definitely should be reducable, >> > and parallelization increased w/in the atomic locked merge. Locking >> > of the binpkgs repository in particular seems unneeded since >> > effectively it's an untarring to ${D}, and unpacking the xpak (that >> > annoying trailing data after EOS on tbzs) to ${T} w/in the pkgs build >> > env. After that that point, there shouldn't need to be any further >> > locking on binpkgs- further creating a new binpkg doesn't require a >> > write lock until you actually try to make it live w/in the binpkg repo >> > (the final atomic rename), so at least the compression steps should be >> > able to be done in parallel.
>> Of course it can be done. Again, additional metadata for ABI slots >> will be quite helpful for aggressive --jobs parallelization like this.
>> >> 5. Some packages will make global system changes in their postinstall >> >> scripts, but these scripts are not parallel-safe. Can we make these >> >> scripts parallel-safe? Currently, this is not a big problem for emerge >> >> because it just locks for the entire merge, including postinstall (see >> >> #4), but once we get rid of that lock it becomes a problem.
>> > I'm definitely open to suggestions of how to make them safe, but by >> > pkg_(pre|post)(inst|rm)'s nature, they're free form root level >> > execution. From a format standpoint, for better or worse they >> > technically can do whatever the hell they want, bounded only by QA >> > standards of that particularly repository.
>> > I could see adding a PROPERTIES hint of some sort indicating that the >> > locking isn't needed for that pkg offhand, although that's a bit of a >> > hack.
>> A PROPERTIES or RESTRICT flag sounds pretty reasonable to me. This >> leads to the question of whether it should be opt-in or opt-out. We >> could conceivably make the opt-in/opt-out behavior conditional on >> EAPI, but maybe that won't be necessary.
>> We could also add helpers for locking/unlocking resources. This >> could be implemented in an eclass, but it might be nicer to had the >> helpers on the package manager side (in a new EAPI). >> -- >> Thanks, >> Zac
The patch works much like the one given in #317337, and I think the one in #317337 looks more complete. however #317337 is marked as duplicate of #201499 [1]. In the #201499 thread they actually made a nice proposal to introduce a new dependency system that uses dependency labels [2]. I'm currently planning on implementing it since it looks to be the preferred draft EAPI.
What I'm concerning is that I don't have the confidence I'm able to hack portage to support the new system. The dependency handling code in portage uses complicated algorithm and is lack of document. I'm turning to paludis to see if it's easier to be modified to fit our purpose.
Tien, Zac the upstream portage maintainer and other Gentoo devs hang out on #chromium-os IRC and are very helpful if you have any questions on the internals of portage.
On Mon, Aug 2, 2010 at 7:18 PM, Tien-Ren Chen <trc...@chromium.org> wrote: > The patch works much like the one given in #317337, and I think the > one in #317337 looks more complete. > however #317337 is marked as duplicate of #201499 [1]. > In the #201499 thread they actually made a nice proposal to introduce > a new dependency system that uses dependency labels [2]. > I'm currently planning on implementing it since it looks to be the > preferred draft EAPI.
> What I'm concerning is that I don't have the confidence I'm able to > hack portage to support the new system. > The dependency handling code in portage uses complicated algorithm and > is lack of document. > I'm turning to paludis to see if it's easier to be modified to fit our > purpose.
> On Fri, Jul 30, 2010 at 3:36 PM, Anush Elangovan(அனுஷ்) > <an...@chromium.org> wrote: > > Resending from the correct email address to get to the mailing list.
> > On Fri, Jul 30, 2010 at 12:34 AM, Anush Elangovan(அனுஷ்) < > an...@google.com> > > wrote:
> >> Zac, > >> Here is a patch Tien-Ren came up with for HDEPEND. CC'ing him too. He > >> was planning on revising the patch. Just fyi. > >> Thanks > >> Anush
I was working on writing documentation for the HDEPEND patch when I realized that it seems simpler for ebuild developers if we divide the dependencies as follows :
In EAPI 4 or later, DEPEND has been divided into two parts: DEPEND for build-time host dependencies, and TDEPEND for build-time target dependencies. This division is designed specifically to minimize difficulty in the process of adapting ebuilds that were written for earlier EAPIs, and therefore it also minimizes the adjustments that ebuild developers will have to make to the thought processes involved when writing ebuilds from scratch. In an environment that does not involve cross-compilation, TDEPEND behaves the same as DEPEND. When an ebuild is converted from EAPI 3 or earlier to EAPI 4 or later, in order to support cross-compilation environments, some dependencies may need to be migrated to TDEPEND.
For ebuilds that have EAPI 4 or later, the emerge --root-deps option has no effect since it is made obsolete by division between DEPEND and TDEPEND. If EAPI 3 or earlier ebuilds are used in combination with EAPI 4 or later ebuilds, the --root-deps behavior will still be applied to the EAPI 3 or earlier ebuilds (there is no behavior change for ebuilds having older EAPIs).
The attached patch applies to the master branch of portage (apply on top of commit cc03d96314dc108964c9c1b0431a0466f20ad147 if it doesn't apply to HEAD). You can test this patch by creating ebuilds with EAPI=4_pre0_tdepend (this is a temporary EAPI, intended just for testing purposes). I've also attached this patch to bug #317337:
> The patch works much like the one given in #317337, and I think the > one in #317337 looks more complete. > however #317337 is marked as duplicate of #201499 [1]. > In the #201499 thread they actually made a nice proposal to introduce > a new dependency system that uses dependency labels [2]. > I'm currently planning on implementing it since it looks to be the > preferred draft EAPI.
> What I'm concerning is that I don't have the confidence I'm able to > hack portage to support the new system. > The dependency handling code in portage uses complicated algorithm and > is lack of document. > I'm turning to paludis to see if it's easier to be modified to fit our purpose.
> On Fri, Jul 30, 2010 at 3:36 PM, Anush Elangovan(அனுஷ்) > <an...@chromium.org> wrote: >> Resending from the correct email address to get to the mailing list.
>> On Fri, Jul 30, 2010 at 12:34 AM, Anush Elangovan(அனுஷ்) <an...@google.com> >> wrote:
>>> Zac, >>> Here is a patch Tien-Ren came up with for HDEPEND. CC'ing him too. He >>> was planning on revising the patch. Just fyi. >>> Thanks >>> Anush
Zac Medico (zmed...@gentoo.org) wrote: > I was working on writing documentation for the HDEPEND patch when I > realized that it seems simpler for ebuild developers if we divide the > dependencies as follows :
Thanks much for adding this support:) This will definitely make life much easier for folks who need to cross-compile. But I have one quick question.
Why would TDEPEND be simpler? I suspect that target build dependencies (header files and static libraries) are more common than host build dependencies (build tools). Most packages have no host build dependencies other than the toolchain which is implicit. So for most packages, you wouldn't need to do anything to support cross-compiling for the HDEPEND case. For the TDEPEND case, you'd have to modify a lot of packages. Especially, considering the fact that DEPEND=RDEPEND is the common case.
The other nice property of HDEPEND is that DEPEND and RDEPEND are target and HDEPEND in host. With TDEPEND, DEPEND would be host while RDEPEND would be target. So HDEPEND seems more intuitive. But that might be more intuitive to me cuz I mostly cross-compile. Perhaps not more intuitive for someone who doesn't.
> In EAPI 4 or later, DEPEND has been divided into two parts: > DEPEND for build-time host dependencies, and TDEPEND for > build-time target dependencies. This division is designed > specifically to minimize difficulty in the process of > adapting ebuilds that were written for earlier EAPIs, > and therefore it also minimizes the adjustments that > ebuild developers will have to make to the thought > processes involved when writing ebuilds from scratch. In > an environment that does not involve cross-compilation, > TDEPEND behaves the same as DEPEND. When an ebuild is > converted from EAPI 3 or earlier to EAPI 4 or later, > in order to support cross-compilation environments, some > dependencies may need to be migrated to TDEPEND.
> For ebuilds that have EAPI 4 or later, the emerge > --root-deps option has no effect since it is made obsolete > by division between DEPEND and TDEPEND. If EAPI 3 or > earlier ebuilds are used in combination with EAPI 4 or > later ebuilds, the --root-deps behavior will still be > applied to the EAPI 3 or earlier ebuilds (there is no > behavior change for ebuilds having older EAPIs).
> The attached patch applies to the master branch of portage (apply on > top of commit cc03d96314dc108964c9c1b0431a0466f20ad147 if it doesn't > apply to HEAD). You can test this patch by creating ebuilds with > EAPI=4_pre0_tdepend (this is a temporary EAPI, intended just for > testing purposes). I've also attached this patch to bug #317337:
> 2010/8/2 Tien-Ren Chen <trc...@chromium.org>: > > The patch works much like the one given in #317337, and I think the > > one in #317337 looks more complete. > > however #317337 is marked as duplicate of #201499 [1]. > > In the #201499 thread they actually made a nice proposal to introduce > > a new dependency system that uses dependency labels [2]. > > I'm currently planning on implementing it since it looks to be the > > preferred draft EAPI.
> > What I'm concerning is that I don't have the confidence I'm able to > > hack portage to support the new system. > > The dependency handling code in portage uses complicated algorithm and > > is lack of document. > > I'm turning to paludis to see if it's easier to be modified to fit our purpose.
> > On Fri, Jul 30, 2010 at 3:36 PM, Anush Elangovan(அனுஷ்) > > <an...@chromium.org> wrote: > >> Resending from the correct email address to get to the mailing list.
> >> On Fri, Jul 30, 2010 at 12:34 AM, Anush Elangovan(அனுஷ்) <an...@google.com> > >> wrote:
> >>> Zac, > >>> Here is a patch Tien-Ren came up with for HDEPEND. CC'ing him too. He > >>> was planning on revising the patch. Just fyi. > >>> Thanks > >>> Anush
> -- > Chromium OS Developers mailing list: chromium-os-...@chromium.org > View archives, change email options, or unsubscribe: > http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en > From c660f75b220bf2d6d5d48418b0ecbbf77ea62e4f Mon Sep 17 00:00:00 2001 > From: Zac Medico <zmed...@gentoo.org> > Date: Wed, 11 Aug 2010 04:06:35 -0700 > Subject: [PATCH] In EAPI 4, add a TDEPEND variable for build-time target dependencies.
> In EAPI 4 or later, DEPEND has been divided into two parts: > DEPEND for build-time host dependencies, and TDEPEND for > build-time target dependencies. This division is designed > specifically to minimize difficulty in the process of > adapting ebuilds that were written for earlier EAPIs, > and therefore it also minimizes the adjustments that > ebuild developers will have to make to the thought > processes involved when writing ebuilds from scratch. In > an environment that does not involve cross-compilation, > TDEPEND behaves the same as DEPEND. When an ebuild is > converted from EAPI 3 or earlier to EAPI 4 or later, > in order to support cross-compilation environments, some > dependencies may need to be migrated to TDEPEND.
> For ebuilds that have EAPI 4 or later, the emerge > --root-deps option has no effect since it is made obsolete > by division between DEPEND and TDEPEND. If EAPI 3 or > earlier ebuilds are used in combination with EAPI 4 or > later ebuilds, the --root-deps behavior will still be > applied to the EAPI 3 or earlier ebuilds (there is no > behavior change for ebuilds having older EAPIs). > --- > bin/ebuild.sh | 24 +++++++++++++------ > doc/package/ebuild/eapi/4.docbook | 44 +++++++++++++++++++++++++++++++++++++ > man/ebuild.5 | 18 +++++++++++++++ > man/emerge.1 | 15 +++++++++--- > pym/_emerge/Package.py | 2 +- > pym/_emerge/depgraph.py | 39 +++++++++++++++++++++----------- > pym/_emerge/help.py | 15 +++++++++--- > pym/portage/__init__.py | 4 +- > pym/portage/dbapi/bintree.py | 7 +++-- > pym/portage/dbapi/porttree.py | 2 +- > pym/portage/dbapi/vartree.py | 2 +- > 11 files changed, 134 insertions(+), 38 deletions(-)
> diff --git a/bin/ebuild.sh b/bin/ebuild.sh > index 0142294..f439867 100755 > --- a/bin/ebuild.sh > +++ b/bin/ebuild.sh > @@ -1264,6 +1264,7 @@ inherit() { > local B_DEPEND > local B_RDEPEND > local B_PDEPEND > + local B_TDEPEND > while [ "$1" ]; do > location="${ECLASSDIR}/${1}.eclass" > olocation="" > @@ -1302,20 +1303,21 @@ inherit() { > EBUILD_OVERLAY_ECLASSES="${EBUILD_OVERLAY_ECLASSES} ${location}" > fi
> - #We need to back up the value of DEPEND and RDEPEND to B_DEPEND and B_RDEPEND > + #We need to back up the values of *DEPEND to B_*DEPEND > #(if set).. and then restore them after the inherit call.
> #turn off glob expansion > set -f
> # Retain the old data and restore it later. > - unset B_IUSE B_REQUIRED_USE B_DEPEND B_RDEPEND B_PDEPEND > + unset B_IUSE B_REQUIRED_USE B_DEPEND B_RDEPEND B_PDEPEND B_TDEPEND > [ "${IUSE+set}" = set ] && B_IUSE="${IUSE}" > [ "${REQUIRED_USE+set}" = set ] && B_REQUIRED_USE="${REQUIRED_USE}" > [ "${DEPEND+set}" = set ] && B_DEPEND="${DEPEND}" > [ "${RDEPEND+set}" = set ] && B_RDEPEND="${RDEPEND}" > [ "${PDEPEND+set}" = set ] && B_PDEPEND="${PDEPEND}" > - unset IUSE REQUIRED_USE DEPEND RDEPEND PDEPEND > + [ "${TDEPEND+set}" = set ] && B_TDEPEND="${TDEPEND}" > + unset IUSE REQUIRED_USE DEPEND RDEPEND PDEPEND TDEPEND > #turn on glob expansion > set +f
> # Turn of extended glob matching so that g++ doesn't get incorrectly matched. > shopt -u extglob > @@ -1884,7 +1890,7 @@ if ! hasq "$EBUILD_PHASE" clean cleanrm ; then > # In order to ensure correct interaction between ebuilds and > # eclasses, they need to be unset before this process of > # interaction begins. > - unset DEPEND RDEPEND PDEPEND IUSE REQUIRED_USE > + unset DEPEND RDEPEND PDEPEND TDEPEND IUSE REQUIRED_USE
> if [[ $PORTAGE_DEBUG != 1 || ${-/x/} != $- ]] ; then > source "$EBUILD" || die "error sourcing ebuild" > @@ -1912,9 +1918,10 @@ if ! hasq "$EBUILD_PHASE" clean cleanrm ; then > DEPEND="${DEPEND} ${E_DEPEND}" > RDEPEND="${RDEPEND} ${E_RDEPEND}" > PDEPEND="${PDEPEND} ${E_PDEPEND}" > + TDEPEND="${TDEPEND} ${E_TDEPEND}" > REQUIRED_USE="${REQUIRED_USE}
I agree with Mandeep on this. Its very intuitive to see DEPEND=what I need installed to run. Switching what I've seen as its semantics can lead to the need to modify many ebuilds and cause a lot of confusion. And if some of the dependencies are minor ones (such as I need package X installed to handle corner case Y but I'll build without it) then we can see a lot of subtle bugs accidentally introduced.
-Jon
On Thu, Aug 12, 2010 at 11:19 AM, Mandeep Singh Baines <m...@chromium.org>wrote:
> Zac Medico (zmed...@gentoo.org) wrote: > > I was working on writing documentation for the HDEPEND patch when I > > realized that it seems simpler for ebuild developers if we divide the > > dependencies as follows :
> Thanks much for adding this support:) This will definitely make life > much easier for folks who need to cross-compile. But I have one > quick question.
> Why would TDEPEND be simpler? I suspect that target build dependencies > (header files and static libraries) are more common than host build > dependencies (build tools). Most packages have no host build dependencies > other than the toolchain which is implicit. So for most packages, > you wouldn't need to do anything to support cross-compiling for the > HDEPEND case. For the TDEPEND case, you'd have to modify a lot of > packages. Especially, considering the fact that DEPEND=RDEPEND is the > common case.
> The other nice property of HDEPEND is that DEPEND and RDEPEND are target > and HDEPEND in host. With TDEPEND, DEPEND would be host while RDEPEND > would be target. So HDEPEND seems more intuitive. But that might be > more intuitive to me cuz I mostly cross-compile. Perhaps not more > intuitive for someone who doesn't.
> Regards, > Mandeep
> > In EAPI 4 or later, DEPEND has been divided into two parts: > > DEPEND for build-time host dependencies, and TDEPEND for > > build-time target dependencies. This division is designed > > specifically to minimize difficulty in the process of > > adapting ebuilds that were written for earlier EAPIs, > > and therefore it also minimizes the adjustments that > > ebuild developers will have to make to the thought > > processes involved when writing ebuilds from scratch. In > > an environment that does not involve cross-compilation, > > TDEPEND behaves the same as DEPEND. When an ebuild is > > converted from EAPI 3 or earlier to EAPI 4 or later, > > in order to support cross-compilation environments, some > > dependencies may need to be migrated to TDEPEND.
> > For ebuilds that have EAPI 4 or later, the emerge > > --root-deps option has no effect since it is made obsolete > > by division between DEPEND and TDEPEND. If EAPI 3 or > > earlier ebuilds are used in combination with EAPI 4 or > > later ebuilds, the --root-deps behavior will still be > > applied to the EAPI 3 or earlier ebuilds (there is no > > behavior change for ebuilds having older EAPIs).
> > The attached patch applies to the master branch of portage (apply on > > top of commit cc03d96314dc108964c9c1b0431a0466f20ad147 if it doesn't > > apply to HEAD). You can test this patch by creating ebuilds with > > EAPI=4_pre0_tdepend (this is a temporary EAPI, intended just for > > testing purposes). I've also attached this patch to bug #317337:
> > 2010/8/2 Tien-Ren Chen <trc...@chromium.org>: > > > The patch works much like the one given in #317337, and I think the > > > one in #317337 looks more complete. > > > however #317337 is marked as duplicate of #201499 [1]. > > > In the #201499 thread they actually made a nice proposal to introduce > > > a new dependency system that uses dependency labels [2]. > > > I'm currently planning on implementing it since it looks to be the > > > preferred draft EAPI.
> > > What I'm concerning is that I don't have the confidence I'm able to > > > hack portage to support the new system. > > > The dependency handling code in portage uses complicated algorithm and > > > is lack of document. > > > I'm turning to paludis to see if it's easier to be modified to fit our > purpose.
> > > On Fri, Jul 30, 2010 at 3:36 PM, Anush Elangovan(அனுஷ்) > > > <an...@chromium.org> wrote: > > >> Resending from the correct email address to get to the mailing list.
> > >> On Fri, Jul 30, 2010 at 12:34 AM, Anush Elangovan(அனுஷ்) < > an...@google.com> > > >> wrote:
> > >>> Zac, > > >>> Here is a patch Tien-Ren came up with for HDEPEND. CC'ing him too. > He > > >>> was planning on revising the patch. Just fyi. > > >>> Thanks > > >>> Anush
> > From c660f75b220bf2d6d5d48418b0ecbbf77ea62e4f Mon Sep 17 00:00:00 2001 > > From: Zac Medico <zmed...@gentoo.org> > > Date: Wed, 11 Aug 2010 04:06:35 -0700 > > Subject: [PATCH] In EAPI 4, add a TDEPEND variable for build-time target > dependencies.
> > In EAPI 4 or later, DEPEND has been divided into two parts: > > DEPEND for build-time host dependencies, and TDEPEND for > > build-time target dependencies. This division is designed > > specifically to minimize difficulty in the process of > > adapting ebuilds that were written for earlier EAPIs, > > and therefore it also minimizes the adjustments that > > ebuild developers will have to make to the thought > > processes involved when writing ebuilds from scratch. In > > an environment that does not involve cross-compilation, > > TDEPEND behaves the same as DEPEND. When an ebuild is > > converted from EAPI 3 or earlier to EAPI 4 or later, > > in order to support cross-compilation environments, some > > dependencies may need to be migrated to TDEPEND.
> > For ebuilds that have EAPI 4 or later, the emerge > > --root-deps option has no effect since it is made obsolete > > by division between DEPEND and TDEPEND. If EAPI 3 or > > earlier ebuilds are used in combination with EAPI 4 or > > later ebuilds, the --root-deps behavior will still be > > applied to the EAPI 3 or earlier ebuilds (there is no > > behavior change for ebuilds having older EAPIs). > > --- > > bin/ebuild.sh | 24 +++++++++++++------ > > doc/package/ebuild/eapi/4.docbook | 44 > +++++++++++++++++++++++++++++++++++++ > > man/ebuild.5 | 18 +++++++++++++++ > > man/emerge.1 | 15 +++++++++--- > > pym/_emerge/Package.py | 2 +- > > pym/_emerge/depgraph.py | 39 > +++++++++++++++++++++----------- > > pym/_emerge/help.py | 15 +++++++++--- > > pym/portage/__init__.py | 4 +- > > pym/portage/dbapi/bintree.py | 7 +++-- > > pym/portage/dbapi/porttree.py | 2 +- > > pym/portage/dbapi/vartree.py | 2 +- > > 11 files changed, 134 insertions(+), 38 deletions(-)
> > diff --git a/bin/ebuild.sh b/bin/ebuild.sh > > index 0142294..f439867 100755 > > --- a/bin/ebuild.sh > > +++ b/bin/ebuild.sh > > @@ -1264,6 +1264,7 @@ inherit() { > > local B_DEPEND > > local B_RDEPEND > > local B_PDEPEND > > + local B_TDEPEND > > while [ "$1" ]; do > > location="${ECLASSDIR}/${1}.eclass" > > olocation="" > > @@ -1302,20 +1303,21 @@ inherit() {
> EBUILD_OVERLAY_ECLASSES="${EBUILD_OVERLAY_ECLASSES} ${location}" > > fi
> > - #We need to back up the value of DEPEND and RDEPEND to > B_DEPEND and B_RDEPEND > > + #We need to back up the values of *DEPEND to B_*DEPEND > > #(if set).. and then restore them after the inherit call.
> > #turn off glob expansion > > set -f
> > # Retain the old data and restore it later. > > - unset B_IUSE B_REQUIRED_USE B_DEPEND B_RDEPEND B_PDEPEND > > + unset B_IUSE B_REQUIRED_USE B_DEPEND B_RDEPEND B_PDEPEND > B_TDEPEND > > [ "${IUSE+set}" = set ] && B_IUSE="${IUSE}" > > [ "${REQUIRED_USE+set}" = set ] && > B_REQUIRED_USE="${REQUIRED_USE}" > > [ "${DEPEND+set}" = set ] && B_DEPEND="${DEPEND}" > > [ "${RDEPEND+set}" = set ] && B_RDEPEND="${RDEPEND}" > > [ "${PDEPEND+set}" = set ] && B_PDEPEND="${PDEPEND}" > > - unset IUSE REQUIRED_USE DEPEND RDEPEND PDEPEND > > + [ "${TDEPEND+set}" = set ] && B_TDEPEND="${TDEPEND}" > > + unset IUSE REQUIRED_USE DEPEND RDEPEND PDEPEND TDEPEND > > #turn on glob expansion > > set +f
On Thu, Aug 12, 2010 at 08:19:48AM -0700, Mandeep Singh Baines wrote: > Zac Medico (zmed...@gentoo.org) wrote: > > I was working on writing documentation for the HDEPEND patch when I > > realized that it seems simpler for ebuild developers if we divide the > > dependencies as follows :
> Thanks much for adding this support:) This will definitely make life > much easier for folks who need to cross-compile. But I have one > quick question.
> Why would TDEPEND be simpler? I suspect that target build dependencies > (header files and static libraries) are more common than host build > dependencies (build tools). Most packages have no host build dependencies > other than the toolchain which is implicit.
Just a sidenote, but the implicit toolchain some of us have been eyeing to kill for half a decade now- mainly since not having those deps explicit has made it hard to do new profile bring up among other things. Implicit deps pulls info away from the resolver (which originally couldn't handle that info, thus why it was never available) and makes the system pkgset for better or worse the base.
Scenarios where people were trying to use uclibc caused some issues w/ that- I half expect any clang experiments will be similar also.
> So for most packages, > you wouldn't need to do anything to support cross-compiling for the > HDEPEND case. For the TDEPEND case, you'd have to modify a lot of > packages. Especially, considering the fact that DEPEND=RDEPEND is the > common case.
> The other nice property of HDEPEND is that DEPEND and RDEPEND are target > and HDEPEND in host. With TDEPEND, DEPEND would be host while RDEPEND > would be target. So HDEPEND seems more intuitive. But that might be > more intuitive to me cuz I mostly cross-compile. Perhaps not more > intuitive for someone who doesn't.
Offhand, I'd probably favor HDEPEND for this over TDEPEND since I suspect HDEPEND is easier to sell gentoo devs on converting to- at first glance it looks like it's less conversion effort.
That said, there is a lot of automake, cmake, qt-make, etc, style deps that are in the tree so it's not going to be a "snap your fingers and it's done" scenario. I've not done any pquery scans however- would be useful if someone did anyways, since the same list used for that scan should be inclucded in the GLEP- specifically an initial list of "these are very likely host deps vs target deps".
Corner case, but has anyone looked at how things like python extensions would fit in this? I'm specifically thinking of pkgs that use distutils and friends to install (meaning they need a working python) yet compile extensions for the target- roughly it would be HDEPEND and DEPEND on python, but for that class of ebuilds I don't see how it would be handleded cleanly w/in the ebuild itself.
Alright, I've adjusted the previous TDEPEND patch into an HDEPEND patch. The only essential difference is a minor adjustment to the logic inside depgraph.py. This patch applies on top of commit ffe0e3fb6998c776b08f725c53dfe6c632ce7d02 of the master branch. You can test it by creating ebuilds with EAPI=4_pre0_hdepend (this is a temporary EAPI, intended just for testing purposes).
> Zac Medico (zmed...@gentoo.org) wrote: >> I was working on writing documentation for the HDEPEND patch when I >> realized that it seems simpler for ebuild developers if we divide the >> dependencies as follows :
> Thanks much for adding this support:) This will definitely make life > much easier for folks who need to cross-compile. But I have one > quick question.
> Why would TDEPEND be simpler? I suspect that target build dependencies > (header files and static libraries) are more common than host build > dependencies (build tools). Most packages have no host build dependencies > other than the toolchain which is implicit. So for most packages, > you wouldn't need to do anything to support cross-compiling for the > HDEPEND case. For the TDEPEND case, you'd have to modify a lot of > packages. Especially, considering the fact that DEPEND=RDEPEND is the > common case.
> The other nice property of HDEPEND is that DEPEND and RDEPEND are target > and HDEPEND in host. With TDEPEND, DEPEND would be host while RDEPEND > would be target. So HDEPEND seems more intuitive. But that might be > more intuitive to me cuz I mostly cross-compile. Perhaps not more > intuitive for someone who doesn't.
> Regards, > Mandeep
>> In EAPI 4 or later, DEPEND has been divided into two parts: >> DEPEND for build-time host dependencies, and TDEPEND for >> build-time target dependencies. This division is designed >> specifically to minimize difficulty in the process of >> adapting ebuilds that were written for earlier EAPIs, >> and therefore it also minimizes the adjustments that >> ebuild developers will have to make to the thought >> processes involved when writing ebuilds from scratch. In >> an environment that does not involve cross-compilation, >> TDEPEND behaves the same as DEPEND. When an ebuild is >> converted from EAPI 3 or earlier to EAPI 4 or later, >> in order to support cross-compilation environments, some >> dependencies may need to be migrated to TDEPEND.
>> For ebuilds that have EAPI 4 or later, the emerge >> --root-deps option has no effect since it is made obsolete >> by division between DEPEND and TDEPEND. If EAPI 3 or >> earlier ebuilds are used in combination with EAPI 4 or >> later ebuilds, the --root-deps behavior will still be >> applied to the EAPI 3 or earlier ebuilds (there is no >> behavior change for ebuilds having older EAPIs).
>> The attached patch applies to the master branch of portage (apply on >> top of commit cc03d96314dc108964c9c1b0431a0466f20ad147 if it doesn't >> apply to HEAD). You can test this patch by creating ebuilds with >> EAPI=4_pre0_tdepend (this is a temporary EAPI, intended just for >> testing purposes). I've also attached this patch to bug #317337:
I just shared two documents with all recipient in this thread. I planned to first reply to this thread to explain the purpose of that two documents, but I checked to send notification by accident. *ouch*
The document titled "Cross-compile EAPI draft" is to document the proposed HDEPEND EAPI draft, and added some possible required variables for proper supporting cross-compiling. Correction and comments are welcome, all of you in the CC list have edit permission to it.
The other document titled "cross-compile troubles" describe the trouble I met while hacking with Paludis. Some of the problems are already properly fixed in the current portage-based build flow that I'm just repeating the same fix with Paludis, while some of the problems are real problems that is hidden by implementation-defined behavior of portage (for example, order of merging package) or other toolchain, those packages should be fixed properly for better deterministic behavior. This document is more like a personal note. I'm trying to make it more complete to document all the fix we did for cross-compile, so it gives a better picture for new people to hack on our build system, and also helps us to design the PMS to be more cross-compile friendly. Again, any correction and comments are welcome.
Tien, Would it be possible for you to try out the HDEPEND patch on a ChromiumOS build to replace hard-host-deps. I am not sure if that is what you had done for your initial feedback. Along with portage you will have to upgrade sandbox (2.2). This way we can provide any other related feedback on the EAPI4 HDEPEND proposal.
Btw, I think the document is no longer available so it may be worth sending a pdf too.
On Mon, Aug 16, 2010 at 7:12 AM, Tien-Ren Chen <trc...@chromium.org> wrote: > I just shared two documents with all recipient in this thread. > I planned to first reply to this thread to explain the purpose of that > two documents, > but I checked to send notification by accident. *ouch*
> The document titled "Cross-compile EAPI draft" is to document the > proposed HDEPEND EAPI draft, > and added some possible required variables for proper supporting > cross-compiling. > Correction and comments are welcome, all of you in the CC list have > edit permission to it.
> The other document titled "cross-compile troubles" describe the > trouble I met while hacking with Paludis. > Some of the problems are already properly fixed in the current > portage-based build flow that I'm just repeating the same fix with > Paludis, > while some of the problems are real problems that is hidden by > implementation-defined behavior of portage (for example, order of > merging package) or other toolchain, those packages should be fixed > properly for better deterministic behavior. > This document is more like a personal note. I'm trying to make it more > complete to document all the fix we did for cross-compile, > so it gives a better picture for new people to hack on our build > system, and also helps us to design the PMS to be more cross-compile > friendly. > Again, any correction and comments are welcome.
