Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds)

Showing 1-13 of 13 messages
Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) David James 7/27/10 6:05 AM
+chromiumos-dev

On Mon, Jul 26, 2010 at 10:06 PM, Brian Harring <ferr...@gmail.com> wrote:
> Hola.  This posting caught my eye- namely since this obviously is an
> issue we have in gentoo, and there have been some proposals for how to
> solve it but nothing has been implemented.
>
> If I may ask, is there a reason you didn't look at an EAPI with
> (essentially) ABI slots?  Such a route, while it requires specifying a
> bit more info (or tweaking the manager to generate that info), would
> give fine grained control to the resolver for doing rebuilds of the
> sort mentioned above.

Long term, I think that that's exactly what we want. Right now, it's
hard to tell which packages require dependencies to be rebuilt, and
which packages do not have this requirement. If there were an
automated way to tell which packages need their dependencies to be
rebuilt, that would be great.

The ABI slot solution sounds good, as long as we don't have too many
packages where the maintainers specify their rebuild requirements
incorrectly (e.g., by breaking binary compatibility without realizing
it). If we found that this was a regular problem, we would probably
want to go back to just rebuilding all dependencies whenever a package
changes.

> Aside from that, I'd be very interested if there is an ML, irc
> channel, or any area people are doing discussions of this sort- I'm
> the author of pkgcore, long term hacker of portage
> (refactoring/cleanup- trying to combat the mess rather than creating
> it), and have been working on issues similar to what y'all have been
> rolling out.  What's been a bit problematic is that there isn't
> exactly a lot of people doing the level of binpkg work y'all are- thus
> finding stakeholders to bug for info has been pretty sparse.

Personally, I'd be quite interested in this problem. The
chromium-os-dev list is fine for this type of discussion, but if you
have another mailing list that would be better, I'd be happy to join.

> At least from the pkgcore angle, I'd very much like to be tracking
> what y'alls reqs are, hacks you're using to work around the manager,
> etc, so I can either address it in my own manager or at least plan for
> addressing it down the line- whether it be from the PM implementation
> angle, or from the standpoint of trying to address y'alls needs in a
> new EAPI.

Here are a few other suggestions about things to address in regular emerge:
 1. To address package rebuilds completely correctly, we'll also need
to differentiate between host and target build dependencies. There's a
patch for this, but it looks like it got closed as a duplicate:
  http://bugs.gentoo.org/show_bug.cgi?id=317337
 2. Strangely, Portage often favors the dependencies (and masking) of
built packages over the ebuilds you have in your package dir. I would
rather that Portage look at the ebuilds, because developers often
change the ebuilds and want their new ebuild to be prioritized over
the one that was already built. In parallel_emerge, we work around
this by setting --usepkg=n for dependency calculation and then convert
the source package to a binary package at a later stage.
 3. Does Portage have public APIs for calculating dependencies and
merging packages? Right now, parallel_emerge uses private APIs but I
would like to convert it to public APIs if they exist.
 4. Portage is a bit aggressive about locking directories, so this
makes parallel merging slow. Whenever a package is being merged,
Portage locks the entire package directory. This means essentially
that package merges do not occur in parallel at all. Is it possible to
reduce the scope of the locking so that it does not have such a
significant effect on performance?
 5. Some packages will make global system changes in their postinstall
scripts, but these scripts are not parallel-safe. Can we make these
scripts parallel-safe? Currently, this is not a big problem for emerge
because it just locks for the entire merge, including postinstall (see
#4), but once we get rid of that lock it becomes a problem.

Cheers,

David

Re: Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) Brian Harring 7/28/10 4:22 AM
On Tue, Jul 27, 2010 at 09:05:28AM -0400, David James wrote:
> On Mon, Jul 26, 2010 at 10:06 PM, Brian Harring <ferr...@gmail.com> wrote:
> > If I may ask, is there a reason you didn't look at an EAPI with
> > (essentially) ABI slots?  Such a route, while it requires specifying a
> > bit more info (or tweaking the manager to generate that info), would
> > give fine grained control to the resolver for doing rebuilds of the
> > sort mentioned above.
>
> Long term, I think that that's exactly what we want. Right now, it's
> hard to tell which packages require dependencies to be rebuilt, and
> which packages do not have this requirement. If there were an
> automated way to tell which packages need their dependencies to be
> rebuilt, that would be great.
>
> The ABI slot solution sounds good, as long as we don't have too many
> packages where the maintainers specify their rebuild requirements
> incorrectly (e.g., by breaking binary compatibility without realizing
> it). If we found that this was a regular problem, we would probably
> want to go back to just rebuilding all dependencies whenever a package
> changes.

Bad packaging metadata we can't really do much about realistically,
beyond threaten to wedige the packaging dev- that said a DT_NEEDED
validation QA check to ensure the targets are within stated rdeps
might be useful.

Doesn't do anything for dlopen or static, but it was one of the more
useful features of the old EBD/saviour branch of portage- iirc I named
that feature 'verify-linkage'.  Implementation was ldd based (ick),
but it was a start.


> > Aside from that, I'd be very interested if there is an ML, irc
> > channel, or any area people are doing discussions of this sort- I'm
> > the author of pkgcore, long term hacker of portage
> > (refactoring/cleanup- trying to combat the mess rather than creating
> > it), and have been working on issues similar to what y'all have been
> > rolling out.  What's been a bit problematic is that there isn't
> > exactly a lot of people doing the level of binpkg work y'all are- thus
> > finding stakeholders to bug for info has been pretty sparse.
>
> Personally, I'd be quite interested in this problem. The
> chromium-os-dev list is fine for this type of discussion, but if you
> have another mailing list that would be better, I'd be happy to join.

Unfortunately I'm not particularly aware of any.  Typically I track
down people in irc and pick their brains directly; that said from an
irc standpoint #gentoo-portage on freenode is probably the best spot
to catch the appropriate people (zac medico/zmedico, sebastian
luther/zac, solar, myself) who would have interest/comments.


> Here are a few other suggestions about things to address in regular emerge:
>  1. To address package rebuilds completely correctly, we'll also need
> to differentiate between host and target build dependencies. There's a
> patch for this, but it looks like it got closed as a duplicate:
>   http://bugs.gentoo.org/show_bug.cgi?id=317337

This was proposed as "bdepends" at one point- build time depends,
specifically what is executed during building the package (bash and
sed for example).

The problem w/ a new dependency type is mainly getting developer buy
in- building tools to help developers track/validate these
dependencies.  At least in the past, this sort of proposal died out
since their wasn't a sufficient carrot (nor was it easy to modify
portage for such a capability).


>  2. Strangely, Portage often favors the dependencies (and masking) of
> built packages over the ebuilds you have in your package dir. I would
> rather that Portage look at the ebuilds, because developers often
> change the ebuilds and want their new ebuild to be prioritized over
> the one that was already built. In parallel_emerge, we work around
> this by setting --usepkg=n for dependency calculation and then convert
> the source package to a binary package at a later stage.

Essentially you want binpkgs to be utilized strictly as a cache,
right?  I suspect this isn't a particularly hard mod to implement for
the folk who know the innards of _emerge.* reasonably well.


>  3. Does Portage have public APIs for calculating dependencies and
> merging packages? Right now, parallel_emerge uses private APIs but I
> would like to convert it to public APIs if they exist.

CC'ing the folk who could best answer this one- at this point, there
isn't perse a public api for the most part.  I know they're currently
trying to derive one for packagekit/porthole/SoC, but I'm not aware of
it's status.


>  4. Portage is a bit aggressive about locking directories, so this
> makes parallel merging slow. Whenever a package is being merged,
> Portage locks the entire package directory. This means essentially
> that package merges do not occur in parallel at all. Is it possible to
> reduce the scope of the locking so that it does not have such a
> significant effect on performance?

Heh.  This is a contentious issue.  Roughly, everything after
pkg_setup when it comes to merging is expected to be the only thing
modifying the FS during this- the reasoning is purely due to
pkg_preinst/pkg_postinst being essentially uncontrolled access to the
livefs.  That critical locking unfortunately does need to exist if
those phases are defined.  I'd expect in addition that the treewalk
bits (the merger) wouldn't particularly like having a run in with
another active merge at the same time.

That said, the scope of the locking definitely should be reducable,
and parallelization increased w/in the atomic locked merge.  Locking
of the binpkgs repository in particular seems unneeded since
effectively it's an untarring to ${D}, and unpacking the xpak (that
annoying trailing data after EOS on tbzs) to ${T} w/in the pkgs build
env.  After that that point, there shouldn't need to be any further
locking on binpkgs- further creating a new binpkg doesn't require a
write lock until you actually try to make it live w/in the binpkg repo
(the final atomic rename), so at least the compression steps should be
able to be done in parallel.


>  5. Some packages will make global system changes in their postinstall
> scripts, but these scripts are not parallel-safe. Can we make these
> scripts parallel-safe? Currently, this is not a big problem for emerge
> because it just locks for the entire merge, including postinstall (see
> #4), but once we get rid of that lock it becomes a problem.

I'm definitely open to suggestions of how to make them safe, but by
pkg_(pre|post)(inst|rm)'s nature, they're free form root level
execution.  From a format standpoint, for better or worse they
technically can do whatever the hell they want, bounded only by QA
standards of that particularly repository.

I could see adding a PROPERTIES hint of some sort indicating that the
locking isn't needed for that pkg offhand, although that's a bit of a
hack.

Zac/Sebastian, any comments on the above?

~brian

Re: Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) Zac Medico 7/28/10 6:46 PM

A notable advantage of giving the resolver the metadata (ABI slots)
to account for rebuilds in advance is that it allows for more
informed decisions with respect to aggressive --jobs
parallelization. Considering --fast chromium-os builds, this seems
like an area of interest.

>>> Aside from that, I'd be very interested if there is an ML, irc
>>> channel, or any area people are doing discussions of this sort- I'm
>>> the author of pkgcore, long term hacker of portage
>>> (refactoring/cleanup- trying to combat the mess rather than creating
>>> it), and have been working on issues similar to what y'all have been
>>> rolling out.  What's been a bit problematic is that there isn't
>>> exactly a lot of people doing the level of binpkg work y'all are- thus
>>> finding stakeholders to bug for info has been pretty sparse.
>>
>> Personally, I'd be quite interested in this problem. The
>> chromium-os-dev list is fine for this type of discussion, but if you
>> have another mailing list that would be better, I'd be happy to join.
>
> Unfortunately I'm not particularly aware of any.  Typically I track
> down people in irc and pick their brains directly; that said from an
> irc standpoint #gentoo-portage on freenode is probably the best spot
> to catch the appropriate people (zac medico/zmedico, sebastian
> luther/zac, solar, myself) who would have interest/comments.

The gentoo-po...@gentoo.org list seems applicable.

>> Here are a few other suggestions about things to address in regular emerge:
>>  1. To address package rebuilds completely correctly, we'll also need
>> to differentiate between host and target build dependencies. There's a
>> patch for this, but it looks like it got closed as a duplicate:
>>   http://bugs.gentoo.org/show_bug.cgi?id=317337
>
> This was proposed as "bdepends" at one point- build time depends,
> specifically what is executed during building the package (bash and
> sed for example).
>
> The problem w/ a new dependency type is mainly getting developer buy
> in- building tools to help developers track/validate these
> dependencies.  At least in the past, this sort of proposal died out
> since their wasn't a sufficient carrot (nor was it easy to modify
> portage for such a capability).

Nowadays, it would be pretty trivial to add a BDEPEND variable or
something like that. Adding a new *DEPEND variable like this would
be a lot less invasive that trying to jump to a DEPENDENCIES
variable that unifies *DEPEND.

>>  2. Strangely, Portage often favors the dependencies (and masking) of
>> built packages over the ebuilds you have in your package dir. I would
>> rather that Portage look at the ebuilds, because developers often
>> change the ebuilds and want their new ebuild to be prioritized over
>> the one that was already built. In parallel_emerge, we work around
>> this by setting --usepkg=n for dependency calculation and then convert
>> the source package to a binary package at a later stage.
>
> Essentially you want binpkgs to be utilized strictly as a cache,
> right?  I suspect this isn't a particularly hard mod to implement for
> the folk who know the innards of _emerge.* reasonably well.

I think maybe he's talking about updating the *DEPEND metadata of
the binary packages, so that they match the ebuilds. I have an old
script that does it:

http://dev.gentoo.org/~zmedico/portage/branches/2.1/bin/vdb-tools.py

You could update the metadata of all binary packages with this command:

  vdb-tools.py --transfer --pkgtype=binary

>>  3. Does Portage have public APIs for calculating dependencies and
>> merging packages? Right now, parallel_emerge uses private APIs but I
>> would like to convert it to public APIs if they exist.
>
> CC'ing the folk who could best answer this one- at this point, there
> isn't perse a public api for the most part.  I know they're currently
> trying to derive one for packagekit/porthole/SoC, but I'm not aware of
> it's status.

Brian Dolbeck (porthole developer) has been working on this, but I'm
unsure of progress. I've CC'd him in case he wants to chime in.

>>  4. Portage is a bit aggressive about locking directories, so this
>> makes parallel merging slow. Whenever a package is being merged,
>> Portage locks the entire package directory. This means essentially
>> that package merges do not occur in parallel at all. Is it possible to
>> reduce the scope of the locking so that it does not have such a
>> significant effect on performance?
>
> Heh.  This is a contentious issue.  Roughly, everything after
> pkg_setup when it comes to merging is expected to be the only thing
> modifying the FS during this- the reasoning is purely due to
> pkg_preinst/pkg_postinst being essentially uncontrolled access to the
> livefs.  That critical locking unfortunately does need to exist if
> those phases are defined.  I'd expect in addition that the treewalk
> bits (the merger) wouldn't particularly like having a run in with
> another active merge at the same time.
>
> That said, the scope of the locking definitely should be reducable,
> and parallelization increased w/in the atomic locked merge.  Locking
> of the binpkgs repository in particular seems unneeded since
> effectively it's an untarring to ${D}, and unpacking the xpak (that
> annoying trailing data after EOS on tbzs) to ${T} w/in the pkgs build
> env.  After that that point, there shouldn't need to be any further
> locking on binpkgs- further creating a new binpkg doesn't require a
> write lock until you actually try to make it live w/in the binpkg repo
> (the final atomic rename), so at least the compression steps should be
> able to be done in parallel.

Of course it can be done. Again, additional metadata for ABI slots
will be quite helpful for aggressive --jobs parallelization like this.

>>  5. Some packages will make global system changes in their postinstall
>> scripts, but these scripts are not parallel-safe. Can we make these
>> scripts parallel-safe? Currently, this is not a big problem for emerge
>> because it just locks for the entire merge, including postinstall (see
>> #4), but once we get rid of that lock it becomes a problem.
>
> I'm definitely open to suggestions of how to make them safe, but by
> pkg_(pre|post)(inst|rm)'s nature, they're free form root level
> execution.  From a format standpoint, for better or worse they
> technically can do whatever the hell they want, bounded only by QA
> standards of that particularly repository.
>
> I could see adding a PROPERTIES hint of some sort indicating that the
> locking isn't needed for that pkg offhand, although that's a bit of a
> hack.

A PROPERTIES or RESTRICT flag sounds pretty reasonable to me. This
leads to the question of whether it should be opt-in or opt-out. We
could conceivably make the opt-in/opt-out behavior conditional on
EAPI, but maybe that won't be necessary.

We could also add helpers for locking/unlocking resources. This
could be implemented in an eclass, but it might be nicer to had the
helpers on the package manager side (in a new EAPI).
--
Thanks,
Zac

Re: [cros-dev] Re: Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) Anush Elangovan 7/30/10 12:36 AM
Resending from the correct email address to get to the mailing list. 

On Fri, Jul 30, 2010 at 12:34 AM, Anush Elangovan(அனுஷ்) <an...@google.com> wrote:
Zac,
   Here is a patch Tien-Ren came up with for HDEPEND. CC'ing him too. He was planning on revising the patch. Just fyi. 

Thanks
Anush



--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en


Re: [cros-dev] Re: Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) Tien-Ren Chen 8/2/10 7:18 PM
The patch works much like the one given in #317337, and I think the
one in #317337 looks more complete.
however #317337 is marked as duplicate of #201499 [1].
In the #201499 thread they actually made a nice proposal to introduce
a new dependency system that uses dependency labels [2].
I'm currently planning on implementing it since it looks to be the
preferred draft EAPI.

What I'm concerning is that I don't have the confidence I'm able to
hack portage to support the new system.
The dependency handling code in portage uses complicated algorithm and
is lack of document.
I'm turning to paludis to see if it's easier to be modified to fit our purpose.

Reference
[1] http://bugs.gentoo.org/show_bug.cgi?id=201499
[2] http://bugs.gentoo.org/attachment.cgi?id=137968

On Fri, Jul 30, 2010 at 3:36 PM, Anush Elangovan(அனுஷ்)
<an...@chromium.org> wrote:
> Resending from the correct email address to get to the mailing list.
>
> On Fri, Jul 30, 2010 at 12:34 AM, Anush Elangovan(அனுஷ்) <an...@google.com>
> wrote:
>>
>> Zac,
>>    Here is a patch Tien-Ren came up with for HDEPEND. CC'ing him too. He
>> was planning on revising the patch. Just fyi.
>> Thanks
>> Anush

Re: [cros-dev] Re: Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) Anush Elangovan 8/3/10 9:14 AM
Tien,
    Zac the upstream portage maintainer and other Gentoo devs hang out on #chromium-os IRC and are very helpful if you have any questions on the internals of portage. 

Anush
Re: [cros-dev] Re: Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) Zac Medico 8/11/10 4:38 AM
I was working on writing documentation for the HDEPEND patch when I
realized that it seems simpler for ebuild developers if we divide the
dependencies as follows :

In EAPI 4 or later, DEPEND has been divided into two parts:
DEPEND for build-time host dependencies, and TDEPEND for
build-time target dependencies. This division is designed
specifically to minimize difficulty in the process of
adapting ebuilds that were written for earlier EAPIs,
and therefore it also minimizes the adjustments that
ebuild developers will have to make to the thought
processes involved when writing ebuilds from scratch. In
an environment that does not involve cross-compilation,
TDEPEND behaves the same as DEPEND. When an ebuild is
converted from EAPI 3 or earlier to EAPI 4 or later,
in order to support cross-compilation environments, some
dependencies may need to be migrated to TDEPEND.

For ebuilds that have EAPI 4 or later, the emerge
--root-deps option has no effect since it is made obsolete
by division between DEPEND and TDEPEND. If EAPI 3 or
earlier ebuilds are used in combination with EAPI 4 or
later ebuilds, the --root-deps behavior will still be
applied to the EAPI 3 or earlier ebuilds (there is no
behavior change for ebuilds having older EAPIs).

The attached patch applies to the master branch of portage (apply on
top of commit cc03d96314dc108964c9c1b0431a0466f20ad147 if it doesn't
apply to HEAD). You can test this patch by creating ebuilds with
EAPI=4_pre0_tdepend (this is a temporary EAPI, intended just for
testing purposes). I've also attached this patch to bug #317337:

  http://bugs.gentoo.org/show_bug.cgi?id=317337

2010/8/2 Tien-Ren Chen <trc...@chromium.org>:

Re: [cros-dev] Re: Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) Mandeep Baines 8/12/10 8:19 AM
Zac Medico (zme...@gentoo.org) wrote:
> I was working on writing documentation for the HDEPEND patch when I
> realized that it seems simpler for ebuild developers if we divide the
> dependencies as follows :
>

Thanks much for adding this support:) This will definitely make life
much easier for folks who need to cross-compile. But I have one
quick question.

Why would TDEPEND be simpler? I suspect that target build dependencies
(header files and static libraries) are more common than host build
dependencies (build tools). Most packages have no host build dependencies
other than the toolchain which is implicit. So for most packages,
you wouldn't need to do anything to support cross-compiling for the
HDEPEND case. For the TDEPEND case, you'd have to modify a lot of
packages. Especially, considering the fact that DEPEND=RDEPEND is the
common case.

The other nice property of HDEPEND is that DEPEND and RDEPEND are target
and HDEPEND in host. With TDEPEND, DEPEND would be host while RDEPEND
would be target. So HDEPEND seems more intuitive. But that might be
more intuitive to me cuz I mostly cross-compile. Perhaps not more
intuitive for someone who doesn't.

Regards,
Mandeep

> --
> Chromium OS Developers mailing list: chromiu...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

> From c660f75b220bf2d6d5d48418b0ecbbf77ea62e4f Mon Sep 17 00:00:00 2001
> From: Zac Medico <zme...@gentoo.org>
> Date: Wed, 11 Aug 2010 04:06:35 -0700
> Subject: [PATCH] In EAPI 4, add a TDEPEND variable for build-time target dependencies.


>
> In EAPI 4 or later, DEPEND has been divided into two parts:
> DEPEND for build-time host dependencies, and TDEPEND for
> build-time target dependencies. This division is designed
> specifically to minimize difficulty in the process of
> adapting ebuilds that were written for earlier EAPIs,
> and therefore it also minimizes the adjustments that
> ebuild developers will have to make to the thought
> processes involved when writing ebuilds from scratch. In
> an environment that does not involve cross-compilation,
> TDEPEND behaves the same as DEPEND. When an ebuild is
> converted from EAPI 3 or earlier to EAPI 4 or later,
> in order to support cross-compilation environments, some
> dependencies may need to be migrated to TDEPEND.
>
> For ebuilds that have EAPI 4 or later, the emerge
> --root-deps option has no effect since it is made obsolete
> by division between DEPEND and TDEPEND. If EAPI 3 or
> earlier ebuilds are used in combination with EAPI 4 or
> later ebuilds, the --root-deps behavior will still be
> applied to the EAPI 3 or earlier ebuilds (there is no
> behavior change for ebuilds having older EAPIs).
> ---
>  bin/ebuild.sh                     |   24 +++++++++++++------
>  doc/package/ebuild/eapi/4.docbook |   44 +++++++++++++++++++++++++++++++++++++
>  man/ebuild.5                      |   18 +++++++++++++++
>  man/emerge.1                      |   15 +++++++++---
>  pym/_emerge/Package.py            |    2 +-
>  pym/_emerge/depgraph.py           |   39 +++++++++++++++++++++-----------
>  pym/_emerge/help.py               |   15 +++++++++---
>  pym/portage/__init__.py           |    4 +-
>  pym/portage/dbapi/bintree.py      |    7 +++--
>  pym/portage/dbapi/porttree.py     |    2 +-
>  pym/portage/dbapi/vartree.py      |    2 +-
>  11 files changed, 134 insertions(+), 38 deletions(-)
>
> diff --git a/bin/ebuild.sh b/bin/ebuild.sh
> index 0142294..f439867 100755
> --- a/bin/ebuild.sh
> +++ b/bin/ebuild.sh
> @@ -1264,6 +1264,7 @@ inherit() {
>          local B_DEPEND
>          local B_RDEPEND
>          local B_PDEPEND
> +        local B_TDEPEND
>          while [ "$1" ]; do
>                  location="${ECLASSDIR}/${1}.eclass"
>                  olocation=""
> @@ -1302,20 +1303,21 @@ inherit() {
>                                  EBUILD_OVERLAY_ECLASSES="${EBUILD_OVERLAY_ECLASSES} ${location}"
>                  fi
>  
> -                #We need to back up the value of DEPEND and RDEPEND to B_DEPEND and B_RDEPEND
> +                #We need to back up the values of *DEPEND to B_*DEPEND
>                  #(if set).. and then restore them after the inherit call.
>  
>                  #turn off glob expansion
>                  set -f
>  
>                  # Retain the old data and restore it later.
> -                unset B_IUSE B_REQUIRED_USE B_DEPEND B_RDEPEND B_PDEPEND
> +                unset B_IUSE B_REQUIRED_USE B_DEPEND B_RDEPEND B_PDEPEND B_TDEPEND
>                  [ "${IUSE+set}"       = set ] && B_IUSE="${IUSE}"
>                  [ "${REQUIRED_USE+set}" = set ] && B_REQUIRED_USE="${REQUIRED_USE}"
>                  [ "${DEPEND+set}"     = set ] && B_DEPEND="${DEPEND}"
>                  [ "${RDEPEND+set}"    = set ] && B_RDEPEND="${RDEPEND}"
>                  [ "${PDEPEND+set}"    = set ] && B_PDEPEND="${PDEPEND}"
> -                unset IUSE REQUIRED_USE DEPEND RDEPEND PDEPEND
> +                [ "${TDEPEND+set}"    = set ] && B_TDEPEND="${TDEPEND}"
> +                unset IUSE REQUIRED_USE DEPEND RDEPEND PDEPEND TDEPEND
>                  #turn on glob expansion
>                  set +f
>  
> @@ -1331,6 +1333,7 @@ inherit() {
>                  [ "${DEPEND+set}"     = set ] && export E_DEPEND="${E_DEPEND} ${DEPEND}"
>                  [ "${RDEPEND+set}"    = set ] && export E_RDEPEND="${E_RDEPEND} ${RDEPEND}"
>                  [ "${PDEPEND+set}"    = set ] && export E_PDEPEND="${E_PDEPEND} ${PDEPEND}"
> +                [ "${TDEPEND+set}"    = set ] && export E_TDEPEND="${E_TDEPEND} ${TDEPEND}"
>  
>                  [ "${B_IUSE+set}"     = set ] && IUSE="${B_IUSE}"
>                  [ "${B_IUSE+set}"     = set ] || unset IUSE
> @@ -1347,6 +1350,9 @@ inherit() {
>                  [ "${B_PDEPEND+set}"  = set ] && PDEPEND="${B_PDEPEND}"
>                  [ "${B_PDEPEND+set}"  = set ] || unset PDEPEND
>  
> +                [ "${B_TDEPEND+set}"  = set ] && TDEPEND="${B_TDEPEND}"
> +                [ "${B_TDEPEND+set}"  = set ] || unset TDEPEND
> +
>                  #turn on glob expansion
>                  set +f
>  
> @@ -1606,7 +1612,7 @@ source_all_bashrcs() {
>  # when portage is upgrading itself.
>  
>  READONLY_EBUILD_METADATA="DEFINED_PHASES DEPEND DESCRIPTION
> -        EAPI HOMEPAGE INHERITED IUSE REQUIRED_USE KEYWORDS LICENSE
> +        EAPI TDEPEND HOMEPAGE INHERITED IUSE REQUIRED_USE KEYWORDS LICENSE
>          PDEPEND PROVIDE RDEPEND RESTRICT SLOT SRC_URI"
>  
>  READONLY_PORTAGE_VARS="D EBUILD EBUILD_PHASE \
> @@ -1780,7 +1786,7 @@ preprocess_ebuild_env() {
>  export SANDBOX_ON="1"
>  export S=${WORKDIR}/${P}
>  
> -unset E_IUSE E_REQUIRED_USE E_DEPEND E_RDEPEND E_PDEPEND
> +unset E_IUSE E_REQUIRED_USE E_DEPEND E_RDEPEND E_PDEPEND E_TDEPEND
>  
>  # Turn of extended glob matching so that g++ doesn't get incorrectly matched.
>  shopt -u extglob
> @@ -1884,7 +1890,7 @@ if ! hasq "$EBUILD_PHASE" clean cleanrm ; then
>                  # In order to ensure correct interaction between ebuilds and
>                  # eclasses, they need to be unset before this process of
>                  # interaction begins.
> -                unset DEPEND RDEPEND PDEPEND IUSE REQUIRED_USE
> +                unset DEPEND RDEPEND PDEPEND TDEPEND IUSE REQUIRED_USE
>  
>                  if [[ $PORTAGE_DEBUG != 1 || ${-/x/} != $- ]] ; then
>                          source "$EBUILD" || die "error sourcing ebuild"
> @@ -1912,9 +1918,10 @@ if ! hasq "$EBUILD_PHASE" clean cleanrm ; then
>                  DEPEND="${DEPEND} ${E_DEPEND}"
>                  RDEPEND="${RDEPEND} ${E_RDEPEND}"
>                  PDEPEND="${PDEPEND} ${E_PDEPEND}"
> +                TDEPEND="${TDEPEND} ${E_TDEPEND}"
>                  REQUIRED_USE="${REQUIRED_USE} ${E_REQUIRED_USE}"
>                  
> -                unset ECLASS E_IUSE E_REQUIRED_USE E_DEPEND E_RDEPEND E_PDEPEND
> +                unset ECLASS E_IUSE E_REQUIRED_USE E_DEPEND E_RDEPEND E_PDEPEND E_TDEPEND
>  
>                  # alphabetically ordered by $EBUILD_PHASE value
>                  case "$EAPI" in
> @@ -2159,13 +2166,14 @@ ebuild_main() {
>  
>                  auxdbkeys="DEPEND RDEPEND SLOT SRC_URI RESTRICT HOMEPAGE LICENSE
>                          DESCRIPTION KEYWORDS INHERITED IUSE REQUIRED_USE PDEPEND PROVIDE EAPI
> -                        PROPERTIES DEFINED_PHASES UNUSED_05 UNUSED_04
> +                        PROPERTIES DEFINED_PHASES TDEPEND UNUSED_04
>                          UNUSED_03 UNUSED_02 UNUSED_01"
>  
>                  #the extra $(echo) commands remove newlines
>                  [ -n "${EAPI}" ] || EAPI=0
>                  case "$EAPI" in
>                          0|1|2|3)
> +                                unset TDEPEND
>                                  unset REQUIRED_USE
>                                  ;;
>                  esac
> diff --git a/doc/package/ebuild/eapi/4.docbook b/doc/package/ebuild/eapi/4.docbook
> index 8d7942b..ad40de3 100644
> --- a/doc/package/ebuild/eapi/4.docbook
> +++ b/doc/package/ebuild/eapi/4.docbook
> @@ -17,6 +17,50 @@ The dodoc helper now has a -r option which enables recursion.
>  </section>
>  <section id='package-ebuild-eapi-4_pre1-metadata'>
>  <title>Metadata</title>
> +<section id='package-ebuild-eapi-4_pre1-metadata-dependencies'>
> +<title>Dependencies</title>
> +<section id='package-ebuild-eapi-4_pre1-metadata-dependencies-tdepend'>
> +<title>TDEPEND</title>
> +<para>
> +In EAPI 4 or later, DEPEND has been divided into two parts:
> +DEPEND for build-time host dependencies, and TDEPEND for
> +build-time target dependencies.
> +<table><title>Build-Time Dependencies</title>
> +<tgroup cols='2' align='left' >
> +<colspec colname='variable'/>
> +<colspec colname='description'/>
> +<thead>
> +<row>
> +<entry>Variable</entry>
> +<entry>Description</entry>
> +</row>
> +</thead>
> +<tbody>
> +<row>
> +<entry>DEPEND</entry>
> +<entry>build-time host dependencies</entry>
> +</row>
> +<row>
> +<entry>TDEPEND</entry>
> +<entry>build-time target dependencies</entry>
> +</row>
> +</tbody>
> +</tgroup>
> +</table>
> +This division is designed
> +specifically to minimize difficulty in the process of
> +adapting ebuilds that were written for earlier EAPIs,
> +and therefore it also minimizes the adjustments that
> +ebuild developers will have to make to the thought
> +processes involved when writing ebuilds from scratch. In
> +an environment that does not involve cross-compilation,
> +TDEPEND behaves the same as DEPEND. When an ebuild is
> +converted from EAPI 3 or earlier to EAPI 4 or later,
> +in order to support cross-compilation environments, some
> +dependencies may need to be migrated to TDEPEND.
> +</para>
> +</section>
> +</section>
>  <section id='package-ebuild-eapi-4_pre1-metadata-required-use'>
>  <title>REQUIRED_USE</title>
>  <para>
> diff --git a/man/ebuild.5 b/man/ebuild.5
> index 1fcc383..cfe7dae 100644
> --- a/man/ebuild.5
> +++ b/man/ebuild.5
> @@ -415,6 +415,24 @@ will use that to consider the dependency satisfied.
>  
>  .RE
>  .TP
> +\fBTDEPEND\fR
> +In EAPI 4 or later, \fBDEPEND\fR has been divided into two parts:
> +\fBDEPEND\fR for build-time host dependencies, and
> +\fBTDEPEND\fR for build-time target dependencies.
> +This division is designed
> +specifically to minimize difficulty in the process of
> +adapting ebuilds that were written for earlier \fBEAPI\fRs, and
> +therefore it also minimizes the adjustments that ebuild
> +developers will have to make to the thought processes
> +involved when writing ebuilds from scratch. In an
> +environment that does not involve cross-compilation,
> +\fBTDEPEND\fR behaves the same as \fBDEPEND\fR. When an ebuild is
> +converted from \fBEAPI 3\fR or earlier to \fBEAPI 4\fR or later,
> +in order to support cross-compilation environments, some
> +dependencies may need to be migrated to \fBTDEPEND\fR.
> +.br
> +You may use the same syntax to vary dependencies as seen above in \fBDEPEND\fR.
> +.TP
>  \fBRDEPEND\fR
>  This should contain a list of all packages that are required for this
>  program to run (aka runtime depend). If this is not set in \fBEAPI\fR
> diff --git a/man/emerge.1 b/man/emerge.1
> index 73c90d3..82e0098 100644
> --- a/man/emerge.1
> +++ b/man/emerge.1
> @@ -548,10 +548,17 @@ If no argument is given then build\-time dependencies of packages for
>  \fBROOT\fR instead of /. If the \fBrdeps\fR argument is given then discard
>  all build\-time dependencies of packages for \fBROOT\fR. This option is
>  only meaningful when used together with \fBROOT\fR and it should not
> -be enabled under normal circumstances. For currently supported
> -\fBEAPI\fR values, the build-time dependencies are specified in the
> -\fBDEPEND\fR variable. However, behavior may change for new
> -\fBEAPI\fRs when related extensions are added in the future.
> +be enabled under normal circumstances. For ebuilds that have \fBEAPI\fR
> +3 or earlier, the build\-time dependencies are specified in the
> +\fBDEPEND\fR variable. For ebuilds that have \fBEAPI 4\fR or later,
> +the emerge \-\-root\-deps option has no effect
> +since it is made obsolete by division
> +between \fBDEPEND\fR and \fBTDEPEND\fR. If \fBEAPI 3\fR or
> +earlier ebuilds are used in combination with
> +\fBEAPI 4\fR or later ebuilds, the \-\-root\-deps
> +behavior will still be applied to the \fBEAPI
> +3\fR or earlier ebuilds (there is no behavior
> +change for ebuilds having older \fBEAPI\fRs).
>  .TP
>  .BR "\-\-select"[=n]
>  Add specified packages to the world set (inverse of
> diff --git a/pym/_emerge/Package.py b/pym/_emerge/Package.py
> index 992851b..d06ae45 100644
> --- a/pym/_emerge/Package.py
> +++ b/pym/_emerge/Package.py
> @@ -31,7 +31,7 @@ class Package(Task):
>                  "INHERITED", "IUSE", "KEYWORDS",
>                  "LICENSE", "PDEPEND", "PROVIDE", "RDEPEND",
>                  "repository", "PROPERTIES", "RESTRICT", "SLOT", "USE",
> -                "_mtime_", "DEFINED_PHASES", "REQUIRED_USE"]
> +                "_mtime_", "DEFINED_PHASES", "REQUIRED_USE", "TDEPEND"]
>  
>          def __init__(self, **kwargs):
>                  Task.__init__(self, **kwargs)
> diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
> index c660728..0df8b00 100644
> --- a/pym/_emerge/depgraph.py
> +++ b/pym/_emerge/depgraph.py
> @@ -1137,7 +1137,7 @@ class depgraph(object):
>                  removal_action = "remove" in self._dynamic_config.myparams
>  
>                  edepend={}
> -                depkeys = ["DEPEND","RDEPEND","PDEPEND"]
> +                depkeys = ["DEPEND","RDEPEND","PDEPEND","TDEPEND"]
>                  for k in depkeys:
>                          edepend[k] = metadata[k]
>  
> @@ -1147,7 +1147,6 @@ class depgraph(object):
>                          "empty" not in self._dynamic_config.myparams:
>                          edepend["RDEPEND"] = ""
>                          edepend["PDEPEND"] = ""
> -                bdeps_optional = False
>  
>                  if pkg.built and not removal_action:
>                          if self._frozen_config.myopts.get("--with-bdeps", "n") == "y":
> @@ -1158,29 +1157,41 @@ class depgraph(object):
>                                  # could make --with-bdeps=y less effective if it is used to
>                                  # adjust merge order to prevent built_with_use() calls from
>                                  # failing.
> -                                bdeps_optional = True
> +                                pass
>                          else:
>                                  # built packages do not have build time dependencies.
>                                  edepend["DEPEND"] = ""
> +                                edepend["TDEPEND"] = ""
>  
>                  if removal_action and self._frozen_config.myopts.get("--with-bdeps", "y") == "n":
>                          edepend["DEPEND"] = ""
> +                        edepend["TDEPEND"] = ""
>  
>                  if removal_action:
> -                        bdeps_root = myroot
> +                        depend_root = myroot
>                  else:
> -                        bdeps_root = "/"
> -                        root_deps = self._frozen_config.myopts.get("--root-deps")
> -                        if root_deps is not None:
> -                                if root_deps is True:
> -                                        bdeps_root = myroot
> -                                elif root_deps == "rdeps":
> -                                        edepend["DEPEND"] = ""
> +                        depend_root = "/"
> +                        if pkg.metadata['EAPI'] in ('0', '1', '2', '3'):
> +                                # For ebuilds that have EAPI 4 or later, --root-deps
> +                                # has no effect, since DEPEND has been divided into two parts:
> +                                # DEPEND for build-time host dependencies, and TDEPEND for
> +                                # build-time target dependencies.
> +
> +                                root_deps = self._frozen_config.myopts.get("--root-deps")
> +                                if root_deps is not None:
> +                                        if root_deps is True:
> +                                                depend_root = myroot
> +                                        elif root_deps == "rdeps":
> +                                                edepend["DEPEND"] = ""
>  
>                  deps = (
> -                        (bdeps_root, edepend["DEPEND"],
> -                                self._priority(buildtime=(not bdeps_optional),
> -                                optional=bdeps_optional),
> +                        (depend_root, edepend["DEPEND"],
> +                                self._priority(buildtime=(not pkg.built),
> +                                optional=pkg.built),
> +                                pkg.built),
> +                        (myroot, edepend["TDEPEND"],
> +                                self._priority(buildtime=(not pkg.built),
> +                                optional=pkg.built),
>                                  pkg.built),
>                          (myroot, edepend["RDEPEND"],
>                                  self._priority(runtime=True),
> diff --git a/pym/_emerge/help.py b/pym/_emerge/help.py
> index 72bb56d..1c619af 100644
> --- a/pym/_emerge/help.py
> +++ b/pym/_emerge/help.py
> @@ -601,10 +601,17 @@ def help(myopts, havecolor=1):
>                          "ROOT instead of /. If the rdeps argument is given then discard " + \
>                          "all build-time dependencies of packages for ROOT. This option is " + \
>                          "only meaningful when used together with ROOT and it should not " + \
> -                        "be enabled under normal circumstances. For currently supported " + \
> -                        "EAPI values, the build-time dependencies are specified in the " + \
> -                        "DEPEND variable. However, behavior may change for new " + \
> -                        "EAPIs when related extensions are added in the future."
> +                        "be enabled under normal circumstances. For ebuilds that have EAPI " + \
> +                        "3 or earlier, the build-time dependencies are specified in the " + \
> +                        "DEPEND variable. For ebuilds that have EAPI 4 or later, " + \
> +                        "the emerge --root-deps option has no effect " + \
> +                        "since it is made obsolete by division " + \
> +                        "between DEPEND and TDEPEND. If EAPI 3 or " + \
> +                        "earlier ebuilds are used in combination with " + \
> +                        "EAPI 4 or later ebuilds, the --root-deps " + \
> +                        "behavior will still be applied to the EAPI " + \
> +                        "3 or earlier ebuilds (there is no behavior " + \
> +                        "change for ebuilds having older EAPIs)."
>                  for line in wrap(desc, desc_width):
>                          print(desc_indent + line)
>                  print()
> diff --git a/pym/portage/__init__.py b/pym/portage/__init__.py
> index cb84218..b55ff96 100644
> --- a/pym/portage/__init__.py
> +++ b/pym/portage/__init__.py
> @@ -418,7 +418,7 @@ def abssymlink(symlink):
>  
>  _doebuild_manifest_exempt_depend = 0
>  
> -_testing_eapis = frozenset()
> +_testing_eapis = frozenset(['4_pre0_tdepend',])
>  _deprecated_eapis = frozenset(["3_pre2", "3_pre1", "2_pre3", "2_pre2", "2_pre1"])
>  
>  def _eapi_is_deprecated(eapi):
> @@ -480,7 +480,7 @@ auxdbkeys = (
>          'RESTRICT',  'HOMEPAGE',  'LICENSE',   'DESCRIPTION',
>          'KEYWORDS',  'INHERITED', 'IUSE', 'REQUIRED_USE',
>          'PDEPEND',   'PROVIDE', 'EAPI',
> -        'PROPERTIES', 'DEFINED_PHASES', 'UNUSED_05', 'UNUSED_04',
> +        'PROPERTIES', 'DEFINED_PHASES', 'TDEPEND', 'UNUSED_04',
>          'UNUSED_03', 'UNUSED_02', 'UNUSED_01',
>  )
>  auxdbkeylen=len(auxdbkeys)
> diff --git a/pym/portage/dbapi/bintree.py b/pym/portage/dbapi/bintree.py
> index d840316..f11ce25 100644
> --- a/pym/portage/dbapi/bintree.py
> +++ b/pym/portage/dbapi/bintree.py
> @@ -55,7 +55,7 @@ class bindbapi(fakedbapi):
>                          ["BUILD_TIME", "CHOST", "DEPEND", "EAPI", "IUSE", "KEYWORDS",
>                          "LICENSE", "PDEPEND", "PROPERTIES", "PROVIDE",
>                          "RDEPEND", "repository", "RESTRICT", "SLOT", "USE", "DEFINED_PHASES",
> -                        "REQUIRED_USE"])
> +                        "REQUIRED_USE", "TDEPEND"])
>                  self._aux_cache_slot_dict = slot_dict_class(self._aux_cache_keys)
>                  self._aux_cache = {}
>  
> @@ -226,7 +226,7 @@ class binarytree(object):
>                                  ["BUILD_TIME", "CHOST", "DEPEND", "DESCRIPTION", "EAPI",
>                                  "IUSE", "KEYWORDS", "LICENSE", "PDEPEND", "PROPERTIES",
>                                  "PROVIDE", "RDEPEND", "repository", "SLOT", "USE", "DEFINED_PHASES",
> -                                "REQUIRED_USE"]
> +                                "REQUIRED_USE", "TDEPEND"]
>                          self._pkgindex_aux_keys = list(self._pkgindex_aux_keys)
>                          self._pkgindex_use_evaluated_keys = \
>                                  ("LICENSE", "RDEPEND", "DEPEND",
> @@ -252,7 +252,8 @@ class binarytree(object):
>                                  "SLOT"    : "0",
>                                  "USE"     : "",
>                                  "DEFINED_PHASES" : "",
> -                                "REQUIRED_USE" : ""
> +                                "REQUIRED_USE" : "",
> +                                "TDEPEND"        : "",
>                          }
>                          self._pkgindex_inherited_keys = ["CHOST", "repository"]
>                          self._pkgindex_default_header_data = {
> diff --git a/pym/portage/dbapi/porttree.py b/pym/portage/dbapi/porttree.py
> index 4b266b4..21edc57 100644
> --- a/pym/portage/dbapi/porttree.py
> +++ b/pym/portage/dbapi/porttree.py
> @@ -319,7 +319,7 @@ class portdbapi(dbapi):
>                  self._aux_cache_keys = set(
>                          ["DEPEND", "EAPI", "INHERITED", "IUSE", "KEYWORDS", "LICENSE",
>                          "PDEPEND", "PROPERTIES", "PROVIDE", "RDEPEND", "repository",
> -                        "RESTRICT", "SLOT", "DEFINED_PHASES", "REQUIRED_USE"])
> +                        "RESTRICT", "SLOT", "DEFINED_PHASES", "REQUIRED_USE", "TDEPEND"])
>  
>                  self._aux_cache = {}
>                  self._broken_ebuilds = set()
> diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
> index 6f587f7..d84b75c 100644
> --- a/pym/portage/dbapi/vartree.py
> +++ b/pym/portage/dbapi/vartree.py
> @@ -850,7 +850,7 @@ class vardbapi(dbapi):
>                          "EAPI", "HOMEPAGE", "IUSE", "KEYWORDS",
>                          "LICENSE", "PDEPEND", "PROPERTIES", "PROVIDE", "RDEPEND",
>                          "repository", "RESTRICT" , "SLOT", "USE", "DEFINED_PHASES",
> -                        "REQUIRED_USE"])
> +                        "REQUIRED_USE", "TDEPEND"])
>                  self._aux_cache_obj = None
>                  self._aux_cache_filename = os.path.join(self.root,
>                          CACHE_PATH, "vdb_metadata.pickle")
> --
> 1.7.1.1
>

Re: [cros-dev] Re: Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) Jonathan Kliegman 8/12/10 8:33 AM
I agree with Mandeep on this.  Its very intuitive to see DEPEND=what I need installed to run.  Switching what I've seen as its semantics can lead to the need to modify many ebuilds and cause a lot of confusion.  And if some of the dependencies are minor ones (such as I need package X installed to handle corner case Y but I'll build without it) then we can see a lot of subtle bugs accidentally introduced.

-Jon
Re: [cros-dev] Re: Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) Brian Harring 8/12/10 10:33 AM
On Thu, Aug 12, 2010 at 08:19:48AM -0700, Mandeep Singh Baines wrote:
> Zac Medico (zme...@gentoo.org) wrote:
> > I was working on writing documentation for the HDEPEND patch when I
> > realized that it seems simpler for ebuild developers if we divide the
> > dependencies as follows :
> >
>
> Thanks much for adding this support:) This will definitely make life
> much easier for folks who need to cross-compile. But I have one
> quick question.
>
> Why would TDEPEND be simpler? I suspect that target build dependencies
> (header files and static libraries) are more common than host build
> dependencies (build tools). Most packages have no host build dependencies
> other than the toolchain which is implicit.

Just a sidenote, but the implicit toolchain some of us have been
eyeing to kill for half a decade now- mainly since not having those
deps explicit has made it hard to do new profile bring up among other
things.   Implicit deps pulls info away from the resolver (which
originally couldn't handle that info, thus why it was never available)
and makes the system pkgset for better or worse the base.

Scenarios where people were trying to use uclibc caused some issues w/
that- I half expect any clang experiments will be similar also.


> So for most packages,
> you wouldn't need to do anything to support cross-compiling for the
> HDEPEND case. For the TDEPEND case, you'd have to modify a lot of
> packages. Especially, considering the fact that DEPEND=RDEPEND is the
> common case.
>
> The other nice property of HDEPEND is that DEPEND and RDEPEND are target
> and HDEPEND in host. With TDEPEND, DEPEND would be host while RDEPEND
> would be target. So HDEPEND seems more intuitive. But that might be
> more intuitive to me cuz I mostly cross-compile. Perhaps not more
> intuitive for someone who doesn't.

Offhand, I'd probably favor HDEPEND for this over TDEPEND since I
suspect HDEPEND is easier to sell gentoo devs on converting to- at
first glance it looks like it's less conversion effort.

That said, there is a lot of automake, cmake, qt-make, etc, style
deps that are in the tree so it's not going to be a "snap your
fingers and it's done" scenario.  I've not done any pquery scans
however- would be useful if someone did anyways, since the same list
used for that scan should be inclucded in the GLEP- specifically an
initial list of "these are very likely host deps vs target deps".

Corner case, but has anyone looked at how things like python
extensions would fit in this?  I'm specifically thinking of pkgs that
use distutils and friends to install (meaning they need a working
python) yet compile extensions for the target- roughly it would be
HDEPEND and DEPEND on python, but for that class of ebuilds I don't
see how it would be handleded cleanly w/in the ebuild itself.

~brian

Re: [cros-dev] Re: Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) Zac Medico 8/12/10 8:16 PM
Alright, I've adjusted the previous TDEPEND patch into an HDEPEND
patch. The only essential difference is a minor adjustment to the
logic inside depgraph.py. This patch applies on top of commit
ffe0e3fb6998c776b08f725c53dfe6c632ce7d02 of the master branch. You can
test it by creating ebuilds with EAPI=4_pre0_hdepend (this is a

temporary EAPI, intended just for testing purposes).

2010/8/12 Mandeep Singh Baines <m...@chromium.org>:

Re: [cros-dev] Re: Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) Tien-Ren Chen 8/16/10 7:12 AM
I just shared two documents with all recipient in this thread.
I planned to first reply to this thread to explain the purpose of that
two documents,
but I checked to send notification by accident. *ouch*

The document titled "Cross-compile EAPI draft" is to document the
proposed HDEPEND EAPI draft,
and added some possible required variables for proper supporting
cross-compiling.
Correction and comments are welcome, all of you in the CC list have
edit permission to it.

The other document titled "cross-compile troubles" describe the
trouble I met while hacking with Paludis.
Some of the problems are already properly fixed in the current
portage-based build flow that I'm just repeating the same fix with
Paludis,
while some of the problems are real problems that is hidden by
implementation-defined behavior of portage (for example, order of
merging package) or other toolchain, those packages should be fixed
properly for better deterministic behavior.
This document is more like a personal note. I'm trying to make it more
complete to document all the fix we did for cross-compile,
so it gives a better picture for new people to hack on our build
system, and also helps us to design the PMS to be more cross-compile
friendly.
Again, any correction and comments are welcome.

Re: [cros-dev] Re: Upstreaming emerge improvements (was: Goodbye Frankenbuild: New workflow for correct incremental builds) Anush Elangovan 8/24/10 3:43 PM
Tien,
   Would it be possible for you to try out the HDEPEND patch on a ChromiumOS build to replace hard-host-deps. I am not sure if that is what you had done for your initial feedback. Along with portage you will have to upgrade sandbox (2.2).  This way we can provide any other related feedback on the EAPI4 HDEPEND proposal. 

Btw, I think the document is no longer available so it may be worth sending a pdf too. 

Thanks
Anush