Re: [chromium-discuss] BINHOST vs GENTOO_MIRRORS

[Chris Sosa]

-chromium-discuss, +chromium-os-discuss

Wrong discuss, I think you'll get some better answers over on
chromium-os-discuss, sending.

On Mon, Feb 13, 2012 at 5:51 PM, Sam Xiao <s.x...@sta.samsung.com> wrote:
> Hi all,
>
> I'm looking for a way to make a local development mirror of the
> ChromiumOS.   I know the /etc/make.conf is where they store the
> information regarding to where it finds the pre-build binaries, but
> I'm confused as to what's the differences between GENTOO_MIRRORS and
> PORTAGE_BINHOST.
>
> I would like to host my local-built binaries for ChromiumOS, so it's
> much faster for engineers to download them locally than going all the
> way to Google.
>
>
>
> GENTOO_MIRRORS="https://commondatastorage.googleapis.com/chromeos-
> localmirror"
> GENTOO_MIRRORS="$GENTOO_MIRRORS https://commondatastorage.googleapis.com/chromeos-mirror/gentoo"
>
> source /usr/local/portage/chromiumos/chromeos/binhost/host/amd64-
> PREFLIGHT_BINHOST.conf
> FULL_BINHOST="https://commondatastorage.googleapis.com/chromeos-
> prebuilt/host/amd64/amd64-host/chroot-13.02.12.084905/packages/"
> PORTAGE_BINHOST="$PREFLIGHT_BINHOST $FULL_BINHOST"
>
> --
> Chromium Discussion mailing list: chromium...@chromium.org
> View archives, change email options, or unsubscribe:
>    http://groups.google.com/a/chromium.org/group/chromium-discuss

[Mike Frysinger]

from make.conf(5) man page:

PORTAGE_BINHOST = [space delimited URI list]
This is a list of hosts from which portage will grab prebuilt-binary
packages. <more stuff>

GENTOO_MIRRORS = [URIs]
Insert your space-separated list of local mirrors here. These
locations are used to download files before the ones listed in the
ebuild scripts.

so one is for prebuilt binary pkgs (*.tbz2) and the other is for
source tarballs (SRC_URI).
-mike

> --
> Chromium OS discuss mailing list: chromium-...@chromium.org

> View archives, change email options, or unsubscribe:

> http://groups.google.com/a/chromium.org/group/chromium-os-discuss?hl=en

[Sam Xiao]

Thanks!

Can I set override the PORTAGE_BINHOST to a different location before
I run "./setup_board --board=${BOARD}", but after "./chromite/bin/
cros_sdk" ?

The reason I need to do that because I want to setup a local mirror of
the binaries hosted for ChromiumOS, because my external network is
very slow, so I want to have a local mirror setup which syncs with
Google on a nightly basis, but everyone else uses a local internal
BINHOST when they do builds.

Please advise.
Thanks.
Sam.

> >> GENTOO_MIRRORS="$GENTOO_MIRRORShttps://commondatastorage.googleapis.com/chromeos-mirror/gentoo"

>
> >> source /usr/local/portage/chromiumos/chromeos/binhost/host/amd64-
> >> PREFLIGHT_BINHOST.conf
> >> FULL_BINHOST="https://commondatastorage.googleapis.com/chromeos-
> >> prebuilt/host/amd64/amd64-host/chroot-13.02.12.084905/packages/"
> >> PORTAGE_BINHOST="$PREFLIGHT_BINHOST $FULL_BINHOST"
>
> >> --

> >> Chromium Discussion mailing list: chromium-disc...@chromium.org

> >> View archives, change email options, or unsubscribe:
> >>    http://groups.google.com/a/chromium.org/group/chromium-discuss
>
> > --

> > Chromium OS discuss mailing list: chromium-os-disc...@chromium.org

[Mike Frysinger]

i'd suggest editing your /etc/make.conf.user in the cros-sdk chroot.
in there, do:
PORTAGE_BINHOST="http://some-local-samsung-url/ $PORTAGE_BINHOST"

this way your local binary host will get searched first and if a
binpkg isn't found there, it'll fall back to the standard google ones
-mike

> Chromium OS discuss mailing list: chromium-...@chromium.org

[Sam Xiao]

Thanks Mike.

How does the portage system determines which one is the newer binary?
by timestamp or by the version string in the filename ?

Thanks.
Sam.

[Mike Frysinger]

i'm not sure what you mean. when portage searches for a binpkg, it
walks the listed URI bases until it gets a hit. so it downloads the
first match.
-mike

> Chromium OS discuss mailing list: chromium-...@chromium.org

[Sam Xiao]

Let me clarify a bit, sorry for confusing you.

What if the BINHOST in make.conf.user has binaries that are older (by
version or by timestamp) than the one from Google, which one does it
pick first?

[Mike Frysinger]

if there are binpkgs already on the local system, then portage won't
even look at the BINHOST
-mike

On Tue, Feb 14, 2012 at 16:21, Sam Xiao <s.x...@sta.samsung.com> wrote:
> Let me clarify a bit, sorry for confusing you.
>
> What if the BINHOST in make.conf.user has binaries that are older (by
> version or by timestamp) than the one from Google, which one does it
> pick first?
>
>
> On Feb 14, 1:15 pm, Mike Frysinger <vap...@chromium.org> wrote:
>> i'm not sure what you mean.  when portage searches for a binpkg, it
>> walks the listed URI bases until it gets a hit.  so it downloads the
>> first match.
>>

[Sam Xiao]

Thanks Mike.
But what about "local internal BINHOST" or "Google BINHOST" has newer
binaries, will the local system look at them?

And how do Portage determines which one to use? Is it by timestamp or

the version string in the filename?


Thanks.
Sam.

[Mike Frysinger]

this is as i said: portage walks the URI's found in $PORTAGE_BINHOST.
the first one that has the file requested is used and then it stops
searching.

so if your internal URI is listed first, and has the binary, portage
will d/l from that and not even check the google ones.
-mike

On Tue, Feb 14, 2012 at 16:31, Sam Xiao <s.x...@sta.samsung.com> wrote:
> Thanks Mike.

> But what about "local internal BINHOST" or "Google BINHOST" has newer
> binaries, will the local system look at them?
>
> And how do Portage determines which one to use?  Is it by timestamp or
> the version string in the filename?
>
>
> Thanks.
> Sam.
>
> On Feb 14, 1:26 pm, Mike Frysinger <vap...@chromium.org> wrote:
>> if there are binpkgs already on the local system, then portage won't
>> even look at the BINHOST
>>

> --
> Chromium OS discuss mailing list: chromium-...@chromium.org

[Sam Xiao]

Thanks Mike.
I tried what you said, but looking at the log file, I can't confirm
the binaries are actually downloaded from my local BINHOST as it
contains nothing about the URI.

I use this command as a test, but the output doesn't say it's
downloaded from my local BINHOST as it has no URI information:
"emerge-x86-generic -G app-laptop/laptop-mode-tools"

What do you recommend I can use to confirm it's actually getting it
from my local BINHOST first?

Sorry, I'm also new to Gentoo/Emerge/Portage system.

Thanks.
Sam.

> > Chromium OS discuss mailing list: chromium-os-disc...@chromium.org

[Mike Frysinger]

hmm, make.conf.user isn't sourced as late as i thought it would be.
we might want to add a make.conf.user.last ...

in the mean time, simply doing this should work (run it as soon as you
enter the chroot and before you execute any commands):
export PORTAGE_BINHOST="http://someurl/"
-mike

> Chromium OS discuss mailing list: chromium-...@chromium.org

[Sam Xiao]

Hello Mike,

I did that before I ran "./setup_board --board=${BOARD}", but there's
still no way to confirm it is downloading from my internal BINHOST or
not by the log files. Anyway I can get that information in verbose
because I turn on verbose/debug mode and still can't find it.

Please advise.



>>> Emerging binary (1 of 1) chromeos-base/kernel-headers-2.6.38-r1 from chromiumos for /build/x86-generic/
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
100 594k 100 594k 0 0 89.5M 0 --:--:-- --:--:--
--:--:-- 96.7M
* kernel-headers-2.6.38-r1.tbz2 MD5 SHA1
size ;-) ... [ ok ]
>>> Extracting info
* Running stacked hooks for pre_pkg_setup
*
sysroot_build_bin_dir ...
[ ok ]
>>> Extracting chromeos-base/kernel-headers-2.6.38-r1
pbzip2: *WARNING: Trailing garbage after EOF ignored!

[Mike Frysinger]

you have to manually update the fetch commands to add verbose flags.  then when you do the `emerge`, it'll include info about the host its talking to.

-mike

--- a/chromeos/config/make.conf.common-target
+++ b/chromeos/config/make.conf.common-target
@@ -35,11 +35,11 @@ PORTDIR_OVERLAY="
PORTAGE_BZIP2_COMMAND="pbzip2"
PORTAGE_BUNZIP2_COMMAND="pbunzip2 --ignore-trailing-garbage=1"

-FETCHCOMMAND_GS="bash -c 'BOTO_CONFIG=/home/\${PORTAGE_USERNAME}/.boto gsutil cp \"${URI}\" \"${DISTDIR}/${FILE}\"'"
+FETCHCOMMAND_GS="bash -c 'BOTO_CONFIG=/home/\${PORTAGE_USERNAME}/.boto gsutil -d cp \"${URI}\" \"${DISTDIR}/${FILE}\"'"
-RESUMECOMMAND_GS="bash -c 'BOTO_CONFIG=/home/\${PORTAGE_USERNAME}/.boto gsutil cp \"${URI}\" \"${DISTDIR}/${FILE}\"'"
+RESUMECOMMAND_GS="bash -c 'BOTO_CONFIG=/home/\${PORTAGE_USERNAME}/.boto gsutil -d cp \"${URI}\" \"${DISTDIR}/${FILE}\"'"

-FETCHCOMMAND='curl -y 30 -f --retry 9 -L --output \${DISTDIR}/\${FILE} \${URI}'
+FETCHCOMMAND='curl -y 30 -f -v --retry 9 -L --output \${DISTDIR}/\${FILE} \${URI}'
-RESUMECOMMAND='curl -y 30 -f -C - --retry 9 -L --output \${DISTDIR}/\${FILE} \${URI}'
+RESUMECOMMAND='curl -y 30 -f -v -C - --retry 9 -L --output \${DISTDIR}/\${FILE} \${URI}'

# Print a checkpoint message every 10MB while archiving.
PORTAGE_BINPKG_TAR_OPTS="--checkpoint=1000"

[Sam Xiao]

Thanks Mike.
Will this "export PORTAGE_BINHOST" act as an overlay on top of the
Google's OR it's replacing it?

Thanks.
Sam.

[Mike Frysinger]

completely replace

-mike

--
Chromium OS discuss mailing list: chromium-...@chromium.org

[Sam Xiao]

Thanks Mike. You've helped a lot!
It seems to be working now.

> > Chromium OS discuss mailing list: chromium-os-disc...@chromium.org