Problems building chromeos-chrome from LOCAL_SOURCE

[Steven Bennetts]

I'm attempting to build chromes-chrome for the first time in a week or so, and I am running into the following error. I did a 'repo sync', then rebuilt my chroot following the steps here: http://www.chromium.org/chromium-os/building-chromium-os/using-cros_workon#TOC-Create-a-chroot, up to and including the build_packages step, then executed the following:

$ CHROME_ROOT=/tmp/chrome_root CHROME_ORIGIN=LOCAL_SOURCE FEATURES="-usersandbox" emerge-x86-generic chromeos-base/chromeos-chrome

And got the following output as a result. There seems to be a problem executing gclient runhooks --force inside the chrroot. Any thoughts before I start to look deeper? My chrome tree is also recently updated.

Thanks,

-Steven

$ ./enter_chroot.sh --chrome_root=/home/stevenjb/local/chromium/chromium --chrome_root_mount=/tmp/chrome_root

$(chroot) CHROME_ROOT=/tmp/chrome_root CHROME_ORIGIN=LOCAL_SOURCE FEATURES="-usersandbox" emerge-x86-generic chromeos-base/chromeos-chrome

 * IMPORTANT: 1 news items need reading for repository 'gentoo'.

 * Use eselect news to read news items.

Calculating dependencies... done!

>>> Emerging (1 of 1) chromeos-base/chromeos-chrome-9999-r1029 from chromeos-overlay for /build/x86-generic/

!!! Manifest file not found: '/home/stevenjb/trunk/src/private-overlays/chromeos-overlay/chromeos-base/chromeos-chrome/Manifest'

 * CPV:  chromeos-base/chromeos-chrome-9999-r1029

 * REPO: chromeos-overlay

 * USE:  autotest build_tests elibc_glibc hardened kernel_linux tests_PyAutoFunctionalTests tests_desktopui_BrowserTest tests_desktopui_SyncIntegrationTests tests_desktopui_UITest userland_GNU x86

>>> Unpacking source...

 * CHROME_ORIGIN VALUE is LOCAL_SOURCE

>>> Source unpacked in /build/x86-generic/tmp/portage/chromeos-base/chromeos-chrome-9999-r1029/work

>>> Preparing source in /build/x86-generic/tmp/portage/chromeos-base/chromeos-chrome-9999-r1029/work/chromeos-chrome-9999 ...

 * /tmp/chrome_root should be set here properly

________ running '/usr/bin/python2.6 src/build/gyp_chromium' in '/tmp/chrome_root'

Updating projects from gyp files...

Traceback (most recent call last):

  File "../tools/grit/grit_info.py", line 94, in <module>

    sys.exit(main(sys.argv))

  File "../tools/grit/grit_info.py", line 78, in main

    inputs = Inputs(f)

  File "../tools/grit/grit_info.py", line 62, in Inputs

    files.extend(node.GetHtmlResourceFilenames())

  File "/tmp/chrome_root/src/tools/grit/grit/node/include.py", line 95, in GetHtmlResourceFilenames

    return grit.format.html_inline.GetResourceFilenames(self.FilenameToOpen())

  File "/tmp/chrome_root/src/tools/grit/grit/format/html_inline.py", line 211, in GetResourceFilenames

    return InlineFile(filename, None, None)

  File "/tmp/chrome_root/src/tools/grit/grit/format/html_inline.py", line 194, in InlineFile

    flat_text)

  File "/usr/lib64/python2.6/re.py", line 151, in sub

    return _compile(pattern, 0).sub(repl, string, count)

  File "/tmp/chrome_root/src/tools/grit/grit/format/html_inline.py", line 101, in SrcReplace

    return SrcInlineAsDataURL(src_match, filepath, distribution, inlined_files)

  File "/tmp/chrome_root/src/tools/grit/grit/format/html_inline.py", line 68, in SrcInlineAsDataURL

    inline_data = base64.standard_b64encode(ReadFile(filepath))

  File "/tmp/chrome_root/src/tools/grit/grit/format/html_inline.py", line 37, in ReadFile

    f = open(input_filename, 'rb')

IOError: [Errno 2] No such file or directory: u'browser/resources/../../app/theme/google_chrome/product_logo.png'

Traceback (most recent call last):

  File "src/build/gyp_chromium", line 97, in <module>

    sys.exit(gyp.main(args))

  File "src/tools/gyp/pylib/gyp/__init__.py", line 445, in main

    options.circular_check)

  File "src/tools/gyp/pylib/gyp/__init__.py", line 84, in Load

    depth, generator_input_info, check, circular_check)

  File "src/tools/gyp/pylib/gyp/input.py", line 2148, in Load

    depth, check)

  File "src/tools/gyp/pylib/gyp/input.py", line 422, in LoadTargetBuildFile

    includes, depth, check)

  File "src/tools/gyp/pylib/gyp/input.py", line 422, in LoadTargetBuildFile

    includes, depth, check)

  File "src/tools/gyp/pylib/gyp/input.py", line 380, in LoadTargetBuildFile

    build_file_path)

  File "src/tools/gyp/pylib/gyp/input.py", line 967, in ProcessVariablesAndConditionsInDict

    build_file)

  File "src/tools/gyp/pylib/gyp/input.py", line 982, in ProcessVariablesAndConditionsInList

    ProcessVariablesAndConditionsInDict(item, is_late, variables, build_file)

  File "src/tools/gyp/pylib/gyp/input.py", line 889, in ProcessVariablesAndConditionsInDict

    variables, build_file, 'variables')

  File "src/tools/gyp/pylib/gyp/input.py", line 967, in ProcessVariablesAndConditionsInDict

    build_file)

  File "src/tools/gyp/pylib/gyp/input.py", line 986, in ProcessVariablesAndConditionsInList

    expanded = ExpandVariables(item, is_late, variables, build_file)

  File "src/tools/gyp/pylib/gyp/input.py", line 653, in ExpandVariables

    (contents, p.returncode))

Exception: Call to 'python ../tools/grit/grit_info.py --inputs browser/browser_resources.grd common/common_resources.grd renderer/renderer_resources.grd' returned exit status 1. while loading dependencies of src/chrome/browser/sync/tools/sync_tools.gyp while loading dependencies of src/build/all.gyp while trying to load src/build/all.gyp

Error: /usr/bin/python2.6 src/build/gyp_chromium in /tmp/chrome_root returned 1

 * ERROR: chromeos-base/chromeos-chrome-9999-r1029 failed:

 *   Failed to run  /home/stevenjb/depot_tools/gclient runhooks

 * 

 * Call stack:

 *     ebuild.sh, line  54:  Called src_prepare

 *   environment, line 3154:  Called die

 * The specific snippet of code:

 *       ${EGCLIENT} runhooks --force || die "Failed to run  ${EGCLIENT} runhooks"

 * 

 * If you need support, post the output of 'emerge --info =chromeos-base/chromeos-chrome-9999-r1029',

 * the complete build log and the output of 'emerge -pqv =chromeos-base/chromeos-chrome-9999-r1029'.

 * This ebuild is from an overlay named 'chromeos-overlay': '/home/stevenjb/trunk/src/private-overlays/chromeos-overlay/'

 * The complete build log is located at '/build/x86-generic/tmp/portage/chromeos-base/chromeos-chrome-9999-r1029/temp/build.log'.

 * The ebuild environment file is located at '/build/x86-generic/tmp/portage/chromeos-base/chromeos-chrome-9999-r1029/temp/environment'.

 * S: '/build/x86-generic/tmp/portage/chromeos-base/chromeos-chrome-9999-r1029/work/chromeos-chrome-9999'

>>> Failed to emerge chromeos-base/chromeos-chrome-9999-r1029 for /build/x86-generic/, Log file:

>>>  '/build/x86-generic/tmp/portage/chromeos-base/chromeos-chrome-9999-r1029/temp/build.log'

 * Messages for package chromeos-base/chromeos-chrome-9999-r1029 merged to /build/x86-generic/:

 * CHROME_ORIGIN VALUE is LOCAL_SOURCE

 * /tmp/chrome_root should be set here properly

 * ERROR: chromeos-base/chromeos-chrome-9999-r1029 failed:

 *   Failed to run  /home/stevenjb/depot_tools/gclient runhooks

 * 

 * Call stack:

 *     ebuild.sh, line  54:  Called src_prepare

 *   environment, line 3154:  Called die

 * The specific snippet of code:

 *       ${EGCLIENT} runhooks --force || die "Failed to run  ${EGCLIENT} runhooks"

 * 

 * If you need support, post the output of 'emerge --info =chromeos-base/chromeos-chrome-9999-r1029',

 * the complete build log and the output of 'emerge -pqv =chromeos-base/chromeos-chrome-9999-r1029'.

 * This ebuild is from an overlay named 'chromeos-overlay': '/home/stevenjb/trunk/src/private-overlays/chromeos-overlay/'

 * The complete build log is located at '/build/x86-generic/tmp/portage/chromeos-base/chromeos-chrome-9999-r1029/temp/build.log'.

 * The ebuild environment file is located at '/build/x86-generic/tmp/portage/chromeos-base/chromeos-chrome-9999-r1029/temp/environment'.

 * S: '/build/x86-generic/tmp/portage/chromeos-base/chromeos-chrome-9999-r1029/work/chromeos-chrome-9999'

 * IMPORTANT: 1 news items need reading for repository 'gentoo'.

 * Use eselect news to read news items.

[Antoine Labour]

On Thu, Sep 30, 2010 at 3:50 PM, Steven Bennetts <stev...@chromium.org> wrote:

I'm attempting to build chromes-chrome for the first time in a week or so, and I am running into the following error. I did a 'repo sync', then rebuilt my chroot following the steps here: http://www.chromium.org/chromium-os/building-chromium-os/using-cros_workon#TOC-Create-a-chroot, up to and including the build_packages step, then executed the following:

$ CHROME_ROOT=/tmp/chrome_root CHROME_ORIGIN=LOCAL_SOURCE FEATURES="-usersandbox" emerge-x86-generic chromeos-base/chromeos-chrome

And got the following output as a result. There seems to be a problem executing gclient runhooks --force inside the chrroot. Any thoughts before I start to look deeper? My chrome tree is also recently updated.

Thanks,

-Steven

I think you're trying to build an official build of Google Chrome but only have checked out the public sources. You need to check out the internal ones as well. See recent thread on the internal mailing list.

Antoine

--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

[Chris Masone]

Did we ever get resolution on why we're having Googlers check out the
internal sources for Chrome OS?

[Steven Bennetts]

Thanks, that did the trick, or at least got me further. I do recall the thread, but the significance didn't quite stick. +1 to Chris's question, and/or, is there an easy way to do a build that doesn't require the internal sources?

-Steven

[David James]

On Thu, Sep 30, 2010 at 5:23 PM, Steven Bennetts <stev...@google.com> wrote:
>
> Thanks, that did the trick, or at least got me further. I do recall the thread, but the significance didn't quite stick. +1 to Chris's question, and/or, is there an easy way to do a build that doesn't require the internal sources?

Hi Steven,

Are you building Chrome OS, or Chromium OS?

Quoting the instructions (See
http://www.chromium.org/chromium-os/building-chromium-os/using-cros_workon):
> repo init -u http://git.chromium.org/git/manifest -m minilayout.xml
> # internal users use: repo init -u ssh://gitrw.chromium.org:9222/manifest-internal -m minilayout.xml

If you choose the internal version, then that means you also need the
internal version of Chrome. If you want to use the public layout then
use the public version above. (i.e. manifest, not manifest-internal)

Cheers,

David

[Chris Masone]

On Thu, Sep 30, 2010 at 5:27 PM, David James <david...@google.com> wrote:
> On Thu, Sep 30, 2010 at 5:23 PM, Steven Bennetts <stev...@google.com> wrote:
>>
>> Thanks, that did the trick, or at least got me further. I do recall the thread, but the significance didn't quite stick. +1 to Chris's question, and/or, is there an easy way to do a build that doesn't require the internal sources?
>
> Hi Steven,
>
> Are you building Chrome OS, or Chromium OS?
>
> Quoting the instructions (See
> http://www.chromium.org/chromium-os/building-chromium-os/using-cros_workon):
>> repo init -u http://git.chromium.org/git/manifest -m minilayout.xml
>> # internal users use: repo init -u ssh://gitrw.chromium.org:9222/manifest-internal -m minilayout.xml
>
> If you choose the internal version, then that means you also need the
> internal version of Chrome.

Which is absolutely not clear from those instructions :-/ Everyone at
Google is an "internal user", but most people working on the browser
build Chromium (as they should; we're an open source project).

Should we at Google all be building Google Chrome OS? If so, that's
fine, but we should lay out in the instructions a path that will
actually work. It seems to me that most of us don't do anything
specific to Google Chrome OS, and should actually be building Chromium
OS -- and so we should not be telling people to do otherwise.

[Antoine Labour]

On Thu, Sep 30, 2010 at 5:27 PM, David James <david...@google.com> wrote:

I think this problem comes up because the wording is confusing. "internal" users may not want to build the "official Google Chrome OS" builds, but this doesn't mean that's what it's doing.

3 things are confused here: whether or not you're a committer (so you use gitrw), whether or not you are a Google employee and have access to the internal bits (manifest-internal instead of manifest), and whether or not you want to build an official Google Chrome OS or Chromium OS. It actually has nothing to do with whether you're "internal" (of the Corp network ?) or not.

I think it would be greatly desirable to have access to the internal bits, but still be able to build Chromium OS. That's what happens in Chrome, where, even if you have checked out the Google Chrome binary bits, you have to opt in to build Google Chrome instead of Chromium.

Antoine

 

Cheers,

David

[David James]

On Thu, Sep 30, 2010 at 5:39 PM, Antoine Labour <pi...@chromium.org> wrote:
> I think this problem comes up because the wording is confusing. "internal"
> users may not want to build the "official Google Chrome OS" builds, but this
> doesn't mean that's what it's doing.
> 3 things are confused here: whether or not you're a committer (so you use
> gitrw), whether or not you are a Google employee and have access to the
> internal bits (manifest-internal instead of manifest), and whether or not
> you want to build an official Google Chrome OS or Chromium OS. It actually
> has nothing to do with whether you're "internal" (of the Corp network ?) or
> not.
> I think it would be greatly desirable to have access to the internal bits,
> but still be able to build Chromium OS. That's what happens in Chrome,
> where, even if you have checked out the Google Chrome binary bits, you have
> to opt in to build Google Chrome instead of Chromium.

That's certainly possible. We could add a flag for setup_board to
ignore private-overlays/chromeos-overlay. This would be trivial to
add. Feel free to file a bug against me for it.

Cheers,

David

[Jonathan Kliegman]

Antoine are you sure?

If I check out public Chromium and the internal repo of Chromium OS, I try to build Chrome.

I started a thread on this which unfortunately died.  The problem right now is that if you use the instructions on the website you will try to build internal versions.  Its somewhat masked as the default downloads are binary images which work.  The thread was sent internally on the Chrome OS list (since it was related to internal bits being checked out and not to the public code). 

-Jon

[Jonathan Kliegman]

I think we should go one step further.  The default checkout for Googlers should be the same manifest and overlay files as those checked out externally.  For those working on official builds or private code they can checkout the appropriate private manifest as needed.  But the default checkout should look the same.  If I'm working on code that will be checked in publicly I want to make sure that all of my development (except for final testing as appropriate) is done with the same bits as the public code has.

There are two major reasons for this:

1 - security.  If I'm testing with some internal bits available to me I can easily see a case where I accidentally post a debug log to the main mailing list when discussing my feature that contains confidential information

2 - speed.  There are less bits public than not  Checking out the private repo the other day cost me at least 1/2 GB of data (possibly more).  This causes everyones development to slow down if we're pulling bits we don't need.

I strongly believe we need to change the instructions and manifests such that the public code is pulled down unless there's a reason to access internal code.  Note that I'm not proposing we lockdown the private code - those who need it can still check it out with the current permissions.  Its just that people should explicitly download it rather than it just showing up.

-Jon

Cheers,

David

[Chris Masone]

On Thu, Sep 30, 2010 at 6:19 PM, Jonathan Kliegman <kli...@chromium.org> wrote:
> Antoine are you sure?
> If I check out public Chromium and the internal repo of Chromium OS, I try
> to build Chrome.

Antoine is referring to people who work on the browser and only the
browser. In the world of the browser team, you can have all the bits
checked out and build whichever you choose.

[Chris Masone]

I don't know that this is the right solution. It really seems like
the problem is in the mechanism by which we decide to build Google
Chrome into the image instead of Chromium. We just have an ebuild in
a private overlay with a higher number. Surely the way to deal with
this is not to say at the time I setup my toolchain "hey, by the way,
I'm not going to want to build Google Chrome for this board, ever"

I'd be surprised if there exists no better way to do this with virtual
packages or something.

>
> Cheers,
>
> David
>

[Anush Elangovan(அனுஷ்)]

On Fri, Oct 1, 2010 at 3:28 AM, Chris Masone <cma...@chromium.org> wrote:

On Thu, Sep 30, 2010 at 6:18 PM, David James <david...@google.com> wrote:
> On Thu, Sep 30, 2010 at 5:39 PM, Antoine Labour <pi...@chromium.org> wrote:
>> I think this problem comes up because the wording is confusing. "internal"
>> users may not want to build the "official Google Chrome OS" builds, but this
>> doesn't mean that's what it's doing.
>> 3 things are confused here: whether or not you're a committer (so you use
>> gitrw), whether or not you are a Google employee and have access to the
>> internal bits (manifest-internal instead of manifest), and whether or not
>> you want to build an official Google Chrome OS or Chromium OS. It actually
>> has nothing to do with whether you're "internal" (of the Corp network ?) or
>> not.
>> I think it would be greatly desirable to have access to the internal bits,
>> but still be able to build Chromium OS. That's what happens in Chrome,
>> where, even if you have checked out the Google Chrome binary bits, you have
>> to opt in to build Google Chrome instead of Chromium.
>
> That's certainly possible. We could add a flag for setup_board to
> ignore private-overlays/chromeos-overlay. This would be trivial to
> add. Feel free to file a bug against me for it.

I don't know that this is the right solution.  It really seems like
the problem is in the mechanism by which we decide to build Google
Chrome into the image instead of Chromium.  We just have an ebuild in
a private overlay with a higher number.

Yes the higher number hack is bad and fragile :( 

 

 Surely the way to deal with
this is not to say at the time I setup my toolchain "hey, by the way,
I'm not going to want to build Google Chrome for this board, ever"

setup_board shouldn't need another flag for this. Just a new internal profile which pulls in internal bits including chrome. Until we support "switching profiles" officially you may have to setup the board/profile again to build one or the other (ofcourse if you are comfortable with portage you will know how to workaround this i.e unmerge old virtual provider and emerge new provider but you need to watch for deps etc). 

 

I'd be surprised if there exists no better way to do this with virtual
packages or something.

Hopefully we will get to convert to using per board profiles soon (Anyone willing to help ?;) ) and also use a virtual/chrome (or maybe it should be called virtual/chromium)  to pull in the right browser based on the profile. 

Thanks

anush

 

>
> Cheers,
>
> David

[Chris Masone]

As a bandaid, I've taken a stab at updating the directions.  Please feel free to revise them in a more elegant fashion.

[Eric Shienbrood]

Right now, the choice about whether or not to use internal repos is
all or nothing, based on which manifest repo you use. Since I need to
use some internal repos for the 3G work, it means I also get chrome
instead of chromium. This doesn't matter unless I'm using
LOCAL_SOURCE, as I did last week when I ran across the same problem
that Steven had. I think we need more fine-grained control, to allow
us to use the minimal number of internal repos we need to do our jobs.

[Jonathan Kliegman]

Good point Eric.

I propose this:

1)  The normal manifests pulled down according to the website instructions include the minimal manifest needed to build the public external version

2)  cros_workon is extended to allow it to specify that you want to use an internal version (and also to allow you to switch back to the public).

3)  A "full internal" manifest is available which builds the internal/release (is there a difference?) version of all packages

This allows the majority of users who don't need anything internal to have the simplest and fastest path for development.

The users who only need one or two packages can now enable them individually

Those who need everything (buildbots or engineers) can also enable everything in a simple manner.

I picked cros_workon as it is the tool already used to specify packages to use locally and has the logic for how to find repos and modify both manifests and portage.  I'm ok with it being another tool, but putting it into cros_workon reduces the number of tools a developer needs to know about which is a good thing.

[Zdenek Behan]

There's a very easy step to make sure you'll never build the internal chrome again (per board), and stick with chromium:

echo ">chromeos-base/chromeos-chrome-9999-r1000" > /build/$board/package.mask/chromium

Well, until you decide to recreate your /build/$board, that is.

Besides, chrome is really one of a kind issue and probably doesn't require significant system changes to support it, rather some changes in the ebuild which will make distinguishing chrome from chromium easier, and also make it obvious for the developer which one are they building.