Understanding emerge-$board and equery-$board uses

[asharif]

I had a chroot where I had emerged chromeos-chrome without USE="pgo_generate". I wanted to build an image with USE="pgo_generate" so I did the following:

USE="pgo_generate" emerge-lumpy -pv chromeos-chrome

./build_image --board=lumpy test # Should I have specified USE="pgo_generate" here?

# As per my understanding, it is not necessary, but I see it in cbuildbot logs.

I imaged my machine only to find out that the image did not have chromeos-chrome with USE="pgo_generate".

In my chroot I ran the following commands (pasted verbatim):

(cr) ((adf8a00...)) asharif@asharif ~/trunk/src/scripts $ equery-lumpy uses chromeos-chrome

[ Legend : U - final flag setting for installation]

[        : I - package is installed with flag     ]

[ Colors : set, unset                             ]

 * Found these USE flags for chromeos-base/chromeos-chrome-25.0.1363.3_rc-r1:

 U I

 - - asan               : Build with Address Sanitizer support

 + - autotest           : <unknown>

 + - autotest           : <unknown>

 + - build_tests        : Build test targets

 + - buildcheck         : <unknown>

 + - chrome_debug       : Build with debug symbols (-g)

 - - chrome_debug_tests : Don't strip test targets

 - - chrome_internal    : Add chrome branding

 - - chrome_media       : Build with additional codecs and containers

 - - chrome_pdf         : Build pdf reader

 + - chrome_remoting    : Build chrome remoting feature

 - - clang              : Build with CC and C++ set to clang

 - - component_build    : Build components as shared libraries (faster link)

 - - drm                : <unknown>

 + - gold               : Use gold linker

 - - hardfp             : <unknown>

 + - highdpi            : Add high DPI resources

 + - nacl               : Build Native Client (NaCL) support

 - - neon               : Use neon instructions (ARM only)

 - - ninja              : <unknown>

 - - oem_wallpaper      : <unknown>

 - - opengles           : <unknown>

 - - pgo_generate       : Generate profile-guided optimization data

 - - pgo_use            : Use profile data for profile-guided optimization

 + - reorder            : Re-order symbols using profile data

 + - runhooks           : Run build hooks

 + - verbose            : Verbose build output

 - - widevine_cdm       : Build widevine plugin

(cr) ((adf8a00...)) asharif@asharif ~/trunk/src/scripts $ USE="pgo_generate" emerge-lumpy -pv chromeos-chrome

These are the packages that would be merged, in order:

Calculating dependencies... done!

[ebuild   R    ] chromeos-base/chromeos-chrome-25.0.1363.3_rc-r1 to /build/lumpy/ USE="autotest build_tests buildcheck chrome_debug chrome_remoting gold highdpi nacl pgo_generate reorder runhooks verbose -asan -chrome_debug_tests -chrome_internal -chrome_media -chrome_pdf -clang -component_build -drm -hardfp -neon -ninja -oem_wallpaper -opengles -pgo_use -widevine_cdm" 0 kB [1]

Total: 1 package (1 reinstall), Size of downloads: 0 kB

Portage tree and overlays:

 [0] /usr/local/portage/stable

 [1] /usr/local/portage/chromiumos

(cr) ((adf8a00...)) asharif@asharif ~/trunk/src/scripts $ emerge-lumpy -pv chromeos-chrome

These are the packages that would be merged, in order:

Calculating dependencies... done!

[ebuild   R    ] chromeos-base/chromeos-chrome-25.0.1363.3_rc-r1 to /build/lumpy/ USE="autotest build_tests buildcheck chrome_debug chrome_remoting gold highdpi nacl reorder runhooks verbose -asan -chrome_debug_tests -chrome_internal -chrome_media -chrome_pdf -clang -component_build -drm -hardfp -neon -ninja -oem_wallpaper -opengles -pgo_generate* -pgo_use -widevine_cdm" 0 kB [1]

Total: 1 package (1 reinstall), Size of downloads: 0 kB

Portage tree and overlays:

 [0] /usr/local/portage/stable

 [1] /usr/local/portage/chromiumos

(cr) ((adf8a00...)) asharif@asharif ~/trunk/src/scripts $ 

This is inconsistent output. equery-$board uses thinks pgo_generate is disabled, while emerge-$board -pv $package without that flag thinks the package is installed with USE="pgo_generate". Who should I trust? Should one of {emerge-$board,equery-$board} be fixed to be consistent with the other? What's the canonical way of checking USE flags for a package?

Also, I'd appreciate if someone knowledgeable could answer about passing in USE flags to build_image.

[Jonathan Kliegman]

If I'm reading the log right you ran thee commands in this order:

equery-lumpy uses chromeos-chrome

USE="pgo_generate" emerge-lumpy -pv chromeos-chrome

emerge-lumpy -pv chromeos-chrome

The first showed that pgo_generate wasn't set.

The second shows that if you did run the command, it would set pgo_generate.  Which makes sense as you se USE=pgo-generate

The third command shows pgo_generate would not be set, which makes sense as that's the default and you didn't set a USE flag.

The '-pv' option to emerge means 'pretend, verbose'.  It doesn't actually emerge it.

If you run without the '-p' it should work.

As for build_image, I believe its configured to ignore USE flags and build with whatever package is available (assuming there aren't any USE flag dependencies that can't be met).  So running:
USE="pgo_generate" emerge-lumpy chromeos-chrome

./build_image --board=lumpy  test

should work fine.

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en
 
 
 

[asharif]

On Tue, Dec 18, 2012 at 6:25 PM, Jonathan Kliegman <kli...@chromium.org> wrote:

If I'm reading the log right you ran thee commands in this order:

equery-lumpy uses chromeos-chrome

USE="pgo_generate" emerge-lumpy -pv chromeos-chrome

emerge-lumpy -pv chromeos-chrome

Yes.

 

The first showed that pgo_generate wasn't set.

The second shows that if you did run the command, it would set pgo_generate.  Which makes sense as you se USE=pgo-generate

I don't think so. Note that there is no asterisk (*) next to pgo_generate. That means that emerge does not think that USE flag is different from the installed package.

 

The third command shows pgo_generate would not be set, which makes sense as that's the default and you didn't set a USE flag.

This command actually shows the asterisk, which means emerge thinks the installed and the one that I'm ordering are different.

 

The '-pv' option to emerge means 'pretend, verbose'.  It doesn't actually emerge it.

If you run without the '-p' it should work.

Sorry, in my initial email I added the -p erroneously.

Here is some more contradictory evidence (verbatim):

(cr) ((adf8a00...)) asharif@asharif ~/trunk/src/scripts $ cat /build/lumpy/var/db/pkg/chromeos-base/chromeos-chrome-25.0.1363.3_rc-r1/USE 

amd64 autotest build_tests buildcheck chrome_debug chrome_remoting elibc_glibc gold highdpi kernel_linux nacl pgo_generate reorder runhooks userland_GNU verbose

As for build_image, I believe its configured to ignore USE flags and build with whatever package is available (assuming there aren't any USE flag dependencies that can't be met).  So running:
USE="pgo_generate" emerge-lumpy chromeos-chrome

./build_image --board=lumpy  test

should work fine.

On Tue, Dec 18, 2012 at 8:45 PM, asharif <ash...@chromium.org> wrote:

I had a chroot where I had emerged chromeos-chrome without USE="pgo_generate". I wanted to build an image with USE="pgo_generate" so I did the following:

USE="pgo_generate" emerge-lumpy -pv chromeos-chrome

Sorry, I did not pass in -p here.

[asharif]

On Tue, Dec 18, 2012 at 6:25 PM, Jonathan Kliegman <kli...@chromium.org> wrote:

If I'm reading the log right you ran thee commands in this order:

equery-lumpy uses chromeos-chrome

USE="pgo_generate" emerge-lumpy -pv chromeos-chrome

emerge-lumpy -pv chromeos-chrome

The first showed that pgo_generate wasn't set.

The second shows that if you did run the command, it would set pgo_generate.  Which makes sense as you se USE=pgo-generate

The third command shows pgo_generate would not be set, which makes sense as that's the default and you didn't set a USE flag.

The '-pv' option to emerge means 'pretend, verbose'.  It doesn't actually emerge it.

If you run without the '-p' it should work.

As for build_image, I believe its configured to ignore USE flags and build with whatever package is available (assuming there aren't any USE flag dependencies that can't be met).  So running:
USE="pgo_generate" emerge-lumpy chromeos-chrome

./build_image --board=lumpy  test

should work fine.

It's interesting that the USE flags are being passed down in cbuildbot:

http://chromegw.corp.google.com/i/chromeos/builders/daisy%20canary/builds/662/steps/BuildTarget/logs/stdio

Example log above. Search for build_image on that page. I am worried that at some point build_image will be changed to maybe do something different with different USE flags. I think cbuildbot should not pass in USE flags to build_image.

[Jonathan Kliegman]

Looks like gmail may have lost some of the threading so sorry for responding out of context to some points (did you respond to me twice?  I only see one level of quoting from you on reply).

Also, I'm far from a portage expert so don't trust everything I say.  Its based on things I've seen in the past other cases.

I remember now that running 'equery uses foo' seems to rely on the USE flags set.  It seems to me to be more of a "if I built this, here's what I'd set".  For example:

(cr) ((642c3be...)) kliegs@kliegs-cros ~/trunk/src/scripts $ equery-daisy uses chromeos

[ Legend : U - final flag setting for installation]

[        : I - package is installed with flag     ]

[ Colors : set, unset                             ]

 * Found these USE flags for chromeos-base/chromeos-0.0.1-r180:

 U I

 - - coreboot  : <unknown>

(cr) ((642c3be...)) kliegs@kliegs-cros ~/trunk/src/scripts $ USE=coreboot equery-daisy uses chromeos

[ Legend : U - final flag setting for installation]

[        : I - package is installed with flag     ]

[ Colors : set, unset                             ]

 * Found these USE flags for chromeos-base/chromeos-0.0.1-r180:

 U I

 + + coreboot  : <unknown>

(other flags removed for brevity).

I seem to recall being told to use 'portageq' in the past and that might be more reliable (I haven't used it enough to know the syntax offhand, unfortunately).

I'm also curious - despite what the tools said, when you did the USE=... emerge-...  and then build_image - did that image end up with the correct package and USE flags actually built?  I think it would have given you the right package, just some confusing queries.

On Tue, Dec 18, 2012 at 9:40 PM, asharif <ash...@chromium.org> wrote:

On Tue, Dec 18, 2012 at 6:25 PM, Jonathan Kliegman <kli...@chromium.org> wrote:

If I'm reading the log right you ran thee commands in this order:

equery-lumpy uses chromeos-chrome

USE="pgo_generate" emerge-lumpy -pv chromeos-chrome

emerge-lumpy -pv chromeos-chrome

The first showed that pgo_generate wasn't set.

The second shows that if you did run the command, it would set pgo_generate.  Which makes sense as you se USE=pgo-generate

The third command shows pgo_generate would not be set, which makes sense as that's the default and you didn't set a USE flag.

The '-pv' option to emerge means 'pretend, verbose'.  It doesn't actually emerge it.

If you run without the '-p' it should work.

As for build_image, I believe its configured to ignore USE flags and build with whatever package is available (assuming there aren't any USE flag dependencies that can't be met).  So running:
USE="pgo_generate" emerge-lumpy chromeos-chrome

./build_image --board=lumpy  test

should work fine.

It's interesting that the USE flags are being passed down in cbuildbot:

http://chromegw.corp.google.com/i/chromeos/builders/daisy%20canary/builds/662/steps/BuildTarget/logs/stdio

Example log above. Search for build_image on that page. I am worried that at some point build_image will be changed to maybe do something different with different USE flags. I think cbuildbot should not pass in USE flags to build_image.

Hmm. I think I might've been the one to make the change to do that.  If I recall it was needed for a case where a package depended on another package being installed with a certain set of USE flags.  So if those flags were not set, then build_image would fail, even if the binary package was built with the flags. Portage needed to see them actually set (either through defaults, config files or environment variables).

Either way - I agree that we should always pass the same USE flags to build_packages, emerge and build_image.  I just don't think it mattered in this case.