Disable building/running of tests?

[Greg Spencer]

How do I build packages without building tests?

I thought that if I just removed "test" from the FEATURES that it would do that, but it doesn't.  I tried adding "-test" to the features as well, to no avail.

Here's what I use to build release packages with:

--

$SCRIPT_DIR/enter_chroot.sh --chrome_root=${CHROME_DIR} -- \

BUILDTYPE="Release" \

RESTRICT="binchecks" \

FEATURES="strip\ -usersandbox" \

USE="gold" \

CHROME_ORIGIN="LOCAL_SOURCE" \

$CHROOT_SCRIPT_DIR/build_packages --board=${BOARD} --jobs=16

--

But it still spends copius amounts of time (>20mins) linking tests that I don't care to run.

-Greg.

[Greg Spencer]

Actually, it appears that it is building Chrome tests.  Is there any way to disable building of chrome tests when building from local source?

-Greg

[Zelidrag Hornung]

USE=-build_tests

--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

[Trevor Bourget]

yes, and i don't know why building tests is the default.
but you want USE=-build_tests
-- trevor

On Tue, 05 Oct 2010 11:47:13 -0700, Greg Spencer <gspe...@chromium.org>
wrote:

[Greg Spencer]

So (anyone), what the heck is the difference between FEATURES and USE and RESTRICT (which just seems to have the opposite sense of FEATURES and USE)?  And why aren't they all one variable with one set of options (and the ability to add a "-" to turn them off)?

Is there a way to list the various options for each of these, or do I just have to go spelunking in all the ebuild files?

-Greg.

[Trevor Bourget]

i think you will be looking for mandeep's recommendation to rtpm (read the
portage manual)
<http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=1> so
i save him the effort.

for a quick rationale, FEATURES is about the build system, USE and
RESTRICT are about the ebuild itself.

every entry does have the option to + and -, so please clarify the
question there if you have a specific issue.

i doubt there is a tool, there is a great feature request you could make,
that would document all the iuse flags available for all ebuilds available
in the overlay.
the features is part of portage itself. in addition to being part of the
handbook
<http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=3> you
can also use 'man make.conf' inside of the chroot.

-- trevor

On Tue, 05 Oct 2010 11:58:53 -0700, Greg Spencer <gspe...@chromium.org>
wrote:

--
Using Opera's revolutionary email client: http://www.opera.com/mail/

[Mandeep Singh Baines]

Greg Spencer (gspe...@chromium.org) wrote:
> So (anyone), what the heck is the difference between FEATURES and USE and
> RESTRICT (which just seems to have the opposite sense of FEATURES and USE)?
> And why aren't they all one variable with one set of options (and the
> ability to add a "-" to turn them off)?
>
> Is there a way to list the various options for each of these, or do I just
> have to go spelunking in all the ebuild files?
>

There are quite a few man pages in addition to the some excellent
docs referred to here:

http://dev.chromium.org/chromium-os/how-tos-and-troubleshooting/portage-build-faq#TOC-Where-to-look-for-details.

and here:

http://dev.chromium.org/chromium-os/building-chromium-os/portage-based-build#TOC-Further-Reading

The following docs are required reading:

http://devmanual.gentoo.org
http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml

Man pages:
==========

FEATURES (Global config options in general) :

$ man 5 make.conf

RESTRCIT (ebuild config options in general):

$ man 5 ebuild

Portage (build system in general):

$ man 5 portage

Emerge command:

$ man 1 emerge

Ebuild command:

$ man 1 ebuild

Equery command (super awesome command for querying package db):

$ man 1 equery

Show me all the use flags for a particular ebuild

$ equery-<board> uses <ebuild-name>

I often google search using one of the following conditions:

site:gentoo.org
site:devmanual.gentoo.org
site:bugzilla.gentoo.org

Hope that helps:)

Mandeep

[Greg Spencer]

On Wed, Oct 6, 2010 at 11:22 AM, Mandeep Singh Baines <m...@chromium.org> wrote:

[large list of docs...]

Hope that helps:)

Yes, that does help, and a sincere thank you for the list.

[the following comes from a genuine interest in making things better -- I know how f-ing hard it is to get all this stuff to line up, so please don't read it like a rant]

I guess my lament here is that as a developer it's waaaaay complex to build our stuff, and there's no single place to get information.  I'm totally used to learning lots of things to work on a large project: it's not the amount of reading (although it's a toss-up whether or not I'll be done reading by the time we release! LOL), it's just getting impossible to hold it all in my head and to keep up to date on the way to build things each week.

I've been writing a command line tool for my own use that allows me to do my builds without having to remember all the flags I need to use.  It now has almost 300 lines of code in it, and all it does is build release or debug of chromeos with chrome from local sources.  I find myself editing how this script works at least once a week, either because I learned something new or because something changed. I'm happy to try and write something for general use that does this, but it seems that would just be yet another band-aid wrapper script, and that's not what we need.

It seems to me that something is broken if I have to write my own script to build: I should be able to build the ~10 most common build types with a single command and three or less arguments that don't keep changing from week to week and have reasonable defaults.  I should be able to start with an empty directory, type one command, walk away, and come back to a built image and local sources I can modify (hopefully not two days later! :-).  I should be able to re-run that command and get a fast incremental build that only builds the things that changed. If I'm working on something esoteric, then fine, let me specify more arguments.

Is the problem that we are ALL building something that would be considered esoteric by someone else on the team?  And if that's the case, are we sure we're testing all of these esoteric combinations in try bots, etc?

Is the problem that we just haven't converged yet on one tool (well, obviously this is one problem, but I mean: will it all go away when we do, and will it ever really happen)?

Part of my frustration comes from the heterogeneous  environment and the home-grown scripts that wrap them.  I currently have five tools that deal with repos (git, svn, repo, gclient, cros_workon), all of which I still have to use, and remember which directories require which tools, and know how they all interact.  I have two different build systems I need to deal with (make, scons), and three meta-build systems (gyp, portage, configure).  All of them have reams of parameters, switches and modes where some are "built in" and some are added locally, and some are configured per build or computed on the fly from something else -- there is no central place to find out what they all are, let alone document what each one implies.  And then there's the rest of the home grown wrappers (build_packages, build_image, make_chroot) that all have their part in building stuff.

I realize we build a lot of configurations, but shouldn't it be possible to classify orthogonal categories for the configuration parameters (e.g. board type, debug/release, test/no test etc) and just specify one or two arguments to set the kind of build you want, defaulting to the most common type for each parameter?

After thinking about this for a whole five minutes, my ideal environment is something like:

- use only git to deal with repos and create, upload and submit CLs (and I would love it if we had only one git repo, but I know that ship has sailed.)

- use one other tool (probably homegrown) to build our entire source base (chrome + chromeos) soup to nuts, with a minimum of clear, orthogonal, (self) documented, immutable arguments.

When building it should build the fastest possible build that meets the configuration I specified (e.g. if I specify tests, then build the tests, but don't otherwise).  It shouldn't build anything extra, and it shouldn't run anything (e.g. tests) unless I ask it to.  I don't want to have to optimize the build steps myself, because it means every team member has to separately learn what to do to optimize it when the person who thought up the optimization could just have added it to the build tool.

There should be a way to 'name' popular build types so that groups of parameters don't need to be specified (e.g. I should be able to build a vanilla chromeos release image with one argument).  bonus points if I can set up my own names.

Basically, I think we've come to a place where only a person who knows hundreds (gross exaggeration, perhaps :) of different tools and languages can possibly work effectively in our codebase, and I'd like it to be more like 20.

-Greg.

[Mandeep Singh Baines]

"Make everything as simple as possible, but no simpler."
- Albert Einstein

Sincere apologies for all the churn on build but I think we're
past all that now. Thanks to anush, davidjames, sosa, zbehan, tgao, and
nsanders we now have a pretty amazing build system. We are able to
build 300+ packages from source using safe (i.e. -fPIE) CFLAGS, for
two different architectures, supporting multiple boards and multiple
different build targets and have a pre-flight build bot (aka
commit queue) which guarantees that a fresh build always works.

For the most part we're using two off-the-shelf tools: repo and portage.
repo is used by Anroid as well as many of its partners and has users
outside of the android community. portage has had 10 tens of development
with thousands of users and a ton of google searchable docs and some
great man pages.

We decided to abstract the mastering step a bit and so wrote our
own scripts there: build_package and build_image. These scripts
are kinda ugly so folks are working on cleaning that up but that won't
happen for a while (mostly to reduce build system churn).

We also created cros_workon to let the build system know which packages
to build from local source and which packages to build from known good
sources.

While gentoo's docs are great ours are OK. So we appreciate it when
folks update/clarify them when they are incorrect.

Sounds like you are really passionate about build and have ideas. The
build team is always looking for 20% help:)

There has been a lot of churn on build over the last few months but its
all been for the better. I think we're at a "happy" place now (minus some
doc cleanup) so would prefer not to rip it all out now.

Folks just need to understand a few commands:

* make/enter_chroot
* build_packages
* build_image
* cros_workon
* git
* repo
* git_cl

Each command solves one problem. There is work in progess to sqash the
first three and the last two. So we should be down to 4 commands.
But not in the short term.

The learning curve for ebuilds is a little steep but necessary. Its
solving a difficult problem. But its well designed and comes with
great documenation (not to mention man pages).

[Chris Masone]

First, let me say that the tree is much more stable than it was three months ago, and there are many kudos to go around here.  That said, there are some flaws in what you say below...

 

For the most part we're using two off-the-shelf tools: repo and portage.

From your perspective, this may be true.  I think one of the big problems our team has yet to solve is the disconnect between the half of the team that does most of their work in chrome and the half of the team that does not.  Building chrome from LOCAL_SOURCE is not a fringe use case, it's something many of us do all the time, and getting those builds into the image has perennially been awkward.  Those of us who straddle this line DO have to deal with gclient and svn and git and repo.  Thus, extra complexity elsewhere is more aggravating because we're already dealing with the cognitive load of multiple workflows.

repo is used by Anroid as well as many of its partners and has users
outside of the android community. portage has had 10 tens of development
with thousands of users and a ton of google searchable docs and some
great man pages.
 

We decided to abstract the mastering step a bit and so wrote our
own scripts there: build_package and build_image. These scripts
are kinda ugly so folks are working on cleaning that up but that won't
happen for a while (mostly to reduce build system churn).

We also created cros_workon to let the build system know which packages
to build from local source and which packages to build from known good
sources.

While gentoo's docs are great ours are OK. So we appreciate it when
folks update/clarify them when they are incorrect.

Come on, man...Gentoo's docs are great for open-source documentation, ours are sometimes inaccurate and not-infrequently stale :-/

 

Sounds like you are really passionate about build and have ideas. The
build team is always looking for 20% help:)

There has been a lot of churn on build over the last few months but its
all been for the better. I think we're at a "happy" place now (minus some
doc cleanup) so would prefer not to rip it all out now.

I don't think anyone's saying rip it out, I think people are saying that we're not in a place where Chrome OS is pleasant to work on yet.

 

Folks just need to understand a few commands:

* make/enter_chroot
* build_packages
* build_image
* cros_workon
* git
* repo
* git_cl

Note, though, that none of those are development tools.  They're source control and packaging tools.  I also have to understand what I need to do to develop the packages I work on.  For various reasons that totally made sense at the time of authoring, knowing how to edit/compile/test flimflam, entd, window_manager, update_engine, etc, etc, etc are all somewhat different.  There's still no standard way for me to compile and test a package I'm working on without ebuild-x86-generic'ing it.  That makes it take longer for me to do my job, and contributes to making Chrome OS frustration to develop.  And that's not even touching on the extra memes I need to understand to write and run autotests.

We need to bring homogeneity to our packages.  Pick scons or make.  Provide some shared infrastructure so that every package doesn't roll its own mechanism to pull in state from the build environment.  Get everyone's unit tests to be runnable in the same way.  We need to provide a usable developer workflow that doesn't involve packaging.  Our tools right now optimize for the need to build and package a distro, which makes sense because that was absolutely the P0.  Now, though, we need to acknowledge that this has slowed down our ability to develop Chrome OS and start optimizing that.

 

Each command solves one problem. There is work in progess to sqash the
first three and the last two. So we should be down to 4 commands.
But not in the short term. 

The learning curve for ebuilds is a little steep but necessary. Its
solving a difficult problem. But its well designed and comes with
great documenation (not to mention man pages).

> When building it should build the fastest possible build that meets the
> configuration I specified (e.g. if I specify tests, then build the tests,
> but don't otherwise).  It shouldn't build anything extra, and it shouldn't
> run anything (e.g. tests) unless I ask it to.  I don't want to have to
> optimize the build steps myself, because it means every team member has to
> separately learn what to do to optimize it when the person who thought up
> the optimization could just have added it to the build tool.
>
> There should be a way to 'name' popular build types so that groups of
> parameters don't need to be specified (e.g. I should be able to build a
> vanilla chromeos release image with one argument).  bonus points if I can
> set up my own names.
>
> Basically, I think we've come to a place where only a person who knows
> hundreds (gross exaggeration, perhaps :) of different tools and languages
> can possibly work effectively in our codebase, and I'd like it to be more
> like 20.
>
> -Greg.

[Mandeep Singh Baines]

Hmm, it would be interesting to see your script. You must be doing something
different from the workflow documented here:

http://sites.google.com/a/chromium.org/dev/chromium-os/building-chromium-os/using-cros_workon

My workflow looks like this:

Done once:
==========

$ repo init -u http://git.chromium.org/git/manifest -m minilayout.xml

$ repo sync

$ ./make_chroot

$ ./enter_chroot

(chroot) $ ./setup_board --board=x86-generic --default

(chroot) $ ./cros_workon start kernel

Done for every change:
======================

$ repo sync

$ repo start bug45 .

$ git commit -a

$ git cl upload --send-mail -r te...@example.com

$ git cl push

$ repo abandon bug45 .

Done for every compile:
=======================

(chroot) $ ./build_packages && ./build_image

(chroot) $ ./image_to_usb --test_image --from=blah --to=/dev/sdX

[Greg Spencer]

It's attached.  (Note that the locations for code that I use are hard coded, as is building chrome from local source, so it's not really general).

Admittedly, I am doing more than just the workflow (adding logging for instance), but my build_packages looks more like this:

$SCRIPT_DIR/enter_chroot.sh --chrome_root=${CHROME_DIR} -- \

      BUILDTYPE="Release" \

      RESTRICT="binchecks" \

      FEATURES="strip\\\ -usersandbox" \

      USE="-build_tests\\\ gold" \

      CHROME_ORIGIN="LOCAL_SOURCE" \

      $CHROOT_SCRIPT_DIR/build_packages --board=${BOARD} --jobs=16

Than just "./build_packages".

So, say 150 lines is real... but still.

-Greg.

[Mandeep Singh Baines]

Hmm, there are things that can be done to make emerge faster (ccache).
Otherwise, its trivial to wrap ebuild to take a package name instead of
a filename. The ebuild workflow (if folks need to use it) is pretty simple.
We just need to write a nice tutorial.

I guess the disconnect is that folks are using undocumented worfkflows.
If you are using ebuild-, write a doc on how you are using it. You'll
benefit others that are doing similar things. If your workflow
sucks, document it and point folks to it. There may be a better way.
Its difficult to solve a problem you don't know exists.

> We need to bring homogeneity to our packages. Pick scons or make. Provide
> some shared infrastructure so that every package doesn't roll its own
> mechanism to pull in state from the build environment. Get everyone's unit
> tests to be runnable in the same way. We need to provide a usable developer
> workflow that doesn't involve packaging. Our tools right now optimize for
> the need to build and package a distro, which makes sense because that was
> absolutely the P0. Now, though, we need to acknowledge that this has slowed
> down our ability to develop Chrome OS and start optimizing that.
>

What portage does is abstract the underlying build into a homogeneous
framework. It implements the adapter pattern. src_unpack, src_compile,
src_configure, etc, should do semantically the same thing regardless
of whether or not the underlying build is raw Makefiles, scons,
autotools, or a shell script.

I'm not sure standardizing on scons or make is feasible. We don't
have a choice on upstream bits or bits we went to get upstream. For
example, if you want to get a package into xorg, I believe it must
be autotools.

[Chris Masone]

No, that's not all that needs to happen.  I use ebuild in my own development cycle, and it's really no longer good enough once you get beyond a couple of source files.  Even with ccache, it takes nontrivially longer to do a recompile.  In order to debug my unit test binary, I have to chroot into the portage sandbox, which gets blown away every time I recompile.  For stuff like flimflam, which uses autotools, there's no portage-y way to avoid re-running autoconf every time you run ebuild.

In short, I'm already using the workflow you're recommending, and it's not meeting my needs.  

 

I guess the disconnect is that folks are using undocumented worfkflows.
If you are using ebuild-, write a doc on how you are using it. You'll
benefit others that are doing similar things. If your workflow
sucks, document it and point folks to it. There may be a better way.
Its difficult to solve a problem you don't know exists.

I have already pointed folks at it:-)  I've filed issues in tracker for it that are assigned to anush right now.

 

> We need to bring homogeneity to our packages.  Pick scons or make.  Provide
> some shared infrastructure so that every package doesn't roll its own
> mechanism to pull in state from the build environment.  Get everyone's unit
> tests to be runnable in the same way.  We need to provide a usable developer
> workflow that doesn't involve packaging.  Our tools right now optimize for
> the need to build and package a distro, which makes sense because that was
> absolutely the P0.  Now, though, we need to acknowledge that this has slowed
> down our ability to develop Chrome OS and start optimizing that.
>

What portage does is abstract the underlying build into a homogeneous
framework. It implements the adapter pattern. src_unpack, src_compile,
src_configure, etc, should do semantically the same thing regardless
of whether or not the underlying build is raw Makefiles, scons,
autotools, or a shell script.

Yes, but portage is a packaging tool.  It's not meant to be used to compile code that you're actively working on.  I can't just run src_compile twice in a row and have it do what I want, because it's not designed that way.  My choices are either: give up source control and work in portage's scratch directory so I can have fast incremental compiles, or blow away the source every time I compile and deal with the overhead of pulling every object out of ccache even though i didn't change it at all.   

Why? 

I'm not sure standardizing on scons or make is feasible. We don't
have a choice on upstream bits or bits we went to get upstream. For
example, if you want to get a package into xorg, I believe it must
be autotools.

I'm not talking about upstream stuff.  I'm talking about packages that we wrote here, which we control completely.

[Mandeep Singh Baines]

Cool. Doc URL or bug number?

These aren't the only choices. I'm sure we could come up with something
better. One idea would be symlink in the sources for worked on packages
if an configuration variable is set instead of copying them into portage's
temp directory. Anyway, I'm sure a good solution can be developed. Maybe
schedule some time for a brainstorming session?

[Chris Masone]

http://code.google.com/p/chromium-os/issues/detail?id=6843

http://code.google.com/p/chromium-os/issues/detail?id=6845

I was just saying that these are the only choices right now.

 

I'm sure we could come up with something
better. One idea would be symlink in the sources for worked on packages
if an configuration variable is set instead of copying them into portage's
temp directory. Anyway, I'm sure a good solution can be developed. Maybe
schedule some time for a brainstorming session?

I've already brainstormed with davidjames and anush a couple of times.  The issue is not that this is an insurmountable task, it's that no one is devoting cycles to it, and that some folks don't even agree that there's a problem here.

[Mandeep Singh Baines]

Thanks.

Actually, our super-intern zbehan did devote cycles to it. He disabled some
unnecessary elf checks in the chrome ebuild. He added support for
pbzip2 to speed up package create time and various other speedups I'm
forgetting right now.

He opened this bug here:

http://code.google.com/p/chromium-os/issues/detail?id=6665

I think he cut chrome build time down by at least half, probably
even more. His last comment in the bug is encouraging:

"There probably still is significant space for improvements."

[Chris Masone]

And that was awesome for speeding up building and packaging Chrome from scratch on the build bots.  However, it still takes vastly longer to do that than it does to just rebuild the one source file i modified in Chrome and relink it.

Like...why am I zipping anything at all if I just want to rebuild my binary and run the unit tests locally?

It remains the case that no one has spent any cycles on excising packaging time from our edit/compile/test cycle.  The fact that you hold up zdenek's work as an example of someone working on the problem indicates to me that we're still talking past one another :-/

[Anush Elangovan(அனுஷ்)]

On Wed, Oct 6, 2010 at 1:15 PM, Greg Spencer <gspe...@chromium.org> wrote:

On Wed, Oct 6, 2010 at 11:22 AM, Mandeep Singh Baines <m...@chromium.org> wrote:

[large list of docs...]

Hope that helps:)

Yes, that does help, and a sincere thank you for the list.

[the following comes from a genuine interest in making things better -- I know how f-ing hard it is to get all this stuff to line up, so please don't read it like a rant]

I guess my lament here is that as a developer it's waaaaay complex to build our stuff, and there's no single place to get information.  I'm totally used to learning lots of things to work on a large project: it's not the amount of reading (although it's a toss-up whether or not I'll be done reading by the time we release! LOL), it's just getting impossible to hold it all in my head and to keep up to date on the way to build things each week.

I've been writing a command line tool for my own use that allows me to do my builds without having to remember all the flags I need to use.  It now has almost 300 lines of code in it, and all it does is build release or debug of chromeos with chrome from local sources.  I find myself editing how this script works at least once a week, either because I learned something new or because something changed. I'm happy to try and write something for general use that does this, but it seems that would just be yet another band-aid wrapper script, and that's not what we need.

It seems to me that something is broken if I have to write my own script to build: I should be able to build the ~10 most common build types with a single command and three or less arguments that don't keep changing from week to week and have reasonable defaults.  I should be able to start with an empty directory, type one command, walk away, and come back to a built image and local sources I can modify (hopefully not two days later! :-).  I should be able to re-run that command and get a fast incremental build that only builds the things that changed. If I'm working on something esoteric, then fine, let me specify more arguments.

Is the problem that we are ALL building something that would be considered esoteric by someone else on the team?  And if that's the case, are we sure we're testing all of these esoteric combinations in try bots, etc?

Is the problem that we just haven't converged yet on one tool (well, obviously this is one problem, but I mean: will it all go away when we do, and will it ever really happen)?

Part of my frustration comes from the heterogeneous  environment and the home-grown scripts that wrap them.  I currently have five tools that deal with repos (git, svn, repo, gclient, cros_workon), all of which I still have to use, and remember which directories require which tools, and know how they all interact.  I have two different build systems I need to deal with (make, scons), and three meta-build systems (gyp, portage, configure).  All of them have reams of parameters, switches and modes where some are "built in" and some are added locally, and some are configured per build or computed on the fly from something else -- there is no central place to find out what they all are, let alone document what each one implies.  And then there's the rest of the home grown wrappers (build_packages, build_image, make_chroot) that all have their part in building stuff.

I realize we build a lot of configurations, but shouldn't it be possible to classify orthogonal categories for the configuration parameters (e.g. board type, debug/release, test/no test etc) and just specify one or two arguments to set the kind of build you want, defaulting to the most common type for each parameter?

After thinking about this for a whole five minutes, my ideal environment is something like:

- use only git to deal with repos and create, upload and submit CLs (and I would love it if we had only one git repo, but I know that ship has sailed.)

- use one other tool (probably homegrown) to build our entire source base (chrome + chromeos) soup to nuts, with a minimum of clear, orthogonal, (self) documented, immutable arguments.

Thanks for the good summary of your pain points as a developer. This is exactly what we want to address and move our build system towards.

We currently have a started a 20%ish project to enhance the build system called "Chromite". Our current build system did wonders for when it was first written (getting a full working cross compile environment etc). However now we need it to scale and be efficient for the developers. Firstly the build config exists in the build tool itself. For example this requires changing build_image for every new image type that we come across. 

Without going into all the problems here long term we want to do just what you asked. You check out the code, and you call the equivalent of "make <target>" and it spews out an image. Even stepping into a chroot should be optional (i.e the tool will drop into a chroot, and if a chroot isn't available, try to fetch the last know good chroot build or build it from source). It is not a silver bullet and doesn't address concerns of quick turnaround of development iteration cmasone brought up later in the thread nor does it solve the "bolt on" chrome on chrome OS dev model. 

Here is the Chromite "design" doc.

 https://docs.google.com/document/edit?id=1ie8RH3mTNxOOOBMGwKwswFkI5SMTBXnUzEyeDcQbgaU&hl=en 

So I would say the problem is acknowledged. We have a plan/design to address the problems (which we can review and iterate on). Now we just need developers to help out :)

We can split these tasks into 2 week chunks for developers  and they are welcome to take more than one chunk :)

Thanks

Anush

 

When building it should build the fastest possible build that meets the configuration I specified (e.g. if I specify tests, then build the tests, but don't otherwise).  It shouldn't build anything extra, and it shouldn't run anything (e.g. tests) unless I ask it to.  I don't want to have to optimize the build steps myself, because it means every team member has to separately learn what to do to optimize it when the person who thought up the optimization could just have added it to the build tool.

There should be a way to 'name' popular build types so that groups of parameters don't need to be specified (e.g. I should be able to build a vanilla chromeos release image with one argument).  bonus points if I can set up my own names.

Basically, I think we've come to a place where only a person who knows hundreds (gross exaggeration, perhaps :) of different tools and languages can possibly work effectively in our codebase, and I'd like it to be more like 20.

-Greg.

--

[Chris Masone]

On Wed, Oct 6, 2010 at 7:45 PM, Anush Elangovan(அனுஷ்) <an...@chromium.org> wrote:

On Wed, Oct 6, 2010 at 1:15 PM, Greg Spencer <gspe...@chromium.org> wrote:

On Wed, Oct 6, 2010 at 11:22 AM, Mandeep Singh Baines <m...@chromium.org> wrote:

[large list of docs...]

Hope that helps:)

Yes, that does help, and a sincere thank you for the list.

[the following comes from a genuine interest in making things better -- I know how f-ing hard it is to get all this stuff to line up, so please don't read it like a rant]

I guess my lament here is that as a developer it's waaaaay complex to build our stuff, and there's no single place to get information.  I'm totally used to learning lots of things to work on a large project: it's not the amount of reading (although it's a toss-up whether or not I'll be done reading by the time we release! LOL), it's just getting impossible to hold it all in my head and to keep up to date on the way to build things each week.

I've been writing a command line tool for my own use that allows me to do my builds without having to remember all the flags I need to use.  It now has almost 300 lines of code in it, and all it does is build release or debug of chromeos with chrome from local sources.  I find myself editing how this script works at least once a week, either because I learned something new or because something changed. I'm happy to try and write something for general use that does this, but it seems that would just be yet another band-aid wrapper script, and that's not what we need.

It seems to me that something is broken if I have to write my own script to build: I should be able to build the ~10 most common build types with a single command and three or less arguments that don't keep changing from week to week and have reasonable defaults.  I should be able to start with an empty directory, type one command, walk away, and come back to a built image and local sources I can modify (hopefully not two days later! :-).  I should be able to re-run that command and get a fast incremental build that only builds the things that changed. If I'm working on something esoteric, then fine, let me specify more arguments.

Is the problem that we are ALL building something that would be considered esoteric by someone else on the team?  And if that's the case, are we sure we're testing all of these esoteric combinations in try bots, etc?

Is the problem that we just haven't converged yet on one tool (well, obviously this is one problem, but I mean: will it all go away when we do, and will it ever really happen)?

Part of my frustration comes from the heterogeneous  environment and the home-grown scripts that wrap them.  I currently have five tools that deal with repos (git, svn, repo, gclient, cros_workon), all of which I still have to use, and remember which directories require which tools, and know how they all interact.  I have two different build systems I need to deal with (make, scons), and three meta-build systems (gyp, portage, configure).  All of them have reams of parameters, switches and modes where some are "built in" and some are added locally, and some are configured per build or computed on the fly from something else -- there is no central place to find out what they all are, let alone document what each one implies.  And then there's the rest of the home grown wrappers (build_packages, build_image, make_chroot) that all have their part in building stuff.

I realize we build a lot of configurations, but shouldn't it be possible to classify orthogonal categories for the configuration parameters (e.g. board type, debug/release, test/no test etc) and just specify one or two arguments to set the kind of build you want, defaulting to the most common type for each parameter?

After thinking about this for a whole five minutes, my ideal environment is something like:

- use only git to deal with repos and create, upload and submit CLs (and I would love it if we had only one git repo, but I know that ship has sailed.)

- use one other tool (probably homegrown) to build our entire source base (chrome + chromeos) soup to nuts, with a minimum of clear, orthogonal, (self) documented, immutable arguments.

Thanks for the good summary of your pain points as a developer. This is exactly what we want to address and move our build system towards.

We currently have a started a 20%ish project to enhance the build system called "Chromite". Our current build system did wonders for when it was first written (getting a full working cross compile environment etc). However now we need it to scale and be efficient for the developers. Firstly the build config exists in the build tool itself. For example this requires changing build_image for every new image type that we come across. 

Without going into all the problems here long term we want to do just what you asked. You check out the code, and you call the equivalent of "make <target>" and it spews out an image. Even stepping into a chroot should be optional (i.e the tool will drop into a chroot, and if a chroot isn't available, try to fetch the last know good chroot build or build it from source). It is not a silver bullet and doesn't address concerns of quick turnaround of development iteration cmasone brought up later in the thread nor does it solve the "bolt on" chrome on chrome OS dev model. 

Here is the Chromite "design" doc.

 https://docs.google.com/document/edit?id=1ie8RH3mTNxOOOBMGwKwswFkI5SMTBXnUzEyeDcQbgaU&hl=en 

So I would say the problem is acknowledged.

I don't know that I agree, since you admit that there's still no plan in "Chromite" for getting all of us back to a sane workflow in which we can develop at a reasonable speed.  I master images a lot less often than I build the code I'm developing.

[Mandeep Singh Baines]

Yeah, but it takes much less time than it did.

> Like...why am I zipping anything at all if I just want to rebuild my binary
> and run the unit tests locally?
>
> It remains the case that no one has spent any cycles on excising packaging
> time from our edit/compile/test cycle. The fact that you hold up zdenek's

Not sure its necessary to excise this. I think the pbzip2 work cut down
this time substationally.

> work as an example of someone working on the problem indicates to me that
> we're still talking past one another :-/
>

Bottom line: we want faster chrome build from ChromeOS, no? I think
you are getting too hung up on details of the solution rather than
solving the problem. Yeah, we do some unneccessary stuff. But if
that stuff is only consuming a fraction of the cycle time its OK.

I think the goal should be to reduce build time to the same order
as the incremental build time. I don't think its necessary to
have parity or do what an incremental build does and nothing more.
Just get close enough that's its no longer an issue.

If I'm interpretting zbehan's data correctly, we're currently at:

Noop emerge - 18m24
Noop make - 3m48s

So 4.5x. I think if we can get it down to 1.5 or 2x, we can put a fork
in it.

[Chris Masone]

...I really feel like we're not talking about the same problem here.  I am not talking about building Chrome to packaging into Chrome OS.  I'm talking about the edit/compile/test cycle for each individual project, one of which is chrome.  As such, it doesn't matter that packaging Chrome is faster than it used to be, because packaging it at all is a waste of time when all you want to do is compile the binary and then run it.

 

> Like...why am I zipping anything at all if I just want to rebuild my binary
> and run the unit tests locally?
>
> It remains the case that no one has spent any cycles on excising packaging
> time from our edit/compile/test cycle.  The fact that you hold up zdenek's

Not sure its necessary to excise this. I think the pbzip2 work cut down
this time substationally.

It is absolutely necessary.  I compile and test dozens of times as I work to write and debug my code.  Multiply that by every engineer on the entire team and we're wasting lots of time and money.

 

> work as an example of someone working on the problem indicates to me that
> we're still talking past one another :-/
>

Bottom line: we want faster chrome build from ChromeOS, no?

No, that's not the bottom line.  I feel like I've said this already, but perhaps I have not done so explicitly.  I'm not talking about building and packaging the entire project.  I am talking about the cycle of edit a file -> compile the binary -> test the binary.  Perhaps you can just edit code for a couple hours, compile it, and run the unit tests once and be done with it.  I'm not like that, and I don't think I'm alone.  An extra 30 seconds added to every compile adds up to a lot of wasted time.

 

I think
you are getting too hung up on details of the solution rather than
solving the problem. Yeah, we do some unneccessary stuff. But if
that stuff is only consuming a fraction of the cycle time its OK.

No, I think you're not understanding what the problem is that I'm talking about.  The "cycle time" I'm talking about is the time it takes me to incrementally compile the source I just edited.  This is normally on the order of seconds.

 

I think the goal should be to reduce build time to the same order
as the incremental build time. I don't think its necessary to
have parity or do what an incremental build does and nothing more.
Just get close enough that's its no longer an issue.

If I'm interpretting zbehan's data correctly, we're currently at:

Noop emerge - 18m24
Noop make - 3m48s

So 4.5x. I think if we can get it down to 1.5 or 2x, we can put a fork
in it.

This is not the problem I've been talking about.  If this is what you've been talking about the entire time, then we have had a large failure of communication.

[Jonathan Kliegman]

I just want to echo Chris's concern as it mirrors one of my own.

There is one specific workflow I have that needs optimization.  If I've done a cros_workon start foo and then repo sync'd, I'll have a local copy of source code I can start editing.

However if I run 'emerge-board foo', it does a full copy of the source code over and runs all the steps to build, package and install.

Now I run my test and realize that I need to change something in the source.  

I rerun 'emerge-board foo'.  It again copies everything over, reruns all the package setup and configuration and then lets me do my testing.

The extra setup, copies and configuration are all time wasted that add significant overhead.  I can easily spend 10x the time doing overhead (copy, configure, package, install) compared to work (rebuild one object file and relink).  

There are some hacks or workarounds which can help speed up some of this process.  But this workflow should be something that the tools, scripts and mechanisms support in an easy way (the workflow of edit source file and rebuilt just that object file then relink).  

There are many other improvements we can make to the build system.  Speeding up Chrome is good as its a major component (although one hopefully most users can avoid via binary packages).  Improving the packaging speed helps cut down the overhead for when the desired workflow doesn't work, but the step as a whole should be skipped.  These improvements and others are important - they speed up other workflows as well as the release builds which is a good thing.  But they don't solve this problem, they only reduce partially the overhead that should not be present.

-Jon

[Anush Elangovan(அனுஷ்)]

I understand the need for quicker iterative development as a developer (like we had discussed offline).  But as an OS vendor we also need a bullet proof   "mastering" or release process that is reproducible and consistent. Without that however fast our development is we could fail(in varying degrees) in actually releasing what we developed. 

So "Chromite" addresses one side of the coin. We can brainstorm and design a solution for iterative development. But without engineers committing to help it wont be done anytime soon (given the list of items to do). So would you be able help chip in if we started such an effort ? :)

[Mandeep Singh Baines]

Jonathan Kliegman (kli...@chromium.org) wrote:
> I just want to echo Chris's concern as it mirrors one of my own.
>
> There is one specific workflow I have that needs optimization. If I've done
> a cros_workon start foo and then repo sync'd, I'll have a local copy of
> source code I can start editing.
>
> However if I run 'emerge-board foo', it does a full copy of the source code
> over and runs all the steps to build, package and install.
>
> Now I run my test and realize that I need to change something in the source.
>
>
> I rerun 'emerge-board foo'. It again copies everything over, reruns all the
> package setup and configuration and then lets me do my testing.
>

Yeah, we can optimize here. Maybe have a NOCLEAN flag for cros_workon
packages. I have some ideas on how to implement this. The bug to
follow is:

http://code.google.com/p/chromium-os/issues/detail?id=6843

[David James]

On Wed, Oct 6, 2010 at 6:38 PM, Greg Spencer <gspe...@chromium.org> wrote:
> It's attached.  (Note that the locations for code that I use are hard coded,
> as is building chrome from local source, so it's not really general).
>
> Admittedly, I am doing more than just the workflow (adding logging for
> instance), but my build_packages looks more like this:
> $SCRIPT_DIR/enter_chroot.sh --chrome_root=${CHROME_DIR} -- \
>       BUILDTYPE="Release" \
>       RESTRICT="binchecks" \
>       FEATURES="strip\\\ -usersandbox" \
>       USE="-build_tests\\\ gold" \
>       CHROME_ORIGIN="LOCAL_SOURCE" \
>       $CHROOT_SCRIPT_DIR/build_packages --board=${BOARD} --jobs=16
> Than just "./build_packages".
> So, say 150 lines is real... but still.

Hi Greg,

Thanks for attaching this script.

It looks like building from LOCAL_SOURCE is quite complex. There are a
lot of options here that you're specifying on the command-line that
maybe should just be the default.

Here's a few of the options which should be unnecessary already:
1. RESTRICT="bincheck". We already disable the slow QA checks for
Chrome, thanks to Zdenek. So you don't need to specify this manually.
2. FEATURES="test". Does this do anything for Chrome? I believe
chrome uses the "build_tests" flag instead, so this should be
unnecessary.
3. --jobs=16. By default, parallel_emerge launches 1 job per core.
So no need to specify this anymore.

Here's a few places where we could improve the build system:
1. make_chroot should not be flaky. I see in your script that you
ignore the return code because you're concerned about flakiness. Can
you file a bug for this with the errors that you get?
2. setup_board should have a flag to 'setup if missing'. This will
save your script from flakiness in case the setup_board genuinely
fails.
3. We should have an official method for building 'Debug',
'DebugTest', and 'Release' options of Chrome. This way you don't need
to specify arguments on the command-line. This would handle things
like FEATURES="strip" and FEATURES="nostrip" for you and
FEATURES="-build_tests". It should also handle FEATURES="-usersandbox"
if that's really necessary.
4. Looks like you're using --chrome-root=$CHROME_DIR. This option
doesn't work very well because the chrome build tries to use hardlinks
to copy between that dir and /build/x86-foo, so you get "Invalid
cross-device link" warnings. We should find a better way for mounting
your source code into the chroot so that we recognize that it's on the
same filesystem as /build/x86-foo and can efficiently do hardlinks and
such.
5. USE="gold" should be the default soon. We are waiting until after
the official branch before we do this. This should be addressed as
part of 6665.

We should file bugs for 1-4. Greg, if you're interested in doing
this, please assign the bugs to me and mark with milestone R10.

I also have some suggestions on how to improve your script:
1. my_tee seems like a general useful utility. Maybe it should be
moved into a common area?
2. There's a lot of repeated flags for building Chrome. These should
be moved into a function or at least a variable so you don't need to
change it in every place.

And some questions:
- What's the sync_chrome function? Why do you need to 'sleep 15s' to
"let git catch up"? Do all Chrome developers need to do this or is
this just specific to your environment?

Cheers,

David

[Greg Spencer]

On Fri, Oct 8, 2010 at 8:50 AM, David James <david...@chromium.org> wrote:

Here's a few of the options which should be unnecessary already:

 1. RESTRICT="bincheck". We already disable the slow QA checks for
Chrome, thanks to Zdenek. So you don't need to specify this manually.
 2. FEATURES="test". Does this do anything for Chrome? I believe
chrome uses the "build_tests" flag instead, so this should be
unnecessary.
 3. --jobs=16. By default, parallel_emerge launches 1 job per core.
So no need to specify this anymore.

Thanks, this script was grown over time as the options changed, so of course there are out of date things.

If the "test" feature doesn't do anything, shouldn't that be an error/warning along the lines of "invalid feature specified"? 

Here's a few places where we could improve the build system:
 1. make_chroot should not be flaky. I see in your script that you
ignore the return code because you're concerned about flakiness. Can
you file a bug for this with the errors that you get?

OK.  It's not that it's flaky, really.  It just returns failure because of something that's more of a warning.

I think if any of the ebuilds gives a warning (or maybe only in the case of an error), then make_chroot

returns failure even though it printed out that it succeeded (and it probably did).  Unfortunately, I can't

reproduce that behavior anymore, so I can't tell you for sure.  I know at one point updating the chroot

gave some kind of warning/error output during an emerge, and it returned failure.  It could be that it's

fixed now.

 2. setup_board should have a flag to 'setup if missing'. This will
save your script from flakiness in case the setup_board genuinely
fails.

Yep, that'd be cool.

 

 3. We should have an official method for building 'Debug',
'DebugTest', and 'Release' options of Chrome. This way you don't need
to specify arguments on the command-line. This would handle things
like FEATURES="strip" and FEATURES="nostrip" for you and
FEATURES="-build_tests". It should also handle FEATURES="-usersandbox"
if that's really necessary.

Yes, and there's other configurations that would be good too (ReleaseTest, for instance).

This is along the lines of having aliases for common builds.

 

 4. Looks like you're using --chrome-root=$CHROME_DIR. This option
doesn't work very well because the chrome build tries to use hardlinks
to copy between that dir and /build/x86-foo, so you get "Invalid
cross-device link" warnings. We should find a better way for mounting
your source code into the chroot so that we recognize that it's on the
same filesystem as /build/x86-foo and can efficiently do hardlinks and
such.

That's the only way I know how to build it.  There's no other way (yet), right?

 

 5. USE="gold" should be the default soon. We are waiting until after
the official branch before we do this. This should be addressed as
part of 6665.

Cool.  Gold Rocks.

 

We should file bugs for 1-4.  Greg, if you're interested in doing
this, please assign the bugs to me and mark with milestone R10.

I'll file 'em.

 

I also have some suggestions on how to improve your script:
 1. my_tee seems like a general useful utility. Maybe it should be
moved into a common area?

Well, it's in my home directory, and I always use it instead of tee

(/home/gspencer/.cheats/bash/tee.bash), but I wanted to keep this script self contained.

You're welcome to steal it and put it somewhere useful.

 2. There's a lot of repeated flags for building Chrome. These should
be moved into a function or at least a variable so you don't need to
change it in every place.

You're talking about the CHROME_ORIGIN, BUILDTYPE, etc?  I thought about that too, but

1) each case has a different set of values and

2) the fact that they're not evaluated as actual bash variables (they're arguments to enter_chroot) makes the quoting kind of problematic (it would mean moving to a lower level of backslash hell if they were themselves variables).

And some questions:
 - What's the sync_chrome function? Why do you need to 'sleep 15s' to
"let git catch up"? Do all Chrome developers need to do this or is
this just specific to your environment?

This is some voodoo that I threw in there to see if it helped.  It doesn't, really.

Sometimes when I do the "checkout trunk -> pull -> checkout working -> rebase" cycle for chrome, the rebase fails in the middle because of "patch failed".  If I abort the rebase and do it again, it might fail later, and it might succeed.  Eventually, if I do it enough times (maybe 3 or rarely, 4), it succeeds.  One guess I had was that git was doing some kind of background "moving around" of things after exiting (or the process that had the lock hadn't truly exited yet or something), so I added a sleep.  To my knowledge, this doesn't actually happen, but I was stabbing in the dark.

-Greg.

[energy.con...@gmail.com]

I want to know why Greg Spencer picture comes up in my Google Hangouts.  His picture is embedded within my Google Hangouts page?  

On Tuesday, October 5, 2010 at 11:41:05 AM UTC-7, Greg Spencer wrote:

How do I build packages without building tests?

I thought that if I just removed "test" from the FEATURES that it would do that, but it doesn't.  I tried adding "-test" to the features as well, to no avail.

Here's what I use to build release packages with:

--

$SCRIPT_DIR/enter_chroot.sh --chrome_root=${CHROME_DIR} -- \

BUILDTYPE="Release" \

RESTRICT="binchecks" \

FEATURES="strip\ -usersandbox" \

USE="gold" \

CHROME_ORIGIN="LOCAL_SOURCE" \

$CHROOT_SCRIPT_DIR/build_packages --board=${BOARD} --jobs=16

--

[energy.con...@gmail.com]

Here is another look at Greg Spencer pic and other information within my google hangouts page.  I will tell you all that I have been hacked and harassed to the point that after I am finished writing this, I am going to tie a usb cord around my neck and triple knot it.  So, if you wanted to drive me to my grave, mission accomplished.  It has been 4 very long years of this never ending every minute of every hour of every day hacking..  destroying my relationships with my family friends, girls, driving me insane, lets not forget the 59 windows computers, 28 tablets, 19 Apple devices, 30 Smart Phones Android, 6 Chromebooks, 14 PC desktops, and dont forget getting me fired from 5 jobs,  driving me into poverty..  I just wished I could of had 2 minutes alone with whoever has been doing this to me, for I would of literally ripped pieces of their body off them and showed them to him.  I have nothing to lose, I am as dangerous as a person can get right at this moment.. Go to Superfish Victim on you tube to see videos and videos,, even when I walked into an Apple store.. the entire Apple Stores CRASHED... What did I do to anyone to deserve this??????

[Mike Frysinger]

your posts are unrelated to this (ancient) thread.  please seek assistance somewhere else.  if you google for "Google Product Forums", you'll find a bunch of places.

-mike

--