CrOS portage, gn/ninja, goma

[Alan Jones]

I was hoping somebody might be able to offer some perspective on the relationship between chromiumos and Chrome's build system. The boundary is pretty fuzzy for me, and I have not found any descriptive docs yet, so I am hoping somebody experienced might just be able to quickly guide me here.

I am fairly new to chromiumos, but I am pretty familiar with gentoo/portage, so I think I understand the whole package build and deploy mechanism I am seeing in build_packages and build_image. We are looking to understand chromiumos in the context of applying it to a new embedded system (no UI, no chrome), particularly looking at how to customize what goes into the image and how best to implement custom packages we will be developing.

My research suggests that gn (and gyp before it) is basically the chrome build system frontend, which produces ninja files, which is used to produce the final binary. Is this used for os-level packages and whatnot, or is it really just for chrome itself, plugins, etc...?

How does the gn build step interface with portage for a chromeos build? Is there just some ebuild somewhere which triggers gn and packages up the built products, or is it more tightly integrated somewhere? Is there some fast modify-build-test cycle involving both chrome and portage, or is portage bypassed for normal chrome development flows?

I am also trying to figure out where goma fits into the above. Is goma only useful for the chrome/ninja build part of things, or does it also integrate with general portage builds across all packages using make or whatever?

Is the rdep of virtual/chromeos-board-default-apps the right place to have our overlay declare what packages we want in our image beyond the platform?

A somewhat related question:

Is there anything which describes the structure of the root level ebuilds used to assemble the image, and ideally the rationale behind each step? I am finding it fairly hard to trace and reason about things just from digging through the ebuilds. Ideally this would be a document, but a tool to walk around the effective portage tree after all overlays might at least be an improvement over what I am doing.

Guidance on where the right places are to overlay in general I suppose.

Tracing things down the road to discover things like this tree:

• virtual/target-os ->
• virtual/target-chrome-os ->
• virtual/target-chromium-os
• virtual/chromeos-board-default-apps ->
• (board overlay)/virtual/chromeos-board-default-apps ->
• chromeos-base/chromeos-board-default-apps-(board)

add to this the extra dimensionality of overlays stacked up

• chromiumos:default/linux ->
• chromiumos:features/* ->
• chromeos ->
• baseboard-* ->
• chipset-* ->
• project-* ->
• private-project ->
• board ->
• private-board ->
• ... random stuff I have not discovered yet ...

it is a fairly time consuming process... and its hard to have any confidence that we found the right place for things in the end (I seem to discover some new virtual package every time I look through this thing).

Thanks

-Alan

[Mike Frysinger]

On Thu, Dec 8, 2016 at 4:32 PM, 'Alan Jones' via Chromium OS dev <chromiu...@chromium.org> wrote:

I was hoping somebody might be able to offer some perspective on the relationship between chromiumos and Chrome's build system. The boundary is pretty fuzzy for me, and I have not found any descriptive docs yet, so I am hoping somebody experienced might just be able to quickly guide me here.

I am fairly new to chromiumos, but I am pretty familiar with gentoo/portage, so I think I understand the whole package build and deploy mechanism I am seeing in build_packages and build_image. We are looking to understand chromiumos in the context of applying it to a new embedded system (no UI, no chrome), particularly looking at how to customize what goes into the image and how best to implement custom packages we will be developing.

build_packages runs emerge from portage on a single package (virtual/target-os).  that ebuild depends on a whole lot more packages and eventually you end up with a depgraph that is your entire OS.

each ebuild is responsible for declaring the source location, how to configure/compile/install the package, and what other packages that it needs in order to be built or run.

every time emerge runs on a package, it creates a binary package (binpkg/tbz2) that is cached.  when you run build_image, it tells emerge to use the binpkgs and the runtime depgraph of virtual/target-os to create a new/fresh image.  then build_image produces the final disk image.

gn/ninja are simply another build system like make or cmake or whatever.  build_packages/build_image/portage/emerge/whatever doesn't care.  all of those details are wrapped in a specific ebuild and emerge only sees the high level ebuild entry points (src_compile/etc...).

My research suggests that gn (and gyp before it) is basically the chrome build system frontend, which produces ninja files, which is used to produce the final binary. Is this used for os-level packages and whatnot, or is it really just for chrome itself, plugins, etc...?

Chromium used to use gyp everywhere, but they've deprecated it in favor of gn.  Chromium OS itself does not use gn or ninja to build ebuilds or emerge things.  whether any particular ebuild happens to use gn is up to that ebuild.

How does the gn build step interface with portage for a chromeos build? Is there just some ebuild somewhere which triggers gn and packages up the built products, or is it more tightly integrated somewhere?

in the case of the Chromium ebuild (chromeos-base/chromeos-chrome -- yes, not a great name), yes, it runs gn & ninja to do the actual build.

Is there some fast modify-build-test cycle involving both chrome and portage, or is portage bypassed for normal chrome development flows?

for people hacking on Chromium itself, people rarely use the ebuild.  instead, they follow:

http://dev.chromium.org/chromium-os/how-tos-and-troubleshooting/building-chromium-browser

I am also trying to figure out where goma fits into the above. Is goma only useful for the chrome/ninja build part of things, or does it also integrate with general portage builds across all packages using make or whatever?

goma is generally not used inside of ebuilds or inside of Chromium OS.  the simple chrome workflow referenced above though will leverage goma.

Chromium OS does have ccache support integrated to speed up repeat builds.  it also pulls down compatible binpkgs (where possible) that were uploaded by our buildbots so you don't even build anything locally (for the most part).

Is the rdep of virtual/chromeos-board-default-apps the right place to have our overlay declare what packages we want in our image beyond the platform?

no, don't touch that package :).  the "apps" part is referring to Chrome extensions/apps, not to packages that are part of the Chromium OS system.

usually your overlay should define a bsp ebuild and set up a virtual/chromeos-bsp to depend on that bsp ebuild.

A somewhat related question:

Is there anything which describes the structure of the root level ebuilds used to assemble the image, and ideally the rationale behind each step? I am finding it fairly hard to trace and reason about things just from digging through the ebuilds. Ideally this would be a document, but a tool to walk around the effective portage tree after all overlays might at least be an improvement over what I am doing.

Guidance on where the right places are to overlay in general I suppose.

at this time, probably not.  the best we have is this incomplete page:

http://dev.chromium.org/chromium-os/packages

Tracing things down the road to discover things like this tree:

• virtual/target-os ->
• virtual/target-chrome-os ->
• virtual/target-chromium-os
• virtual/chromeos-board-default-apps ->
• (board overlay)/virtual/chromeos-board-default-apps ->
• chromeos-base/chromeos-board-default-apps-(board)

add to this the extra dimensionality of overlays stacked up

• chromiumos:default/linux ->
• chromiumos:features/* ->
• chromeos ->
• baseboard-* ->
• chipset-* ->
• project-* ->
• private-project ->
• board ->
• private-board ->
• ... random stuff I have not discovered yet ...

it is a fairly time consuming process... and its hard to have any confidence that we found the right place for things in the end (I seem to discover some new virtual package every time I look through this thing).

yep, the stacking makes things _very_ flexible to the point it almost looks like metaprogramming.  maybe it is :).

-mike