chromeos-bmpblk takes an inordinate amount of build time

[Nathan D]

In recent months I've noticed that the chromeos-bmpblk packages is built from source everytime you run buld_packages and it takes a very long time. I was bothered enough to time it and on a 88 core build machine it took ~20 minutes (emerge-<board> chromeos-bmpblk). It's all python and shell scripts building fonts for a large set of languages every time. I'm sure this stuff doesn't change that often, so can it be made into a binary that just gets downloaded instead of building from source? 

Should we file a bug for this?

Thanks,

Nathan

[Mike Frysinger]

[ +pgeorgi ]

Patrick spent some time on this a while back.  i don't think manual built+uploaded fonts are the way we want to take things.

-mike

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

[Ciobanu, Nathan D]

+Shawn

chell is a good example but there are others (not sure if all).

-Nathan

---

From: Patrick Georgi [pge...@google.com]
Sent: Thursday, June 22, 2017 12:46 PM
To: Mike Frysinger
Cc: Ciobanu, Nathan D; Chromium OS dev
Subject: Re: [cros-dev] chromeos-bmpblk takes an inordinate amount of build time

I can take a look, but it would help to know which board is affected that badly. I think these scripts scale pretty badly with screen resolution, so it'll build much faster for the low-end devices with "HD-ready" screens than for those with 2K-4K screens.

--

Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

[Prathmesh Prabhu Chromium]

+chromeos-infra-discuss +Don Garrett 

I don't see anything fancy with the chromeos-bmpblk ebuild

(Why) are developers having to build this from source?

Is this a fallout of us not publishing prebuilts often enough from the CQ?

In general, developers don't have to care about packages they're not working on being costly to build -- they just get prebuilts by default. iiuc, the complaint here is that this package is getting built from source when the developer isn't actually working on it (no cros-workon start sys-boot/chromeos-bmpblk). Correct?

[nathan.d...@intel.com]

Hi Prathmesh,

On Thursday, June 22, 2017 at 1:46:13 PM UTC-7, Prathmesh Prabhu wrote:

+chromeos-infra-discuss +Don Garrett 

I don't see anything fancy with the chromeos-bmpblk ebuild

(Why) are developers having to build this from source?

Is this a fallout of us not publishing prebuilts often enough from the CQ?

In general, developers don't have to care about packages they're not working on being costly to build -- they just get prebuilts by default. iiuc, the complaint here is that this package is getting built from source when the developer isn't actually working on it (no cros-workon start sys-boot/chromeos-bmpblk). Correct?

Correct, no cros_workon start on anything, just a vanilla setup_board, build_packages.

[michael...@intel.com]

I grepped some build logs and found some boards are more prone to this issue than others.

The list below shows how many times chromeos-bmpblk was built from source in the following form:

<board> <# chromeos-bmpblk built from source>/<# total builds>

auron-yuna 0/113

cyan 0/113

falco 111/113

link 11/113

poppy 111/113

reef 0/113

samus 106/113

squawks 0/91

stumpy 10/113

The build times for chromeos-bmpblk range from 50 to 190 minutes.

[Mason, Michael W]

The 50-190 minutes came from build_packages output, so other things were definitely going on. However, if I just emerge chromeos-bmpblk on a relatively idle 88 core machine, it still takes a long time.  I'm not sure how to explain the difference between your measured time and mine. The more important question is why does chromeos-bmpblk have to be built from scratch so often on some boards and not on others.

time emerge-samus chromeos-bmpblk

real    21m22.952s
user    1239m28.428s
sys    9m40.832s

time emerge-stumpy chromeos-bmpblk

real    21m4.568s
user    1241m32.364s
sys    13m33.156s

---

From: Patrick Georgi [pge...@google.com]
Sent: Monday, June 26, 2017 10:48 PM
To: Mason, Michael W
Cc: Chromium OS dev; Ciobanu, Nathan D; Mike Frysinger; Shawn N; chromeos-infra-discuss; Don Garrett

Subject: Re: [cros-dev] chromeos-bmpblk takes an inordinate amount of build time

$ time emerge-samus chromeos-bmpblk

gave me:

real    1m2.597s

user    1m33.352s

sys     0m35.060s

Building it for stumpy was similar. That's so far from the 50-190 minutes that there must be something else going on.

[nathan.d...@intel.com]

Mike,

Why is there a chromeos-bmpblk-1.0.1-r29.ebuild in this package when there's already a symlink chromeos-bmpblk-0.0.3-r3.ebuild -> chromeos-bmpblk-0.ebuild ?

Here's the diff between chromeos-bmpblk-1.0.1-r29.ebuild and chromeos-bmpblk-9999.ebuild

5,6d4

< CROS_WORKON_COMMIT="ed30b5473c7973e2e77d8fae59a495da31fbd243"

< CROS_WORKON_TREE="6bb8c17e8177d008cb72cc78494f5c0bb58df62b"

93c91

< KEYWORDS="*"

---

> KEYWORDS="~*"

CROS_WORKON_COMMIT and CROS_WOKON_TREE should be in the chromeos-bmpblk-0.ebuild file and chromeos-bmpblk-1.0.1-r29.ebuild should be removed, no?

[Mike Frysinger]

the 0 and 0.0.3-r3 ebuilds are ignored and shouldn't matter here.  we should delete them as it looks like an oversight from earlier changes.

-mike

[Chintan Patel]

This happened recently to me also, building since 40 minutes on 8 Core machine

================

Still building chromeos-bmpblk-1.0.1-r30 (40m40.9s). Logs in /tmp/chromeos-bmpblk-1.0.1-r30-ESC3_G

================

Is there any workaround for this ?

You received this message because you are subscribed to the Google Groups "Chromium OS dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-dev+unsubscribe@chromium.org.

[nathan.d...@intel.com]

Hmmm, seems like the chromeos-kernel-3_18 also builds from src. I will post a full log with --showoutput shortly. 

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-d...@chromium.org.

[Mason, Michael W]

Finally got back to looking at this. I think I see why chromeos-bmpblk is not downloading a prebuilt. The Manifest file in chromiumos-overlay/sys-boot/chromeos-bmpblk refers to an old version. The ebuild is at version 1.0.1, but the Manifest still contains version 0.0.3. In all the other Manifest files I looked at, the ebuild and Manifest versions match.

Ebuild:
chromeos-bmpblk-1.0.1-r30.ebuild

Manifest:
DIST chromeos-bmpblk-0.0.3.tbz2 59446 RMD160 d6a6ea129d5c84404c3959d26614ceb2d2123f35 SHA1 9b38622940fb4d27e694d5d98d525d5cc6158926 SHA256 6920392e21632c41bb2c53ee13973ef22768f6cd43f47b938141d972c51fd8db

Should I file a bug for this?

Mike

---

From: chromiu...@chromium.org [chromiu...@chromium.org] on behalf of nathan.d...@intel.com [nathan.d...@intel.com]
Sent: Wednesday, July 12, 2017 11:43 AM
To: Chromium OS dev
Cc: vap...@chromium.org; Ciobanu, Nathan D; pge...@chromium.org

Subject: Re: [cros-dev] chromeos-bmpblk takes an inordinate amount of build time

[Mike Frysinger]

i don't think that's relevant.  the Manifest file is used to pull in files when the ebuild wants to build from source.  it is completely ignored when binpkgs are used instead.

-mike

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-dev+unsubscribe@chromium.org.

[Mason, Michael W]

I just discovered the --norebuild option in build_packages. It seems that the default is to rebuild the dependencies of a project that's going to be built, even if even if a satisfactory prebuilt binary exists for the dependency.  Is that true?  I tried "./build_packages --board=chell --norebuild" and it eliminated the building of chromeos-bmpblk, as well as several other packages, and significantly reduced the build time. Our overridden packages (kernel, coreboot, depthcharge, etc.) were still built from scratch.

This might be the fix we're looking for, but before we start using it, I'd like to understand why the default is to rebuild dependencies and if there are any negative side effects to using this option.

-MikeM

---

From: vap...@google.com [vap...@google.com] on behalf of Mike Frysinger [vap...@chromium.org]
Sent: Monday, July 31, 2017 10:39 AM
To: Mason, Michael W
Cc: Ciobanu, Nathan D; Chromium OS dev; pge...@chromium.org

[Mason, Michael W]

Turns out the chromeos-bmpblk build takes so long because there's no font cache in /usr/share/cache/fontconfig in the chroot. If I run fc-cache in cros_sdk to create the cache, then emerge-<board> chromeos-bmpblk, the build time goes from 45 minutes to 1.5 minutes. Can we add the font cache to the SDK by default?

-MikeM

---

From: chromiu...@chromium.org [chromiu...@chromium.org] on behalf of Mason, Michael W [michael...@intel.com]
Sent: Friday, August 18, 2017 12:03 PM
To: Mike Frysinger
Cc: Ciobanu, Nathan D; Chromium OS dev; pge...@chromium.org; Waterlander, Patrick
Subject: RE: [cros-dev] chromeos-bmpblk takes an inordinate amount of build time

[Mike Frysinger]

/usr/share/cache/fontconfig is already created when fontconfig is emerged ...

-mike

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-dev+unsubscribe@chromium.org.

[Mason, Michael W]

'emerge-<board> fontconfig' creates the fontconfig cache directory in chroot/build/<board>, but not the cache. 'emerge-<board> chromeos-fonts' creates the actual cache in that directory.

But that's not the issue. 'emerge-<board> chromeos-bmpblk' isn't looking for the cache in the board-specific root. It looks in chroot/usr/share/cache/fontconfig. That's may be wrong, but that's what it's doing.

-MikeM

---

From: vap...@google.com [vap...@google.com] on behalf of Mike Frysinger [vap...@chromium.org]

Sent: Thursday, September 07, 2017 4:01 PM
To: Mason, Michael W
Cc: Ciobanu, Nathan D; Chromium OS dev; pge...@chromium.org; Waterlander, Patrick

[Mike Frysinger]

yes, and fontconfig is already part of the sdk and is responsible for generating the cache in the / directory.  it isn't regenerated though when fonts are installed.

-mike

[michael...@intel.com]

I think I see why a fresh chroot doesn't have the font cache, but some existing chroots do.

Fontconfig and chromeos-fonts are installed during creation of the SDK, in that order. Fontconfig runs fc-cache to create the cache as part of pkg_postinst in the ebuild. Since chromeos-fonts hasn't been installed yet, there are no fonts to cache, so no cache files are created. Then chromeos-fonts installs the fonts and runs fc-cache again, except in the case of the SDK. The chromeos-fonts ebuild does not run fc-cache for the SDK under the assumption that the cache isn't needed. This is why a fresh SDK doesn't have the cache, but the board-specific builds do.

Later when setup_board is run, it checks the SDK against the current tree to see if any packages need to be updated. If fontconfig is outdated, it gets updated and fc-cache is run like before as part of pkg_postinst. This time, however, the fonts are there because chromeos-fonts is already  installed, so the cache files get created. This is probably why many people don't encounter the chromeos-bmpblk build time issue. Fontconfig was updated in their chroot at some point and the cache was created.

I submitted https://chromium-review.googlesource.com/#/c/669915/ to allow chromeos-fonts to run fc-cache during installation in the SDK.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-d...@chromium.org.

[Marc Herbert]

Mike, why doesn't Google seem to care much about  pango-view warming the climate these inordinate build durations? Is it because your buildbots never build from scratch? Because they avail infinite resources? Other?

[marc.h...@intel.com]

Mike implemented an internal automation workaround that manually calls "fc-cache" at the right time. While your mileage may vary, in one example this reduced last week one of our build times from 250 minutes to 100 minutes.

[yuchen...@asus.com]

Hi All,

I have the same problem to take amount time to build chromeos-bmpblk-1.0.1-r39. Do I still need to exec fc-cache in cros_sdk?

[Marc Herbert]

This happened to me again just now in an existing checkout after some repo sync, no idea why. About a dozen pango-view processes heating up the office space when (trying to) build coreboot.

A simple "sudo fc-cache" worked around it again, thx Michael.

[abhijee...@intel.com]

I would give a +1 to "me too!".
The package “chromeos-bmpblk-1.0.1-r54 “ consumed > 4Hrs while building package for octopus.

On Thursday, March 22, 2018 at 5:36:52 AM UTC+5:30, Marc Herbert wrote:

[Patrick Georgi]

Please check out https://chromium-review.googlesource.com/#/c/chromiumos/overlays/chromiumos-overlay/+/1075188

[nathan.d...@intel.com]

Thanks +Patrick Georgi for fixing this and thanks for all who contributed to this thread.

-Nathan