dev tarballs lack profile data for PGO

[Olivier Tilloy]

Hi chromium packagers,

With the promotion of 87.0.4278.0 to the dev channel, linux builds
switched to PGO (from AFDO). When building offline from a source
tarball, the build fails early because profile data (*.profdata) files
cannot be found:

ERROR at //build/config/compiler/pgo/BUILD.gn:58:35: Script returned
non-zero exit code.
pgo_data_path = rebase_path(exec_script("//tools/update_pgo_profiles.py",
^----------
Current dir: <BUILDDIR>/out/argtest/
Command: python <BUILDDIR>/tools/update_pgo_profiles.py --target linux
get_profile_path
Returned 1.
stderr:

Traceback (most recent call last):
File "<BUILDDIR>/tools/update_pgo_profiles.py", line 154, in <module>
sys.exit(main())
File "<BUILDDIR>/tools/update_pgo_profiles.py", line 150, in main
return args.func(args)
File "<BUILDDIR>/tools/update_pgo_profiles.py", line 122, in _get_profile_path
profile_path)
RuntimeError: requested profile
"<BUILDDIR>/chrome/build/pgo_profiles/chrome-linux-master-1601358844-b8defbe977309018021c72a700e18c84f94d16d4.profdata"
doesn't exist, please make sure "checkout_pgo_profiles" is set to True
in the "custom_vars" section of your .gclient file, e.g.:
solutions = [
{
"name": "src",
# ...
"custom_vars": {
"checkout_pgo_profiles": True,
},
},
],
and then run "gclient runhooks" to download it. You can also simply
disable the PGO optimizations by setting |chrome_pgo_phase = 0| in
your GN arguments.

See //build/config/BUILDCONFIG.gn:335:3: which caused the file to be included.
"//build/config/compiler/pgo:default_pgo_flags",
^----------------------------------------------

I'm wondering if *.profdata files missing from the source tarball is
intentional, or an oversight?

Thanks,

Olivier

[Lei Zhang]

Looping in the author of https://crrev.com/809803, which enabled PGO,
for discussion.

> --
> You received this message because you are subscribed to the Google Groups "chromium-packagers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to chromium-packag...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-packagers/CAKyXnk8WHvZTpM89SRJUtgnWL25nzTbxovG0nFMZ20bSFDRYUw%40mail.gmail.com.

[Sébastien Marchand]

(resending after joining chromium-packagers@ as my initial email bounced) 

On Mon, Oct 5, 2020 at 2:28 PM Sébastien Marchand <sebma...@chromium.org> wrote:

I'm not familiar with these packages, in theory we should just include the content of "//chrome/build/" to the archive. It looks like the export_tarball.py script already includes every directory, so my best guess is that we're simply not fetching these profiles on the bots that generate the tarballs. We should simply make these bots use the "checkout_pgo_profile" config to address this.

--

Sébastien Marchand | Software Developer | sebma...@google.com 

--

Sébastien Marchand | Software Developer | sebma...@google.com 

[Raphael Kubo Da Costa]

I don't think just adding that config to the bot will help, as it explicitly does not run any hooks during the checkout process. We had to add code to explicitly download the AFDO profiles in the past, and I guess we'll need to do something similar for the PGO profiles now.

[Sébastien Marchand]

I'm not familiar at all with the tarballs, is there more documentation? I think that we simply need to add a call to

    src/tools/update_pgo_profiles.py --target={platform} update --gs-url-base=chromium-optimization-profiles/pgo_profiles 

to publish_tarball.py but I'm not sure what {platform} should be?  Is it always Linux?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-packagers/a6d3871b-335c-4ef8-bbe2-dcbf36953f6dn%40chromium.org.

[Lei Zhang]

Tom, can you help add this, since you added the AFDO bits previously?

On Tue, Oct 6, 2020 at 9:57 AM Sébastien Marchand

[Kubo Da Costa, Raphael]

I can take care of this tomorrow, I took a quick look earlier today and have a few cleanups to land first, but it should be an easy fix.

---

From: Lei Zhang <the...@chromium.org>
Sent: Tuesday, October 6, 2020 9:02:20 PM
To: Sébastien Marchand; Thomas Anderson
Cc: Kubo Da Costa, Raphael; chromium-packagers; Olivier Tilloy
Subject: Re: [chromium-packagers] dev tarballs lack profile data for PGO

 

You received this message because you are subscribed to a topic in the Google Groups "chromium-packagers" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/chromium-packagers/2ID9c4j6UkY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chromium-packag...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-packagers/CACSHbcSyarvi0z3K2%3DM4yrQJzGxF%3D10tAt1UxLs0Q65Ei_9tpA%40mail.gmail.com.

[Sébastien Marchand]

Thanks, please cc me on the fixes or add me as a reviewer, as I've mentioned before I'm not familiar with these packages but I don't mind learning!

[Thomas Anderson]

Thanks for taking this, Raphael!

[Raphael Kubo Da Costa]

https://chromium-review.googlesource.com/c/infra/infra/+/2453563 should fix this issue.

[Olivier Tilloy]

Thanks Raphael!

On Wed, Oct 7, 2020 at 2:01 PM Raphael Kubo Da Costa