dev tarballs lack profile data for PGO

50 views
Skip to first unread message

Olivier Tilloy

unread,
Oct 5, 2020, 10:19:03 AM10/5/20
to chromium-packagers
Hi chromium packagers,

With the promotion of 87.0.4278.0 to the dev channel, linux builds
switched to PGO (from AFDO). When building offline from a source
tarball, the build fails early because profile data (*.profdata) files
cannot be found:

ERROR at //build/config/compiler/pgo/BUILD.gn:58:35: Script returned
non-zero exit code.
pgo_data_path = rebase_path(exec_script("//tools/update_pgo_profiles.py",
^----------
Current dir: <BUILDDIR>/out/argtest/
Command: python <BUILDDIR>/tools/update_pgo_profiles.py --target linux
get_profile_path
Returned 1.
stderr:

Traceback (most recent call last):
File "<BUILDDIR>/tools/update_pgo_profiles.py", line 154, in <module>
sys.exit(main())
File "<BUILDDIR>/tools/update_pgo_profiles.py", line 150, in main
return args.func(args)
File "<BUILDDIR>/tools/update_pgo_profiles.py", line 122, in _get_profile_path
profile_path)
RuntimeError: requested profile
"<BUILDDIR>/chrome/build/pgo_profiles/chrome-linux-master-1601358844-b8defbe977309018021c72a700e18c84f94d16d4.profdata"
doesn't exist, please make sure "checkout_pgo_profiles" is set to True
in the "custom_vars" section of your .gclient file, e.g.:
solutions = [
{
"name": "src",
# ...
"custom_vars": {
"checkout_pgo_profiles": True,
},
},
],
and then run "gclient runhooks" to download it. You can also simply
disable the PGO optimizations by setting |chrome_pgo_phase = 0| in
your GN arguments.

See //build/config/BUILDCONFIG.gn:335:3: which caused the file to be included.
"//build/config/compiler/pgo:default_pgo_flags",
^----------------------------------------------

I'm wondering if *.profdata files missing from the source tarball is
intentional, or an oversight?

Thanks,

Olivier

Lei Zhang

unread,
Oct 5, 2020, 2:17:40 PM10/5/20
to Olivier Tilloy, Sébastien Marchand, chromium-packagers
Looping in the author of https://crrev.com/809803, which enabled PGO,
for discussion.
> --
> You received this message because you are subscribed to the Google Groups "chromium-packagers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to chromium-packag...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-packagers/CAKyXnk8WHvZTpM89SRJUtgnWL25nzTbxovG0nFMZ20bSFDRYUw%40mail.gmail.com.

Sébastien Marchand

unread,
Oct 5, 2020, 2:43:19 PM10/5/20
to Lei Zhang, Olivier Tilloy, chromium-packagers
(resending after joining chromium-packagers@ as my initial email bounced) 

On Mon, Oct 5, 2020 at 2:28 PM Sébastien Marchand <sebma...@chromium.org> wrote:
I'm not familiar with these packages, in theory we should just include the content of "//chrome/build/" to the archive. It looks like the export_tarball.py script already includes every directory, so my best guess is that we're simply not fetching these profiles on the bots that generate the tarballs. We should simply make these bots use the "checkout_pgo_profile" config to address this.
--
Sébastien Marchand | Software Developer | sebma...@google.com 




--
Sébastien Marchand | Software Developer | sebma...@google.com 


Raphael Kubo Da Costa

unread,
Oct 6, 2020, 10:35:35 AM10/6/20
to chromium-packagers, Sébastien Marchand, Olivier Tilloy, chromium-packagers, Lei Zhang
I don't think just adding that config to the bot will help, as it explicitly does not run any hooks during the checkout process. We had to add code to explicitly download the AFDO profiles in the past, and I guess we'll need to do something similar for the PGO profiles now.

Sébastien Marchand

unread,
Oct 6, 2020, 12:57:42 PM10/6/20
to Raphael Kubo Da Costa, chromium-packagers, Olivier Tilloy, Lei Zhang
I'm not familiar at all with the tarballs, is there more documentation? I think that we simply need to add a call to
    src/tools/update_pgo_profiles.py --target={platform} update --gs-url-base=chromium-optimization-profiles/pgo_profiles 
to publish_tarball.py but I'm not sure what {platform} should be?  Is it always Linux?

Lei Zhang

unread,
Oct 6, 2020, 3:02:33 PM10/6/20
to Sébastien Marchand, Thomas Anderson, Raphael Kubo Da Costa, chromium-packagers, Olivier Tilloy
Tom, can you help add this, since you added the AFDO bits previously?

On Tue, Oct 6, 2020 at 9:57 AM Sébastien Marchand

Kubo Da Costa, Raphael

unread,
Oct 6, 2020, 3:09:45 PM10/6/20
to Lei Zhang, Sébastien Marchand, Thomas Anderson, chromium-packagers, Olivier Tilloy

I can take care of this tomorrow, I took a quick look earlier today and have a few cleanups to land first, but it should be an easy fix.


From: Lei Zhang <the...@chromium.org>
Sent: Tuesday, October 6, 2020 9:02:20 PM
To: Sébastien Marchand; Thomas Anderson
Cc: Kubo Da Costa, Raphael; chromium-packagers; Olivier Tilloy
Subject: Re: [chromium-packagers] dev tarballs lack profile data for PGO
 
You received this message because you are subscribed to a topic in the Google Groups "chromium-packagers" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/chromium-packagers/2ID9c4j6UkY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chromium-packag...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-packagers/CACSHbcSyarvi0z3K2%3DM4yrQJzGxF%3D10tAt1UxLs0Q65Ei_9tpA%40mail.gmail.com.

Sébastien Marchand

unread,
Oct 6, 2020, 3:14:59 PM10/6/20
to Kubo Da Costa, Raphael, Lei Zhang, Thomas Anderson, chromium-packagers, Olivier Tilloy
Thanks, please cc me on the fixes or add me as a reviewer, as I've mentioned before I'm not familiar with these packages but I don't mind learning!

Thomas Anderson

unread,
Oct 6, 2020, 3:24:45 PM10/6/20
to Kubo Da Costa, Raphael, Lei Zhang, chromium-packagers, Olivier Tilloy, Sébastien Marchand
Thanks for taking this, Raphael!

Raphael Kubo Da Costa

unread,
Oct 7, 2020, 8:01:01 AM10/7/20
to chromium-packagers, thomasa...@chromium.org, Lei Zhang, Olivier Tilloy, Sébastien Marchand

Olivier Tilloy

unread,
Oct 7, 2020, 8:59:15 AM10/7/20
to Raphael Kubo Da Costa, chromium-packagers, thomasa...@chromium.org, Lei Zhang, Sébastien Marchand
Thanks Raphael!

On Wed, Oct 7, 2020 at 2:01 PM Raphael Kubo Da Costa
Reply all
Reply to author
Forward
0 new messages