PGO profiles needed for is_official_build = true

[Sunny Sachanandani]

It seems is_official_build = true now requires setting checkout_pgo_profiles to True in .gclient custom_vars. Otherwise, you get build errors like this:

C:\src\chromium\src$ gn gen out/Release
Traceback (most recent call last):
  File "C:/src/chromium/src/tools/update_pgo_profiles.py", line 142, in <module>
    sys.exit(main())
  File "C:/src/chromium/src/tools/update_pgo_profiles.py", line 138, in main
    return args.func(args)
  File "C:/src/chromium/src/tools/update_pgo_profiles.py", line 110, in _get_profile_path
    profile_path)
RuntimeError: requested profile "C:\src\chromium\src\chrome\build\pgo_profiles\chrome-win64-master-1588608427-32d8728cf4ee3d53550729a43d003e4d51839043.profdata" doesn't exist, please make sure "checkout_pgo_profiles" is set to
true in the "custom_vars" section of your .gclient file and then run "gclient runhooks" to download it
ERROR at //build/config/compiler/pgo/BUILD.gn:51:35: Script returned non-zero exit code.
      pgo_data_path = rebase_path(exec_script("//tools/update_pgo_profiles.py",
                                  ^----------
Current dir: C:/src/chromium/src/out/Release/
Command: C:/src/depot_tools/bootstrap-3_8_0_chromium_8_bin/python/bin/python.exe C:/src/chromium/src/tools/update_pgo_profiles.py --target win64 get_profile_path
Returned 1.
See //build/config/BUILDCONFIG.gn:439:3: which caused the file to be included.
  "//build/config/compiler/pgo:default_pgo_flags",
  ^----------------------------------------------

These are my gn args FWIW and revision is d4da4b0e3d43e97025e72ab87f9b0767d9448a45

is_debug = false
is_component_build = false
is_official_build = true
use_goma = true
enable_nacl = false
proprietary_codecs = true
ffmpeg_branding = "Chrome"

Is this expected?  I can't find any posts about this change in behavior.

[Michael Moss]

I'm not sure if this is the prefered way to disable that, but it should work to set the gn arg "chrome_pgo_phase = 0".

Michael

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAKqK8Mn%2BwrjRgvQgzp6NkXsPz9db4gzmwzfCDPS6wOPaNsYmeg%40mail.gmail.com.

[Nico Weber]

Yes, this is expected. We tried to make the error message pretty explicit:

please make sure "checkout_pgo_profiles" is set to
true in the "custom_vars" section of your .gclient file and then run "gclient runhooks" to download it

Did that not work?

--

[Nico Weber]

 …don't do that though. If you don't want official builds, don't set is_official_build. If you do want that, you likely want to match the build config we use to ship, so don't set more stuff that have the effect that your local build is different from the real official build :) If you don't want to get the official build settings, just don't set is_official_build.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAKs%2B6oOttO4_kvAqFUf%2BNRUH5f8PMbtWh98cQmjuDVcXrUpEtA%40mail.gmail.com.

[Michael Moss]

On Wed, May 13, 2020 at 5:13 PM Nico Weber <tha...@chromium.org> wrote:

 …don't do that though. If you don't want official builds, don't set is_official_build. If you do want that, you likely want to match the build config we use to ship

I don't think that's accurate. There are reasons to create "is_official" builds that don't necessarily mean you want to create an exact duplicate of an official Chrome build. And FWIW, even at least one of the Chrome official builders (the Windows ASAN builds) sets "chrome_pgo_phase = 0", so maybe they're trying to duplicate that?

[Sunny Sachanandani]

I understood that setting is_official_build = true was recommended for running perf benchmarks or other performance analysis even on Chromium builds which don't necessarily match official Chrome builds in all aspects. Is that not correct?

[Sébastien Marchand]

That's correct, |is_official_build| really means |use_official_build_settings=true| in this case. Its' true that it'll cause more things to be built but that's the flag you need to set to get a local build that ~ matches the one that we (Chrome) ship.

As PGO has been enabled by default on Win and Mac (we don't know if it'll stick yet so we haven't sent the PSA) we have decided to make this flag automatically do a PGO build (to avoid having multiple definitions of what "official" means). The "checkout_pgo_profiles" custom var will make gclient automatically fetch the PGO profiles you need. We don't enable this by default because these profiles are quite large and not everybody needs them. This shouldn't impact the total build time (you'll get a bunch of "warning: profile data may be out of date" message during the build, I'll work on removing them soon).

Please let me know if you think that the error message isn't clear enough.

--

Sébastien Marchand | Software Developer | sebma...@google.com 

[Nico Weber]

On Wed, May 13, 2020 at 8:18 PM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 5:13 PM Nico Weber <tha...@chromium.org> wrote:

 …don't do that though. If you don't want official builds, don't set is_official_build. If you do want that, you likely want to match the build config we use to ship

I don't think that's accurate. There are reasons to create "is_official" builds that don't necessarily mean you want to create an exact duplicate of an official Chrome build.

Could you share an example use case? This is news to me and I'd like to learn more. (I don't doubt it's true.)

 

And FWIW, even at least one of the Chrome official builders (the Windows ASAN builds) sets "chrome_pgo_phase = 0", so maybe they're trying to duplicate that?

The official Win Asan bot usually builds stuff for A/B tests (and usually not ASan builds, despite the name, as far as I know) or in non-standard build configs. We only have profiles for the "normal" is_official_build=true build config, and doing PGO when the profiles are collected for a different binary doesn't do much good. But that seems like a special case of a special case to me that most devs shouldn't run into.

[Michael Moss]

On Wed, May 13, 2020 at 6:11 PM Nico Weber <tha...@chromium.org> wrote:

On Wed, May 13, 2020 at 8:18 PM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 5:13 PM Nico Weber <tha...@chromium.org> wrote:

 …don't do that though. If you don't want official builds, don't set is_official_build. If you do want that, you likely want to match the build config we use to ship

I don't think that's accurate. There are reasons to create "is_official" builds that don't necessarily mean you want to create an exact duplicate of an official Chrome build.

Could you share an example use case? This is news to me and I'd like to learn more. (I don't doubt it's true.)

Don't other non-Chrome Chromium-based projects use is_official_build for their "official" builds? That was my understanding, although it may be out-dated. It's also shorthand for setting some random options, like creating a compressed installer file (https://source.chromium.org/chromium/chromium/src/+/master:chrome/installer/mini_installer/BUILD.gn;l=20), which might be more convenient than figuring out and setting a bunch of individual args.

And FWIW, even at least one of the Chrome official builders (the Windows ASAN builds) sets "chrome_pgo_phase = 0", so maybe they're trying to duplicate that?

The official Win Asan bot usually builds stuff for A/B tests

I don't want to get off on too much of a tangent here, but no, A/B testing on these builders is the exception, not the rule (and as far as I know, has only been done a few times ever, most recently for the rollout of these PGO configs).

[Lei Zhang]

On Wed, May 13, 2020 at 10:39 PM Michael Moss <mm...@chromium.org> wrote:
> Don't other non-Chrome Chromium-based projects use is_official_build for their "official" builds? That was my understanding, although it may be out-dated. It's also shorthand for setting some random options, like creating a compressed installer file (https://source.chromium.org/chromium/chromium/src/+/master:chrome/installer/mini_installer/BUILD.gn;l=20), which might be more convenient than figuring out and setting a bunch of individual args.

FWIW, someone was doing their "official" build of PDFium and ran into
this: https://groups.google.com/d/topic/pdfium/Hq24vELdxb4/discussion

Although the discussion thread looks like it only has 1 participant, I
actually was part of an offline discussion on that thread. I also
suggested setting chrome_pgo_phase=0 as a workaround, because the PGO
process looked too Chromium/Chrome-centric. For standalone PDFium,
there's no expectation for PGO to work. Presumably some folks set the
official bit because they just want a slightly more optimized build.

[Sébastien Marchand]

Thanks for the link Lei.

We should probably restrict this flag to the chrome target (e.g. add a "target_can_be_pgo_optimized" bit somewhere), I also would like to rename the |chrome_pgo_phase| flag at some point, or at least add some some more explicit flags that allow to control PGO (e.g. |no_pgo|).

[Sébastien Marchand]

I've opened crbug.com/1082736 to track this.

[Nico Weber]

On Thu, May 14, 2020 at 1:38 AM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 6:11 PM Nico Weber <tha...@chromium.org> wrote:

On Wed, May 13, 2020 at 8:18 PM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 5:13 PM Nico Weber <tha...@chromium.org> wrote:

 …don't do that though. If you don't want official builds, don't set is_official_build. If you do want that, you likely want to match the build config we use to ship

I don't think that's accurate. There are reasons to create "is_official" builds that don't necessarily mean you want to create an exact duplicate of an official Chrome build.

Could you share an example use case? This is news to me and I'd like to learn more. (I don't doubt it's true.)

Don't other non-Chrome Chromium-based projects use is_official_build for their "official" builds? That was my understanding, although it may be out-dated.

If it's Chromium-based browsers with not too many downstream patches to core code (blink etc), then the profiles we provide should work pretty well for them.

I don't know why someone using pdfium would set this though. To me, the flag always meant something like "use the most appropriate optimization settings for shipping a browser to end users" to me.

[Marshall Greenblatt]

On May 14, 2020, at 3:08 PM, Nico Weber <tha...@chromium.org> wrote:



On Thu, May 14, 2020 at 1:38 AM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 6:11 PM Nico Weber <tha...@chromium.org> wrote:

On Wed, May 13, 2020 at 8:18 PM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 5:13 PM Nico Weber <tha...@chromium.org> wrote:

 …don't do that though. If you don't want official builds, don't set is_official_build. If you do want that, you likely want to match the build config we use to ship

I don't think that's accurate. There are reasons to create "is_official" builds that don't necessarily mean you want to create an exact duplicate of an official Chrome build.

Could you share an example use case? This is news to me and I'd like to learn more. (I don't doubt it's true.)

Don't other non-Chrome Chromium-based projects use is_official_build for their "official" builds? That was my understanding, although it may be out-dated.

If it's Chromium-based browsers with not too many downstream patches to core code (blink etc), then the profiles we provide should work pretty well for them.

If we wanted to try these profiles with other Chromium-based browsers, what would be the recommended way to measure “work[s] pretty well”? Is there a documented way to generate new profiles based on our own Chromium-based binaries, and integrate those profiles via Chromium tooling for build and comparison purposes?

Thanks,

Marshall

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAMGbLiFvWnL2wNkmLwOY%3DbkxVzg3NDBYm1unW%2B%2By70V%2ByjYDqg%40mail.gmail.com.

[Sébastien Marchand]

On Thu, May 14, 2020 at 3:31 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On May 14, 2020, at 3:08 PM, Nico Weber <tha...@chromium.org> wrote:



On Thu, May 14, 2020 at 1:38 AM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 6:11 PM Nico Weber <tha...@chromium.org> wrote:

On Wed, May 13, 2020 at 8:18 PM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 5:13 PM Nico Weber <tha...@chromium.org> wrote:

 …don't do that though. If you don't want official builds, don't set is_official_build. If you do want that, you likely want to match the build config we use to ship

I don't think that's accurate. There are reasons to create "is_official" builds that don't necessarily mean you want to create an exact duplicate of an official Chrome build.

Could you share an example use case? This is news to me and I'd like to learn more. (I don't doubt it's true.)

Don't other non-Chrome Chromium-based projects use is_official_build for their "official" builds? That was my understanding, although it may be out-dated.

If it's Chromium-based browsers with not too many downstream patches to core code (blink etc), then the profiles we provide should work pretty well for them.

If we wanted to try these profiles with other Chromium-based browsers, what would be the recommended way to measure “work[s] pretty well”? Is there a documented way to generate new profiles based on our own Chromium-based binaries, and integrate those profiles via Chromium tooling for build and comparison purposes?

You could evaluate the impact of these profiles by looking at some of your core success metrics (whether it's a synthetic benchmark score or some user metrics) and compare this to a non-PGO build (add chrome_pgo_phase=0 to your GN args).

I'll work on documenting the profiling step once we have made more progress on the project (it's still not sure if it'll stick). The summary is that you need to:

• Do a local build with the |chrome_pgo_phase=1| flag. This will produce the instrumented build (I recommend disabling the symbols for this build as it makes the linker use a LOT of memory).
• Run some profiling scenarios with this build.
• Merge the raw profiling data (the *.profraw files generated when running the build) with the llvm-profdata utility (you need to add |"checkout_clang_coverage_tools": True| to your .gclient custom_vars section for gclient to fetch this tool).
• Build with this profile, you can specify the profile to use via the |pgo_data_path| GN variable.

[Marshall Greenblatt]

On May 14, 2020, at 3:52 PM, Sébastien Marchand <sebma...@chromium.org> wrote:



On Thu, May 14, 2020 at 3:31 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On May 14, 2020, at 3:08 PM, Nico Weber <tha...@chromium.org> wrote:



On Thu, May 14, 2020 at 1:38 AM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 6:11 PM Nico Weber <tha...@chromium.org> wrote:

On Wed, May 13, 2020 at 8:18 PM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 5:13 PM Nico Weber <tha...@chromium.org> wrote:

 …don't do that though. If you don't want official builds, don't set is_official_build. If you do want that, you likely want to match the build config we use to ship

I don't think that's accurate. There are reasons to create "is_official" builds that don't necessarily mean you want to create an exact duplicate of an official Chrome build.

Could you share an example use case? This is news to me and I'd like to learn more. (I don't doubt it's true.)

Don't other non-Chrome Chromium-based projects use is_official_build for their "official" builds? That was my understanding, although it may be out-dated.

If it's Chromium-based browsers with not too many downstream patches to core code (blink etc), then the profiles we provide should work pretty well for them.

If we wanted to try these profiles with other Chromium-based browsers, what would be the recommended way to measure “work[s] pretty well”? Is there a documented way to generate new profiles based on our own Chromium-based binaries, and integrate those profiles via Chromium tooling for build and comparison purposes?

You could evaluate the impact of these profiles by looking at some of your core success metrics (whether it's a synthetic benchmark score or some user metrics) and compare this to a non-PGO build (add chrome_pgo_phase=0 to your GN args).

I'll work on documenting the profiling step once we have made more progress on the project (it's still not sure if it'll stick). The summary is that you need to:

• Do a local build with the |chrome_pgo_phase=1| flag. This will produce the instrumented build (I recommend disabling the symbols for this build as it makes the linker use a LOT of memory).
• Run some profiling scenarios with this build.
• Merge the raw profiling data (the *.profraw files generated when running the build) with the llvm-profdata utility (you need to add |"checkout_clang_coverage_tools": True| to your .gclient custom_vars section for gclient to fetch this tool).
• Build with this profile, you can specify the profile to use via the |pgo_data_path| GN variable.

Thanks for the details :). Would it be reasonable to develop and share a scripted version of the “run some profiling scenarios” step that iterates through publicly-available sources such as micro-benchmarks, browsing top sites, etc? That would be a useful bootstrap for anyone looking to generate their own profiles.

[Nico Weber]

On Thu, May 14, 2020 at 4:02 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On May 14, 2020, at 3:52 PM, Sébastien Marchand <sebma...@chromium.org> wrote:



On Thu, May 14, 2020 at 3:31 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On May 14, 2020, at 3:08 PM, Nico Weber <tha...@chromium.org> wrote:



On Thu, May 14, 2020 at 1:38 AM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 6:11 PM Nico Weber <tha...@chromium.org> wrote:

On Wed, May 13, 2020 at 8:18 PM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 5:13 PM Nico Weber <tha...@chromium.org> wrote:

 …don't do that though. If you don't want official builds, don't set is_official_build. If you do want that, you likely want to match the build config we use to ship

I don't think that's accurate. There are reasons to create "is_official" builds that don't necessarily mean you want to create an exact duplicate of an official Chrome build.

Could you share an example use case? This is news to me and I'd like to learn more. (I don't doubt it's true.)

Don't other non-Chrome Chromium-based projects use is_official_build for their "official" builds? That was my understanding, although it may be out-dated.

If it's Chromium-based browsers with not too many downstream patches to core code (blink etc), then the profiles we provide should work pretty well for them.

If we wanted to try these profiles with other Chromium-based browsers, what would be the recommended way to measure “work[s] pretty well”? Is there a documented way to generate new profiles based on our own Chromium-based binaries, and integrate those profiles via Chromium tooling for build and comparison purposes?

You could evaluate the impact of these profiles by looking at some of your core success metrics (whether it's a synthetic benchmark score or some user metrics) and compare this to a non-PGO build (add chrome_pgo_phase=0 to your GN args).

I'll work on documenting the profiling step once we have made more progress on the project (it's still not sure if it'll stick). The summary is that you need to:

• Do a local build with the |chrome_pgo_phase=1| flag. This will produce the instrumented build (I recommend disabling the symbols for this build as it makes the linker use a LOT of memory).
• Run some profiling scenarios with this build.
• Merge the raw profiling data (the *.profraw files generated when running the build) with the llvm-profdata utility (you need to add |"checkout_clang_coverage_tools": True| to your .gclient custom_vars section for gclient to fetch this tool).
• Build with this profile, you can specify the profile to use via the |pgo_data_path| GN variable.

I think a better first step is to use is_official_build=True, follow the prompts to download the profiles collected by our bots, and see how that does on your benchmarks. (In a 2nd build dir, build with chrome_pgo_phase=0 set as well to get a non-PGO build for comparison.)

We currently don't autoupdate the profiles so they're currently a bit stale, but that'll hopefully change very soon and the profiles should soon at most a few hours out of date at all times. (They're pinned in src.git and `gclient sync` will pull a profile that matches the revision that you have checked out.)

[Michael Moss]

On Thu, May 14, 2020 at 12:07 PM Nico Weber <tha...@chromium.org> wrote:

On Thu, May 14, 2020 at 1:38 AM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 6:11 PM Nico Weber <tha...@chromium.org> wrote:

On Wed, May 13, 2020 at 8:18 PM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 5:13 PM Nico Weber <tha...@chromium.org> wrote:

 …don't do that though. If you don't want official builds, don't set is_official_build. If you do want that, you likely want to match the build config we use to ship

I don't think that's accurate. There are reasons to create "is_official" builds that don't necessarily mean you want to create an exact duplicate of an official Chrome build.

Could you share an example use case? This is news to me and I'd like to learn more. (I don't doubt it's true.)

Don't other non-Chrome Chromium-based projects use is_official_build for their "official" builds? That was my understanding, although it may be out-dated.

If it's Chromium-based browsers with not too many downstream patches to core code (blink etc), then the profiles we provide should work pretty well for them.

I don't know why someone using pdfium would set this though. To me, the flag always meant something like "use the most appropriate optimization settings for shipping a browser to end users" to me.

Except it's not only used for browser settings/optimizaitons. I'm not too sure about PDFium, but another example of where it's used outside the browser context is for CRD.

[Sébastien Marchand]

On Thu, May 14, 2020 at 4:06 PM Nico Weber <tha...@chromium.org> wrote:

On Thu, May 14, 2020 at 4:02 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On May 14, 2020, at 3:52 PM, Sébastien Marchand <sebma...@chromium.org> wrote:



On Thu, May 14, 2020 at 3:31 PM Marshall Greenblatt <magree...@gmail.com> wrote:

On May 14, 2020, at 3:08 PM, Nico Weber <tha...@chromium.org> wrote:



On Thu, May 14, 2020 at 1:38 AM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 6:11 PM Nico Weber <tha...@chromium.org> wrote:

On Wed, May 13, 2020 at 8:18 PM Michael Moss <mm...@chromium.org> wrote:

On Wed, May 13, 2020 at 5:13 PM Nico Weber <tha...@chromium.org> wrote:

 …don't do that though. If you don't want official builds, don't set is_official_build. If you do want that, you likely want to match the build config we use to ship

I don't think that's accurate. There are reasons to create "is_official" builds that don't necessarily mean you want to create an exact duplicate of an official Chrome build.

Could you share an example use case? This is news to me and I'd like to learn more. (I don't doubt it's true.)

Don't other non-Chrome Chromium-based projects use is_official_build for their "official" builds? That was my understanding, although it may be out-dated.

If it's Chromium-based browsers with not too many downstream patches to core code (blink etc), then the profiles we provide should work pretty well for them.

If we wanted to try these profiles with other Chromium-based browsers, what would be the recommended way to measure “work[s] pretty well”? Is there a documented way to generate new profiles based on our own Chromium-based binaries, and integrate those profiles via Chromium tooling for build and comparison purposes?

You could evaluate the impact of these profiles by looking at some of your core success metrics (whether it's a synthetic benchmark score or some user metrics) and compare this to a non-PGO build (add chrome_pgo_phase=0 to your GN args).

I'll work on documenting the profiling step once we have made more progress on the project (it's still not sure if it'll stick). The summary is that you need to:

• Do a local build with the |chrome_pgo_phase=1| flag. This will produce the instrumented build (I recommend disabling the symbols for this build as it makes the linker use a LOT of memory).
• Run some profiling scenarios with this build.
• Merge the raw profiling data (the *.profraw files generated when running the build) with the llvm-profdata utility (you need to add |"checkout_clang_coverage_tools": True| to your .gclient custom_vars section for gclient to fetch this tool).
• Build with this profile, you can specify the profile to use via the |pgo_data_path| GN variable.

I think a better first step is to use is_official_build=True, follow the prompts to download the profiles collected by our bots, and see how that does on your benchmarks. (In a 2nd build dir, build with chrome_pgo_phase=0 set as well to get a non-PGO build for comparison.)

+1, the instructions I've provided are only useful if the provided profiles don't work for you (e.g. you have a lot of local diffs or you want to optimize for some really specific scenarios).

Adding a Chrome script that automates the "run some profiling scenario" parts is something that I want to do. When I want to generate a profile locally I use this script: https://chromium-review.googlesource.com/c/chromium/src/+/2090512/10/build/run_pgo_profiling_benchmarks.py , note that I haven't submitted this CL because it's not how I want to do this in the long term (ideally this script should consume a json file that contains the list of profiling scenarios to run, this way we'll be able to share this list with our bots). 

Sorry if some things are still a little bit unpolished/undocummented, we've been working hard on the PGO build over the past few weeks but there's still a lot left to do :).

[Venkat Satish]

Hi Sunny,

Even I am facing this issue did you get any resolution for this. If I set chrome_pgo_phase = 0 in gn gen command there is no error but after build the apk and installing it on Android A13 the chromium app is crashing.

Can you help me on this.

Thanks,

Venkat Satish A