Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#1051801: document DEB_BUILD_OPTIONS value nopgo
1 view
Skip to first unread message
Helmut Grohne
Sep 12, 2023, 2:20:04 PM9/12/23
to
Package: debian-policy
Version: 4.6.2.0
Severity: wishlist
X-Debbugs-Cc: debian...@lists.debian.org,rb-ge...@lists.reproducible-builds.org
Hi,
more and more packages implement a technique called profile guided
optimization. The general idea is that it performs a build that is
instrumented for profiling first. It then runs a reasonable workload to
collect profiling data, which in turn is used to guide the optimizer of
a second build which is not thus instrumented. The idea is that this
second build probably is faster than a regular build.
Quite obviously this approach completely breaks cross building. It also
is unclear how it affects reproducible builds since such builds depend
on the performance characteristics of the system performing the build.
This makes it very obvious that the pgo technique has downsides that
warrant disabling it in some situations.
A number of packages have agreed on disabling such optimization when
DEB_BUILD_OPTIONS contains nopgo. I'm aware of the following packages:
* binutils
* cross-toolchain-base
* gcc-VER
* halide
* pythonVER
I'll also be filing a patch for foot to support this option.
Is this sufficient coverage to document the option already? If not, this
bug report can serve as a central point for discussing it and its
adoption.
Proposed wording:
This tag requests that any optimization performed during the build
should not rely on performance characteristics captured during the
build. Such optimization is usually called profile guided
optimization.
The proposed tag intentionally is fairly narrow. It does not cover link
time optimization. It also does not cover the case where profiling
information is recorded ahead of upload and included in the source
package[1]. In both cases, neither cross building nor reproducibility is
impacted.
As for cross builds, it is not clear to me where we want to put the
responsibility to disable pgo. At the time of this writing, most
packages automatically disable pgo when performing a cross build. On the
flip side, we could have any cross builder set nopgo like they set
nocheck already. Doing so would allow performing a pgo-enabled cross
build to i386 on amd64 while still benefiting from the larger address
space for instance. I consider this aspect to be a separate matter
though.
Helmut
[1] https://lists.reproducible-builds.org/pipermail/rb-general/2022-June/002638.html
Version: 4.6.2.0
Severity: wishlist
X-Debbugs-Cc: debian...@lists.debian.org,rb-ge...@lists.reproducible-builds.org
Hi,
more and more packages implement a technique called profile guided
optimization. The general idea is that it performs a build that is
instrumented for profiling first. It then runs a reasonable workload to
collect profiling data, which in turn is used to guide the optimizer of
a second build which is not thus instrumented. The idea is that this
second build probably is faster than a regular build.
Quite obviously this approach completely breaks cross building. It also
is unclear how it affects reproducible builds since such builds depend
on the performance characteristics of the system performing the build.
This makes it very obvious that the pgo technique has downsides that
warrant disabling it in some situations.
A number of packages have agreed on disabling such optimization when
DEB_BUILD_OPTIONS contains nopgo. I'm aware of the following packages:
* binutils
* cross-toolchain-base
* gcc-VER
* halide
* pythonVER
I'll also be filing a patch for foot to support this option.
Is this sufficient coverage to document the option already? If not, this
bug report can serve as a central point for discussing it and its
adoption.
Proposed wording:
This tag requests that any optimization performed during the build
should not rely on performance characteristics captured during the
build. Such optimization is usually called profile guided
optimization.
The proposed tag intentionally is fairly narrow. It does not cover link
time optimization. It also does not cover the case where profiling
information is recorded ahead of upload and included in the source
package[1]. In both cases, neither cross building nor reproducibility is
impacted.
As for cross builds, it is not clear to me where we want to put the
responsibility to disable pgo. At the time of this writing, most
packages automatically disable pgo when performing a cross build. On the
flip side, we could have any cross builder set nopgo like they set
nocheck already. Doing so would allow performing a pgo-enabled cross
build to i386 on amd64 while still benefiting from the larger address
space for instance. I consider this aspect to be a separate matter
though.
Helmut
[1] https://lists.reproducible-builds.org/pipermail/rb-general/2022-June/002638.html
Russ Allbery
Sep 12, 2023, 3:10:04 PM9/12/23
to
Helmut Grohne <hel...@subdivi.de> writes:
> more and more packages implement a technique called profile guided
> optimization. The general idea is that it performs a build that is
> instrumented for profiling first. It then runs a reasonable workload to
> collect profiling data, which in turn is used to guide the optimizer of
> a second build which is not thus instrumented. The idea is that this
> second build probably is faster than a regular build.
> Quite obviously this approach completely breaks cross building. It also
> is unclear how it affects reproducible builds since such builds depend
> on the performance characteristics of the system performing the build.
> This makes it very obvious that the pgo technique has downsides that
> warrant disabling it in some situations.
> A number of packages have agreed on disabling such optimization when
> DEB_BUILD_OPTIONS contains nopgo. I'm aware of the following packages:
> * binutils
> * cross-toolchain-base
> * gcc-VER
> * halide
> * pythonVER
> I'll also be filing a patch for foot to support this option.
> Is this sufficient coverage to document the option already?
Yes, I think that's plenty. I would love to have Policy be a bit more
> more and more packages implement a technique called profile guided
> optimization. The general idea is that it performs a build that is
> instrumented for profiling first. It then runs a reasonable workload to
> collect profiling data, which in turn is used to guide the optimizer of
> a second build which is not thus instrumented. The idea is that this
> second build probably is faster than a regular build.
> Quite obviously this approach completely breaks cross building. It also
> is unclear how it affects reproducible builds since such builds depend
> on the performance characteristics of the system performing the build.
> This makes it very obvious that the pgo technique has downsides that
> warrant disabling it in some situations.
> A number of packages have agreed on disabling such optimization when
> DEB_BUILD_OPTIONS contains nopgo. I'm aware of the following packages:
> * binutils
> * cross-toolchain-base
> * gcc-VER
> * halide
> * pythonVER
> I'll also be filing a patch for foot to support this option.
> Is this sufficient coverage to document the option already?
proactive about documenting things like this.
> Proposed wording:
> This tag requests that any optimization performed during the build
> should not rely on performance characteristics captured during the
> build. Such optimization is usually called profile guided
> optimization.
builds) as an example of why someone may want to set this option? I think
having those sorts of explanations add a lot of value to Policy, since
they help explain to the reader how Debian is designed beyond just a
mechanical set of instructions.
If you have a chance, feel free to send a proposed diff to add this to the
DEB_BUILD_OPTIONS section.
--
Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Bernhard M. Wiedemann
Sep 13, 2023, 2:40:04 AM9/13/23
to
On 11/09/2023 09.25, Helmut Grohne wrote:
> It also
> is unclear how it affects reproducible builds since such builds depend
> on the performance characteristics of the system performing the build.
It is worth noting that the performance (execution time) of a
> It also
> is unclear how it affects reproducible builds since such builds depend
> on the performance characteristics of the system performing the build.
build-system does not matter for profiling, so it is possible to achieve
reproducible builds with PGO enabled. It is just hard.
See https://build.opensuse.org/request/show/499887
linked in
https://github.com/bmwiedemann/theunreproduciblepackage/tree/master/pgo
In that gzip patch we even had to hide the name of the temporary file
from gzip to not get variations from a 'tolower' call that would be
optimized for different amounts of upper/lower-case letters.
Parallel builds and variations in ordering are also problematic, because
some of the performance-counter-logic in gcc is not commutative, so
running A and B calls produces different results from first calling B
and then A.
With all that said, in openSUSE we also have a %do_profiling value to
disable PGO for gcc, python and some others, because these profiling
runs are too large to make deterministic.
Ciao
Bernhard M.
Bill Allombert
Sep 13, 2023, 5:10:05 AM9/13/23
to
On Mon, Sep 11, 2023 at 09:25:11AM +0200, Helmut Grohne wrote:
> Package: debian-policy
> Version: 4.6.2.0
> Severity: wishlist
> X-Debbugs-Cc: debian...@lists.debian.org,rb-ge...@lists.reproducible-builds.org
>
> Hi,
>
> more and more packages implement a technique called profile guided
> optimization. The general idea is that it performs a build that is
> instrumented for profiling first. It then runs a reasonable workload to
> collect profiling data, which in turn is used to guide the optimizer of
> a second build which is not thus instrumented. The idea is that this
> second build probably is faster than a regular build.
Hi!
> Package: debian-policy
> Version: 4.6.2.0
> Severity: wishlist
> X-Debbugs-Cc: debian...@lists.debian.org,rb-ge...@lists.reproducible-builds.org
>
> Hi,
>
> more and more packages implement a technique called profile guided
> optimization. The general idea is that it performs a build that is
> instrumented for profiling first. It then runs a reasonable workload to
> collect profiling data, which in turn is used to guide the optimizer of
> a second build which is not thus instrumented. The idea is that this
> second build probably is faster than a regular build.
Is it possible to save the profile data to reperform the second build later ?
Cheers,
--
Bill. <ball...@debian.org>
Imagine a large red swirl here.
0 new messages