Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
-ffile-prefix-map option and reproducibility
131 views
Skip to first unread message
Stéphane Glondu
Feb 7, 2023, 10:50:05 AM2/7/23
to
Hi,
When building packages, a -ffile-prefix-map option is automatically
injected into CFLAGS. Where does it come from? Since when?
I suspect this was added to improve reproducibility. Ironically, it
makes packages that capture this variable non reproducible, since the
build path seems to be randomized (has it always been the case? since
when?). It is the case of OCaml (see #1030785), and seemingly of R as
well (found by grepping in my /etc). I wouldn't be surprised other
packages are affected as well.
Is there a way to not get this option? More elegant than explicitly
filtering it out of CFLAGS in debian/rules...
Cheers,
--
Stéphane
When building packages, a -ffile-prefix-map option is automatically
injected into CFLAGS. Where does it come from? Since when?
I suspect this was added to improve reproducibility. Ironically, it
makes packages that capture this variable non reproducible, since the
build path seems to be randomized (has it always been the case? since
when?). It is the case of OCaml (see #1030785), and seemingly of R as
well (found by grepping in my /etc). I wouldn't be surprised other
packages are affected as well.
Is there a way to not get this option? More elegant than explicitly
filtering it out of CFLAGS in debian/rules...
Cheers,
--
Stéphane
Johannes Schauer Marin Rodrigues
Feb 7, 2023, 11:50:05 AM2/7/23
to
Hi,
Quoting Stéphane Glondu (2023-02-07 16:41:47)
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/commit/?id=b60c243ba99b8483202a6f6a814476275204fdff
which references:
https://lists.debian.org/debian-devel/2020/10/msg00222.html
https://bugs.debian.org/974087
> I suspect this was added to improve reproducibility. Ironically, it makes
> packages that capture this variable non reproducible, since the build path
> seems to be randomized (has it always been the case? since when?). It is the
> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
> in my /etc). I wouldn't be surprised other packages are affected as well.
>
> Is there a way to not get this option? More elegant than explicitly
> filtering it out of CFLAGS in debian/rules...
See the man page of dpkg-buildflags -- this might do what you want:
export DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath
Thanks!
cheers, josch
Quoting Stéphane Glondu (2023-02-07 16:41:47)
> When building packages, a -ffile-prefix-map option is automatically injected
> into CFLAGS. Where does it come from? Since when?
probably due to
> into CFLAGS. Where does it come from? Since when?
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/commit/?id=b60c243ba99b8483202a6f6a814476275204fdff
which references:
https://lists.debian.org/debian-devel/2020/10/msg00222.html
https://bugs.debian.org/974087
> I suspect this was added to improve reproducibility. Ironically, it makes
> packages that capture this variable non reproducible, since the build path
> seems to be randomized (has it always been the case? since when?). It is the
> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
> in my /etc). I wouldn't be surprised other packages are affected as well.
>
> Is there a way to not get this option? More elegant than explicitly
> filtering it out of CFLAGS in debian/rules...
export DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath
Thanks!
cheers, josch
Guillem Jover
Feb 7, 2023, 12:00:04 PM2/7/23
to
Hi!
On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote:
> When building packages, a -ffile-prefix-map option is automatically injected
> into CFLAGS. Where does it come from? Since when?
This is coming from dpkg-buildflags (in this case probably indirectly
via debhelper). AFAICS it was added in dpkg 1.19.1 disabled by default,
and then switched to enabled by default in dpkg 1.20.6 (see #974087).
> I suspect this was added to improve reproducibility. Ironically, it makes
> packages that capture this variable non reproducible, since the build path
> seems to be randomized (has it always been the case? since when?). It is the
> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
> in my /etc). I wouldn't be surprised other packages are affected as well.
AFAIR this was considered at the time, yes. If the flag is effectively
not fixing anything for the set of packages involved, and is in fact
actually making them unreproducible when they would not then, you can
disable the fixfilepath feature in the reproducible build flags area,
via DEB_BUILD_MAINT_OPTIONS. Perhaps even "globally" from a language
specific packaging helper or similar?
> Is there a way to not get this option? More elegant than explicitly
> filtering it out of CFLAGS in debian/rules...
See above.
I just noticed that several of these build flag features do not have
information in the man page about when they got first introduced, so
I'll be adding that in dpkg 1.22.x, once development opens up again.
Thanks,
Guillem
On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote:
> When building packages, a -ffile-prefix-map option is automatically injected
> into CFLAGS. Where does it come from? Since when?
via debhelper). AFAICS it was added in dpkg 1.19.1 disabled by default,
and then switched to enabled by default in dpkg 1.20.6 (see #974087).
> I suspect this was added to improve reproducibility. Ironically, it makes
> packages that capture this variable non reproducible, since the build path
> seems to be randomized (has it always been the case? since when?). It is the
> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
> in my /etc). I wouldn't be surprised other packages are affected as well.
not fixing anything for the set of packages involved, and is in fact
actually making them unreproducible when they would not then, you can
disable the fixfilepath feature in the reproducible build flags area,
via DEB_BUILD_MAINT_OPTIONS. Perhaps even "globally" from a language
specific packaging helper or similar?
> Is there a way to not get this option? More elegant than explicitly
> filtering it out of CFLAGS in debian/rules...
I just noticed that several of these build flag features do not have
information in the man page about when they got first introduced, so
I'll be adding that in dpkg 1.22.x, once development opens up again.
Thanks,
Guillem
Mattia Rizzolo
Feb 7, 2023, 1:20:05 PM2/7/23
to
On Tue, Feb 07, 2023 at 04:41:47PM +0100, Stéphane Glondu wrote:
> When building packages, a -ffile-prefix-map option is automatically injected
> into CFLAGS. Where does it come from? Since when?
>
> I suspect this was added to improve reproducibility. Ironically, it makes
> packages that capture this variable non reproducible, since the build path
> seems to be randomized (has it always been the case? since when?).
The build path has always been randomized since, or at least it has been
> When building packages, a -ffile-prefix-map option is automatically injected
> into CFLAGS. Where does it come from? Since when?
>
> I suspect this was added to improve reproducibility. Ironically, it makes
> packages that capture this variable non reproducible, since the build path
> seems to be randomized (has it always been the case? since when?).
for as long as I've been involved in Debian.
> It is the
> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
> in my /etc). I wouldn't be surprised other packages are affected as well.
>
> Is there a way to not get this option? More elegant than explicitly
> filtering it out of CFLAGS in debian/rules...
DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath
I actually propose to you to filter out the whole option from being
saved. I've seen a similar pattern in other packages in the past, and
all of those packages already had a filtering function in place to
remove other gcc flags that make no sense being saved (just looking at:
- 8: const("camlConfig__8"="-O2 -fno-strict-aliasing -fwrapv -pthread -fPIC -g -O2 -ffile-prefix-map=/build/ocaml-Vq2uKK/ocaml-4.13.1=. -fstack-protector-strong -Wformat -Werror=format-security");
+ 8: const("camlConfig__8"="-O2 -fno-strict-aliasing -fwrapv -pthread -fPIC -g -O2 -ffile-prefix-map=/build/ocaml-xz3WL7/ocaml-4.13.1=. -fstack-protector-strong -Wformat -Werror=format-security");
makes me believe that many options have been stripped out…)
--
regards,
Mattia Rizzolo
GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`.
More about me: https://mapreri.org : :' :
Launchpad user: https://launchpad.net/~mapreri `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia `-
Sven Joachim
Feb 7, 2023, 2:10:05 PM2/7/23
to
On 2023-02-07 17:50 +0100, Guillem Jover wrote:
> On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote:
>> When building packages, a -ffile-prefix-map option is automatically injected
>> into CFLAGS. Where does it come from? Since when?
>
> This is coming from dpkg-buildflags (in this case probably indirectly
> via debhelper). AFAICS it was added in dpkg 1.19.1 disabled by default,
> and then switched to enabled by default in dpkg 1.20.6 (see #974087).
>
>> I suspect this was added to improve reproducibility. Ironically, it makes
>> packages that capture this variable non reproducible, since the build path
>> seems to be randomized (has it always been the case? since when?). It is the
>> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
>> in my /etc). I wouldn't be surprised other packages are affected as well.
>
> AFAIR this was considered at the time, yes. If the flag is effectively
> not fixing anything for the set of packages involved, and is in fact
> actually making them unreproducible when they would not then, you can
> disable the fixfilepath feature in the reproducible build flags area,
> via DEB_BUILD_MAINT_OPTIONS.
This does not help for packages which capture all build flags and store
> On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote:
>> When building packages, a -ffile-prefix-map option is automatically injected
>> into CFLAGS. Where does it come from? Since when?
>
> This is coming from dpkg-buildflags (in this case probably indirectly
> via debhelper). AFAICS it was added in dpkg 1.19.1 disabled by default,
> and then switched to enabled by default in dpkg 1.20.6 (see #974087).
>
>> I suspect this was added to improve reproducibility. Ironically, it makes
>> packages that capture this variable non reproducible, since the build path
>> seems to be randomized (has it always been the case? since when?). It is the
>> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
>> in my /etc). I wouldn't be surprised other packages are affected as well.
>
> AFAIR this was considered at the time, yes. If the flag is effectively
> not fixing anything for the set of packages involved, and is in fact
> actually making them unreproducible when they would not then, you can
> disable the fixfilepath feature in the reproducible build flags area,
> via DEB_BUILD_MAINT_OPTIONS.
them in some file in the package (as is the case here). With
DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath, dpkg-buildflags falls
back to "-fdebug-prefix-map=<BUILDDIR>=.", and you have the same
problem. If you disable that as well via
DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath,-fixdebugpath, the
-dbgsym packages will most likely end up unreproducible.
Cheers,
Sven
Emilio Pozuelo Monfort
Feb 8, 2023, 5:00:04 AM2/8/23
to
On 07/02/2023 20:00, Sven Joachim wrote:
> On 2023-02-07 17:50 +0100, Guillem Jover wrote:
>
>> On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote:
>>> When building packages, a -ffile-prefix-map option is automatically injected
>>> into CFLAGS. Where does it come from? Since when?
>>
>> This is coming from dpkg-buildflags (in this case probably indirectly
>> via debhelper). AFAICS it was added in dpkg 1.19.1 disabled by default,
>> and then switched to enabled by default in dpkg 1.20.6 (see #974087).
>>
>>> I suspect this was added to improve reproducibility. Ironically, it makes
>>> packages that capture this variable non reproducible, since the build path
>>> seems to be randomized (has it always been the case? since when?). It is the
>>> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
>>> in my /etc). I wouldn't be surprised other packages are affected as well.
>>
>> AFAIR this was considered at the time, yes. If the flag is effectively
>> not fixing anything for the set of packages involved, and is in fact
>> actually making them unreproducible when they would not then, you can
>> disable the fixfilepath feature in the reproducible build flags area,
>> via DEB_BUILD_MAINT_OPTIONS.
>
> This does not help for packages which capture all build flags and store
> them in some file in the package (as is the case here).
What is the purpose of having the build flags in a file in the .deb?
> On 2023-02-07 17:50 +0100, Guillem Jover wrote:
>
>> On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote:
>>> When building packages, a -ffile-prefix-map option is automatically injected
>>> into CFLAGS. Where does it come from? Since when?
>>
>> This is coming from dpkg-buildflags (in this case probably indirectly
>> via debhelper). AFAICS it was added in dpkg 1.19.1 disabled by default,
>> and then switched to enabled by default in dpkg 1.20.6 (see #974087).
>>
>>> I suspect this was added to improve reproducibility. Ironically, it makes
>>> packages that capture this variable non reproducible, since the build path
>>> seems to be randomized (has it always been the case? since when?). It is the
>>> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
>>> in my /etc). I wouldn't be surprised other packages are affected as well.
>>
>> AFAIR this was considered at the time, yes. If the flag is effectively
>> not fixing anything for the set of packages involved, and is in fact
>> actually making them unreproducible when they would not then, you can
>> disable the fixfilepath feature in the reproducible build flags area,
>> via DEB_BUILD_MAINT_OPTIONS.
>
> This does not help for packages which capture all build flags and store
> them in some file in the package (as is the case here).
Cheers,
Emilio
Stéphane Glondu
Feb 8, 2023, 5:20:05 AM2/8/23
to
Thank you all for your answers!
Using:
DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath,-fixdebugpath
makes the package unreproducible in another way that seems difficult to fix.
Le 07/02/2023 à 19:12, Mattia Rizzolo a écrit :
> I actually propose to you to filter out the whole option from being
> saved. [...]
I've gone this way, and managed to make the package reproducible, at
least with the build path variation.
I will upload the fixed ocaml package when the current batch of related
packages waiting in unstable migrates to testing, hopefully in 4 days.
Cheers,
--
Stéphane
Using:
DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath,-fixdebugpath
makes the package unreproducible in another way that seems difficult to fix.
Le 07/02/2023 à 19:12, Mattia Rizzolo a écrit :
> I actually propose to you to filter out the whole option from being
I've gone this way, and managed to make the package reproducible, at
least with the build path variation.
I will upload the fixed ocaml package when the current batch of related
packages waiting in unstable migrates to testing, hopefully in 4 days.
Cheers,
--
Stéphane
Guillem Jover
Feb 8, 2023, 5:50:05 AM2/8/23
to
making the build unreproducible and the package not generating any
debug objects are going to be common at all. I considered mentioning
fixing the build to stop capturing, but felt it might be more effort
than requested. :) Should probably have mentioned anyway, as was done
elsewhere in the thread.
In any case I'm thinking to add something like the attached to the man
page to try to clarify this.
Thanks,
Guillem
Vagrant Cascadian
Feb 8, 2023, 1:00:05 PM2/8/23
to
On 2023-02-08, Stéphane Glondu wrote:
> Thank you all for your answers!
>
> Using:
>
> DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath,-fixdebugpath
>
> makes the package unreproducible in another way that seems difficult to fix.
Most likely reintroducing the things that the -ffile-prefix-map and
> Thank you all for your answers!
>
> Using:
>
> DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath,-fixdebugpath
>
> makes the package unreproducible in another way that seems difficult to fix.
-fdebug-prefix-map was effectively removing...
We track these kinds of issues with the "records build flags" issue,
which has a description of the problem and links to more information:
https://tests.reproducible-builds.org/debian/issues/unstable/records_build_flags_issue.html
There are some potential fixes to the issue more fundamentally, but they
are currently stalled out... one of which I should probably spend some
time on after bookworm release...
You had earlier asked when this was enabled, this can mostly be found in
the dpkg changelog:
fixfilepath (a.k.a. -ffile-prefix-map) Enabled by default:
dpkg (1.20.6) unstable; urgency=medium
...
* dpkg-buildflags: Enable reproducible=fixfilepath by default. Thanks
to Vagrant Cascadian <vag...@reproducible-builds.org>. See
https://lists.debian.org/debian-devel/2020/10/msg00222.html.
Closes: #974087
...
-- Guillem Jover <gui...@debian.org> Fri, 08 Jan 2021 04:39:40 +0100
fixfilepath (a.k.a. -ffile-prefix-map) feature added, and enabled in
reproducible builds infrastructure soon after:
dpkg (1.19.1) unstable; urgency=medium
...
- Dpkg::Vendor::Debian: Add fixfilepath support to reproducible feature.
...
-- Guillem Jover <gui...@debian.org> Wed, 26 Sep 2018 15:13:22 +0200
fixdebugpath (a.k.a. -fdebug-prefix-map) enabled by default:
dpkg (1.18.10) unstable; urgency=medium
...
- Enable fixdebugpath build flag feature by default.
Thanks to Mattia Rizzolo <mat...@debian.org>. Closes: #832179
...
-- Guillem Jover <gui...@debian.org> Sun, 31 Jul 2016 12:57:02 +0200
fixdebugpath (a.k.a. -fdebug-prefix-map) feature added, and presumably
enabled in reproducible builds infrastructure soon after:
dpkg (1.18.5) unstable; urgency=medium
...
- Add fixdebugpath to reproducible feature in Dpkg::Vendor::Debian.
Thanks to Daniel Kahn Gillmor <d...@fifthhorseman.net>. Closes:
#819194
...
-- Guillem Jover <gui...@debian.org> Mon, 02 May 2016 04:14:57 +0200
Of course, this is only for packages respecting dpkg-buildflags.
> Le 07/02/2023 à 19:12, Mattia Rizzolo a écrit :
>> I actually propose to you to filter out the whole option from being
>> saved. [...]
>
> I've gone this way, and managed to make the package reproducible, at
> least with the build path variation.
> I will upload the fixed ocaml package when the current batch of related
> packages waiting in unstable migrates to testing, hopefully in 4 days.
live well,
vagrant
Konstantin Demin
Feb 9, 2023, 2:40:04 AM2/9/23
to
Hi!
I'm end up with following thing:
https://github.com/rockdrilla/debian-container/blob/808dfa15d130f833602a1486362cc551593a0d8a/image/python/template/rules#L543
Hovewer, "execute_before_dh_builddeb" recipe is for information only
(to be viewed by human) and "execute_before_dh_install" recipe is for
actual adjustments.
ср, 8 февр. 2023 г. в 13:39, Guillem Jover <gui...@debian.org>:
--
SY,
Konstantin Demin
I'm end up with following thing:
https://github.com/rockdrilla/debian-container/blob/808dfa15d130f833602a1486362cc551593a0d8a/image/python/template/rules#L543
Hovewer, "execute_before_dh_builddeb" recipe is for information only
(to be viewed by human) and "execute_before_dh_install" recipe is for
actual adjustments.
ср, 8 февр. 2023 г. в 13:39, Guillem Jover <gui...@debian.org>:
SY,
Konstantin Demin
0 new messages