Deterministic builds
39 views
Skip to first unread message
Thomas Braun
Jun 12, 2014, 3:05:43 PM6/12/14
to mingwgi...@googlegroups.com
Currently the mgwport packages, aka hte tar.lzma files, are
non-deterministically build. Therefore creating the same package twice
will not result in identical bits.
It would be nice to have deterministic builds so that if I build a
package locally and our jenkins build-job builds it, we can check easily
check if everything worked.
Getting there involves solving atm the following problems:
1.) Tar stores the file modification time
This can be avoided by setting the mtime to a fixed string for all
builds, e.g.
export TAR_OPTIONS=--mtime='2000-01-01'
before calling mgwport.
2.) The PE header of the executables/dlls/libraries has a time stamp
Binutils-2.24-1 which is not yet in mingw-get can be told to produce
deterministic builds. Unfortunately this is only the default for .exe
and .dll, and not for dll.a and .a. See
https://sourceforge.net/p/mingw/bugs/2158.
A work around would be to call strip --enable-deterministic-archive on
all relevant files before packaging them. This option is also known by
the current binutils package.
3.) Gzip 1.3.12 produces non-deterministic results
Obvious in man pages from the doc packages. See also
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=647522. Supposedly
fixed in
http://git.savannah.gnu.org/cgit/gzip.git/commit/?id=0a284baeaedca68017f46d2646e4c921aa98a90d.
Upgrading to gzip 1.6 might solve the problem.
non-deterministically build. Therefore creating the same package twice
will not result in identical bits.
It would be nice to have deterministic builds so that if I build a
package locally and our jenkins build-job builds it, we can check easily
check if everything worked.
Getting there involves solving atm the following problems:
1.) Tar stores the file modification time
This can be avoided by setting the mtime to a fixed string for all
builds, e.g.
export TAR_OPTIONS=--mtime='2000-01-01'
before calling mgwport.
2.) The PE header of the executables/dlls/libraries has a time stamp
Binutils-2.24-1 which is not yet in mingw-get can be told to produce
deterministic builds. Unfortunately this is only the default for .exe
and .dll, and not for dll.a and .a. See
https://sourceforge.net/p/mingw/bugs/2158.
A work around would be to call strip --enable-deterministic-archive on
all relevant files before packaging them. This option is also known by
the current binutils package.
3.) Gzip 1.3.12 produces non-deterministic results
Obvious in man pages from the doc packages. See also
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=647522. Supposedly
fixed in
http://git.savannah.gnu.org/cgit/gzip.git/commit/?id=0a284baeaedca68017f46d2646e4c921aa98a90d.
Upgrading to gzip 1.6 might solve the problem.
Sebastian Schuberth
Aug 29, 2014, 11:44:33 AM8/29/14
to Thomas Braun, mingwgi...@googlegroups.com
On Thu, Jun 12, 2014 at 9:05 PM, Thomas Braun
<thomas...@virtuell-zuhause.de> wrote:
Just to follow up on this:
> Getting there involves solving atm the following problems:
> 1.) Tar stores the file modification time
> This can be avoided by setting the mtime to a fixed string for all
> builds, e.g.
> export TAR_OPTIONS=--mtime='2000-01-01'
> before calling mgwport.
How about modifying mgwport in a way so that it sets TAR_OPTIONS to
the port file's modification date, or the issue date of the belonging
package file? The latter probably is a bit harder to do in a nice way
as mgwport usually doe snot know about mingw-get's package files.
> 2.) The PE header of the executables/dlls/libraries has a time stamp
>
> Binutils-2.24-1 which is not yet in mingw-get can be told to produce
> deterministic builds. Unfortunately this is only the default for .exe
> and .dll, and not for dll.a and .a. See
> https://sourceforge.net/p/mingw/bugs/2158.
>
> A work around would be to call strip --enable-deterministic-archive on
> all relevant files before packaging them. This option is also known by
> the current binutils package.
Now that we have binutils-2.24-1 upstream, should we build our own
version that is configured with "--enable-deterministic-archive", or
should we modify mgwport (which already calla strip anyway, see
src_postinst.mgwpart) to also pass "--enable-deterministic-archive" to
strip? The latter probably is less work given that we need to modify
mgwport for 1) anyway.
> 3.) Gzip 1.3.12 produces non-deterministic results
> Obvious in man pages from the doc packages. See also
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=647522. Supposedly
> fixed in
> http://git.savannah.gnu.org/cgit/gzip.git/commit/?id=0a284baeaedca68017f46d2646e4c921aa98a90d.
> Upgrading to gzip 1.6 might solve the problem.
It'd be great if you could find the time to check whether gzip 1.6.
fixed the problem, and eventually create PRs for these new packages
;-)
--
Sebastian Schuberth
<thomas...@virtuell-zuhause.de> wrote:
Just to follow up on this:
> Getting there involves solving atm the following problems:
> 1.) Tar stores the file modification time
> This can be avoided by setting the mtime to a fixed string for all
> builds, e.g.
> export TAR_OPTIONS=--mtime='2000-01-01'
> before calling mgwport.
the port file's modification date, or the issue date of the belonging
package file? The latter probably is a bit harder to do in a nice way
as mgwport usually doe snot know about mingw-get's package files.
> 2.) The PE header of the executables/dlls/libraries has a time stamp
>
> Binutils-2.24-1 which is not yet in mingw-get can be told to produce
> deterministic builds. Unfortunately this is only the default for .exe
> and .dll, and not for dll.a and .a. See
> https://sourceforge.net/p/mingw/bugs/2158.
>
> A work around would be to call strip --enable-deterministic-archive on
> all relevant files before packaging them. This option is also known by
> the current binutils package.
version that is configured with "--enable-deterministic-archive", or
should we modify mgwport (which already calla strip anyway, see
src_postinst.mgwpart) to also pass "--enable-deterministic-archive" to
strip? The latter probably is less work given that we need to modify
mgwport for 1) anyway.
> 3.) Gzip 1.3.12 produces non-deterministic results
> Obvious in man pages from the doc packages. See also
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=647522. Supposedly
> fixed in
> http://git.savannah.gnu.org/cgit/gzip.git/commit/?id=0a284baeaedca68017f46d2646e4c921aa98a90d.
> Upgrading to gzip 1.6 might solve the problem.
fixed the problem, and eventually create PRs for these new packages
;-)
--
Sebastian Schuberth
Thomas Braun
Aug 29, 2014, 2:29:49 PM8/29/14
to Sebastian Schuberth, mingwgi...@googlegroups.com
Am 29.08.2014 um 17:44 schrieb Sebastian Schuberth:
> On Thu, Jun 12, 2014 at 9:05 PM, Thomas Braun
> <thomas...@virtuell-zuhause.de> wrote:
>
> Just to follow up on this:
>
>> Getting there involves solving atm the following problems:
>> 1.) Tar stores the file modification time
>> This can be avoided by setting the mtime to a fixed string for all
>> builds, e.g.
>> export TAR_OPTIONS=--mtime='2000-01-01'
>> before calling mgwport.
>
> How about modifying mgwport in a way so that it sets TAR_OPTIONS to
> the port file's modification date, or the issue date of the belonging
> package file? The latter probably is a bit harder to do in a nice way
> as mgwport usually doe snot know about mingw-get's package files.
Well I'm not sure about file modification dates. Git will always reset
> On Thu, Jun 12, 2014 at 9:05 PM, Thomas Braun
> <thomas...@virtuell-zuhause.de> wrote:
>
> Just to follow up on this:
>
>> Getting there involves solving atm the following problems:
>> 1.) Tar stores the file modification time
>> This can be avoided by setting the mtime to a fixed string for all
>> builds, e.g.
>> export TAR_OPTIONS=--mtime='2000-01-01'
>> before calling mgwport.
>
> How about modifying mgwport in a way so that it sets TAR_OPTIONS to
> the port file's modification date, or the issue date of the belonging
> package file? The latter probably is a bit harder to do in a nice way
> as mgwport usually doe snot know about mingw-get's package files.
them on checkouts. Someting like commiter date of the last commit should
be more reliable. And issue dates of the pkg files are a bit far fetched
for me.
>> 2.) The PE header of the executables/dlls/libraries has a time stamp
>>
>> Binutils-2.24-1 which is not yet in mingw-get can be told to produce
>> deterministic builds. Unfortunately this is only the default for .exe
>> and .dll, and not for dll.a and .a. See
>> https://sourceforge.net/p/mingw/bugs/2158.
>>
>> A work around would be to call strip --enable-deterministic-archive on
>> all relevant files before packaging them. This option is also known by
>> the current binutils package.
>
> Now that we have binutils-2.24-1 upstream, should we build our own
> version that is configured with "--enable-deterministic-archive", or
> should we modify mgwport (which already calla strip anyway, see
> src_postinst.mgwpart) to also pass "--enable-deterministic-archive" to
> strip? The latter probably is less work given that we need to modify
> mgwport for 1) anyway.
I've already started in
https://github.com/t-b/mgwport/commits/deterministic-builds, just needs
the TAR_OPTIONS tweak mentioned earlier.
>> 3.) Gzip 1.3.12 produces non-deterministic results
>> Obvious in man pages from the doc packages. See also
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=647522. Supposedly
>> fixed in
>> http://git.savannah.gnu.org/cgit/gzip.git/commit/?id=0a284baeaedca68017f46d2646e4c921aa98a90d.
>> Upgrading to gzip 1.6 might solve the problem.
>
> It'd be great if you could find the time to check whether gzip 1.6.
> fixed the problem, and eventually create PRs for these new packages
> ;-)
https://github.com/t-b/mingwGitDevEnv-packages/commits/msys-gzip which
according to the above mentioned commit fixes the problem.
I'll submit a PR once we moved everything to git-for-windows ;)
Sebastian Schuberth
Aug 29, 2014, 2:32:00 PM8/29/14
to Thomas Braun, mingwgi...@googlegroups.com
On Fri, Aug 29, 2014 at 8:29 PM, Thomas Braun
<thomas...@virtuell-zuhause.de> wrote:
> be more reliable. And issue dates of the pkg files are a bit far fetched
> for me.
Agreed.
<thomas...@virtuell-zuhause.de> wrote:
> be more reliable. And issue dates of the pkg files are a bit far fetched
> for me.
> I have a new gzip in
> https://github.com/t-b/mingwGitDevEnv-packages/commits/msys-gzip which
> according to the above mentioned commit fixes the problem.
> I'll submit a PR once we moved everything to git-for-windows ;)
move the repos.
--
Sebastian Schuberth
Thomas Braun
Aug 29, 2014, 4:48:20 PM8/29/14
to Sebastian Schuberth, mingwgi...@googlegroups.com
Am 29.08.2014 um 20:32 schrieb Sebastian Schuberth:
> On Fri, Aug 29, 2014 at 8:29 PM, Thomas Braun
> <thomas...@virtuell-zuhause.de> wrote:
>
>> be more reliable. And issue dates of the pkg files are a bit far fetched
>> for me.
>
> Agreed.
https://github.com/t-b/mgwport/commit/43fc6f7815a4d3ccac2b4a445f1cbb6306a1ca05
> On Fri, Aug 29, 2014 at 8:29 PM, Thomas Braun
> <thomas...@virtuell-zuhause.de> wrote:
>
>> be more reliable. And issue dates of the pkg files are a bit far fetched
>> for me.
>
> Agreed.
>> I have a new gzip in
>> https://github.com/t-b/mingwGitDevEnv-packages/commits/msys-gzip which
>> according to the above mentioned commit fixes the problem.
>> I'll submit a PR once we moved everything to git-for-windows ;)
>
> Perfect. I'll try to merge the outstanding PRs ASAP so that we can
> move the repos.
Reply all
Reply to author
Forward
0 new messages