Bug#894476: RCC: provide option to encode EPOCH timestamp

[Thomas Preud'homme]

Package: qtbase5-dev-tools
Version: 5.9.2+dfsg-12
Severity: wishlist
Tags: upstream
User: reproduci...@lists.alioth.debian.org
Usertags: timestamps toolchain

Hi,

While investigating ultracopier's lack of build reproducibility, I found
out that rcc encodes the timestamp of the files the QRC file being
compiled references (see end of RCCFileInfo::writeDataInfo()). This
becomes a problem for generated files because the output of rcc is then
different in 2 different builds. It would be nice if rcc had an option
to encode a stable timestamp, eg. EPOCH.

Best regards,

Thomas

-- System Information:
Debian Release: buster/sid
APT prefers unstable
APT policy: (900, 'unstable'), (500, 'unstable-debug'), (500, 'stable-updates'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.15.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE=fr_FR:fr (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages qtbase5-dev-tools depends on:
ii libc6 2.27-2
ii libgcc1 1:8-20180321-1
ii libqt5core5a [qtbase-abi-5-9-2] 5.9.2+dfsg-12
ii libqt5dbus5 5.9.2+dfsg-12
ii libstdc++6 8-20180321-1
ii perl 5.26.1-5
ii qtchooser 64-ga1b6736-5
ii zlib1g 1:1.2.8.dfsg-5

qtbase5-dev-tools recommends no packages.

qtbase5-dev-tools suggests no packages.

-- no debconf information

[Chris Lamb]

Hi,

> While investigating ultracopier's lack of build reproducibility, I found
> out that rcc encodes the timestamp of the files the QRC file being
> compiled references

Indeed, we've been tracking this for a while in the Reproducible Builds
project [0] but, without a patch, we did not wish to bother you with a
bug. :)

It is affecting at least 4 or 5 other packages. I had previously tracked
it down to [1].

Regarding the solution, I think I would actually prefer to see something
consuming SOURCE_DATE_EPOCH[1] rather than adding an option to set a
specific timestamp.

[0] https://reproducible-builds.org/
[1] https://sources.debian.org/src/qtbase-opensource-src/5.9.2+dfsg-12/src/tools/rcc/rcc.cpp/#L207-L212
[2] https://reproducible-builds.org/specs/source-date-epoch/


Best wishes,

--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org / chris-lamb.co.uk
`-

[Lisandro Damián Nicanor Pérez Meyer]

Thanks for the bug! And sorry for the to posting :-(

I'll try to take a look into this.

[Thomas Preud'homme]

On April 3, 2018 7:25:03 PM GMT+01:00, Sune Vuorela <su...@debian.org> wrote:

On Saturday, March 31, 2018 12:05:02 PM CEST Chris Lamb wrote:
 While investigating ultracopier's lack of build reproducibility, I found
 out that rcc encodes the timestamp of the files the QRC file being
 compiled references

I don't actually see why this should be a problem. If input changes, output 
changes.
I do think that using touch(1) on the input should allow different output.

It is quite easy to reproduce:

qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 1
qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 2
qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 3

Gives same output.
Even adding in touch ../qimage.qrc keeps the same output.

qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 4

touch ../images/image.bmp

qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 5

is needed to get different output

14ef6dae8e4992ce907948c1c4af272b  1
14ef6dae8e4992ce907948c1c4af272b  2
14ef6dae8e4992ce907948c1c4af272b  3
14ef6dae8e4992ce907948c1c4af272b  4
54c6f8c09a347955ae2f36e68bbd2539  5


So. What touches the files?

/Sune

Hi Sune,

The problematic files are Qt message files (ie .qm files) generated at build time via lrelease from translation files (ie .ts files). Therefore two different builds will generate those .qm files at different times which will end up with different cpp files generated by rcc. Currently I'm working around it by setting the modified time of those .qm files to EPOCH after they are generated. I think it would be nice if there was a way for rcc to avoid doing that manually. I agree with Chris that honouring SOURCE_DATE_EPOCH in rcc would be a nice solution.

Best regards,

Thomas

[Sune Vuorela]

On Saturday, March 31, 2018 12:05:02 PM CEST Chris Lamb wrote:

> > While investigating ultracopier's lack of build reproducibility, I found
> > out that rcc encodes the timestamp of the files the QRC file being
> > compiled references

--
I didn’t stop pretending when I became an adult, it’s just that when I was a
kid I was pretending that I fit into the rules and structures of this world.
And now that I’m an adult, I pretend that those rules and structures exist.
- zefrank

[Chris Lamb]

Hi Sune!

> I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal
> circumstances

Can you elaborate on what you mean by "normal circumstances"? :)

How about e use S_D_E if it is setup/exported, otherwise use the mtime
of the file as before?

(This is the pattern/logic used extensively elsewhere in toolchains for
this very exact purpose; it would not be unique to rcc)

[Sune Vuorela]

On Tuesday, April 3, 2018 8:46:51 PM CEST Thomas Preud'homme wrote:
> The problematic files are Qt message files (ie .qm files) generated at build
> time via lrelease from translation files (ie .ts files). Therefore two

Right. I didn't think about autogenerated files during build.

> I think it would be nice if there was a way for
> rcc to avoid doing that manually. I agree with Chris that honouring
> SOURCE_DATE_EPOCH in rcc would be a nice solution.

I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal

circumstances, given that I do think that having the actual timestamp of the
non-generated files makes sense.

I have reached out to upstream about it.

[Sune Vuorela]

On Tuesday, April 3, 2018 9:14:23 PM CEST Chris Lamb wrote:
> Hi Sune!
>
> > I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal
> > circumstances
>
> Can you elaborate on what you mean by "normal circumstances"? :)

"normal circumstances" is rcc on a source file, as opposed to an autogenerated
file.

> How about e use S_D_E if it is setup/exported, otherwise use the mtime
> of the file as before?

I think that using S_D_E only makes sense if rcc is run on an autogenerated
file, but I do think that most rcc runs is run on existing source files.


I don't have a good idea to differentiate those.

[Chris Lamb]

Hi Sune,

> "normal circumstances" is rcc on a source file, as opposed
> to an autogenerated file

I'm not /entirely/ sure what the difference is as I'm not in the
Qt/RCC world too often these days (alas !), but why just not use
SOURCE_DATE_EPOCH *iff* it is exported?

Normal systems simply do not have this envvar set, so there is really
no danger of it leaking elsewhere or causing unintended side-effects.

It would also have the benefit of not having to differentiate between
the two... :)

[Thomas Preud'homme]

On April 3, 2018 8:20:22 PM GMT+01:00, Sune Vuorela <su...@debian.org> wrote:

On Tuesday, April 3, 2018 9:14:23 PM CEST Chris Lamb wrote:
 Hi Sune!
 
 I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal
 circumstances
 
 Can you elaborate on what you mean by "normal circumstances"? :)

"normal circumstances" is rcc on a source file, as opposed to an autogenerated 
file.

 How about e use S_D_E if it is setup/exported, otherwise use the mtime
 of the file as before?

I think that using S_D_E only makes sense if rcc is run on an autogenerated 
file, but I do think that most rcc runs is run on existing source files.


I don't have a good idea to differentiate those.

/Sune

One small clarification: in my case rcc *is* run on a nongenerated resource file. It's some of the files that the resource file list that are generated and whose timestamp end up in the cpp file generated by rcc.

In other hand:

foo.qrc mentions bar.qm which is generated from bar.ts.

rcc foo.qrc generates resource_bar.cpp which contains constant data that encodes bar.qm timestamp and this create different resource_bar.o at every build.

Best regards,

Thomas

[Sune Vuorela]

On Tuesday, April 3, 2018 9:24:58 PM CEST Chris Lamb wrote:
> I'm not /entirely/ sure what the difference is as I'm not in the
> Qt/RCC world too often these days (alas !), but why just not use
> SOURCE_DATE_EPOCH *iff* it is exported?
>
> Normal systems simply do not have this envvar set, so there is really
> no danger of it leaking elsewhere or causing unintended side-effects.

We don't know how application users are using this.

The following is some theoretical example, that I'm quite sure could be used
in some real world applications:

QFileInfo embeddedFile(":/foo/data.xml"); // fallback downloaded in the
ancient past
QFileInfo systemfile("/usr/share/foo/data.xml");

if (systemfile.lastModified() > embeddedFile.lastModified())
{
data = readFromFile(systemFile);
}
else
{
data = readFromFile(embeddedFile);
}

If S_D_E gets used, rather than the data.xml modified date in the source, this
will end up using the wrong file if S_D_E is newer than the system copy of the
file.

This is not a unused databit, but a fully available piece of metadata for the
files.

[Lisandro Damián Nicanor Pérez Meyer]

El mar., 3 de abr. de 2018 16:42, Sune Vuorela <su...@debian.org> escribió:

On Tuesday, April 3, 2018 9:24:58 PM CEST Chris Lamb wrote:
> I'm not /entirely/ sure what the difference is as I'm not in the
> Qt/RCC world too often these days (alas !), but why just not use
> SOURCE_DATE_EPOCH *iff* it is exported?
>
> Normal systems simply do not have this envvar set, so there is really
> no danger of it leaking elsewhere or causing unintended side-effects.

We don't know how application users are using this.

The following is some theoretical example, that I'm quite sure could be used
in some real world applications:

QFileInfo embeddedFile(":/foo/data.xml"); // fallback downloaded in the
ancient past
QFileInfo systemfile("/usr/share/foo/data.xml");

if (systemfile.lastModified() > embeddedFile.lastModified())
{
   data = readFromFile(systemFile);
}
else
{
   data = readFromFile(embeddedFile);
}

If S_D_E gets used, rather than the data.xml modified date in the source, this
will end up using the wrong file if S_D_E is newer than the system copy of the
file.

This is not a unused databit, but a fully available piece of metadata for the
files.

I'm afraid Sune is right here, it might be used that way. Questionable? Sure thing, but still valid code.

But after all the same can be said from embedding translations into the binary itself.

So yes, we will need help from upstream with this one.

 

[Thomas Preud'homme]

Maybe the solution is then not in rcc but in whatever generate the files that the qrc that rcc processes refer to. For instance in the case of ultracopier lrelease could have a mean if generating .qm files with the same modified timestamp as the .ts file it processes. This would guarantee stable output for a stable source and this later on rcc would also generate stable output.

Thoughts?

Best regards,

Thomas

[Chris Lamb]

Hi Thomas,

> Maybe the solution is then not in rcc but in whatever generate the files
> that the qrc that rcc processes refer to. For instance in the case of
> ultracopier lrelease could have a mean if generating .qm files with the
> same modified timestamp as the .ts file it processes

That sounds workable..


Regards,

[Thorsten Glaser]

affects 894476 src:musescore
retitle 894476 rcc: please honour SOURCE_DATE_EPOCH
thanks

Hi *,

I just got bitten by this in an FTBR of src:musescore which I tracked
down to this bug.

On Tue, 3 Apr 2018, Sune Vuorela wrote:

> If S_D_E gets used, rather than the data.xml modified date in the
> source, this will end up using the wrong file if S_D_E is newer than
> the system copy of the file.

AIUI, S_D_E is used as a *delimiter*, not as a timestamp *all* files
are to be set to.

So, basically, instead of

encode(dst, sb.st_mtime);

you do

encode(dst, MIN(sb.st_mtime, s_d_e_as_time_t_value));

and not

encode(dst, s_d_e_as_time_t_value);

which would indeed break things the way you described.


Chris/H01ger, please correct me if I’m wrong, but this is
how I understood S_D_E to work.


I don’t know cmake magic enough to be able to do the same
patch to lrelease et al. as Thomas did, so I unfortunately
will have to live with the FTBR in src:musescore until this
issue is fixed.

Thanks,
//mirabilos
--
>> Why don't you use JavaScript? I also don't like enabling JavaScript in
> Because I use lynx as browser.
+1
-- Octavio Alvarez, me and ⡍⠁⠗⠊⠕ (Mario Lang) on debian-devel

[Chris Lamb]

Hi Thorsten,

> AIUI, S_D_E is used as a *delimiter*, not as a timestamp *all* files
> are to be set to.

I've found that having a shared vocabulary is very useful so just in
case it helps we tend to use the term "clamping" when referring to
this MIN(a,b) concept rather than "delimiter" or similar. eg. "foo
clamps the mtimes to SOURCE_DATE_EPOCH".

> Chris/H01ger, please correct me if I’m wrong, but this is
> how I understood S_D_E to work.

It is not required. Does the canonical specification not explain?

https://reproducible-builds.org/specs/source-date-epoch/

I pedantically refer you to that /only/ because if it is not clear
enough then there is a need to revise it. :)

[Thorsten Glaser]

Hi Chris,

> It is not required.

I’m parsing this as “you were not incorrect”, right?
(Sorry, non-English-native speaker here.)

> Does the canonical specification not explain?
>
> https://reproducible-builds.org/specs/source-date-epoch/
>
> I pedantically refer you to that /only/ because if it is not clear
> enough then there is a need to revise it. :)

It is indeed clear enough. I didn’t have it at hand when writing
my previous message and only vaguely recalled having read it
quite some time before. (Thanks for linking it in this thread.)

> > AIUI, S_D_E is used as a *delimiter*, not as a timestamp *all* files
> > are to be set to.
>
> I've found that having a shared vocabulary is very useful so just in
> case it helps we tend to use the term "clamping" when referring to

True… it’s just that I didn’t recall the word, as “clamping”
is not something normally part of my English vocabulary (and,
having looked it up in the dictionary, nothing I’d normally
associate with this), so it’s probably a “lost in translation”
effect. I’ll try to remember next time.

Thanks,
//mirabilos
--
15:41⎜<Lo-lan-do:#fusionforge> Somebody write a testsuite for helloworld :-)

[Chris Lamb]

Hi Thorsten,

> > It is not required.
>
> I’m parsing this as “you were not incorrect”, right?

It doesn't really matter whether you were correct or incorrect in the
past and seems a little silly to debate that; far easier and clearer
IMHO to statelessly state that clamping is not required when using
SOURCE_DATE_EPOCH. :)

> > I've found that having a shared vocabulary is very useful so just in
> > case it helps we tend to use the term "clamping" when referring to
>
> True… it’s just that I didn’t recall the word, as “clamping”

> is not something normally part of my English vocabulary […]

I might agree it's esoteric word/usage, but not only is it already "out
there", we have more problems with "reproducible" to worry about this
one too much alas.

> I’ll try to remember next time.

Again, no question of blame or apologies here… :)


Regards,

[Chris Lamb]

Thorsten,

> > far easier and clearer IMHO to statelessly state that clamping is not
> > required when using SOURCE_DATE_EPOCH. :)
>

> OK, now I do not understand _that_…

(It's clearer to say "X is the situation" vs "when you said Y before
that was correct/incorrect")

[Thorsten Glaser]

On Mon, 9 Jul 2018, Chris Lamb wrote:

> far easier and clearer IMHO to statelessly state that clamping is not
> required when using SOURCE_DATE_EPOCH. :)

OK, now I do not understand _that_…

Doesn’t matter as long as this bug gets fixed, though ☻☺

Have a nice day,

[Bernhard M. Wiedemann]

Is this the same as this upstream bug?
https://bugreports.qt.io/browse/QTBUG-62511
that had this patch merged
https://codereview.qt-project.org/#/c/202999/5/src/tools/rcc/rcc.cpp
with a renamed SOURCE_DATE_EPOCH env var

[Lisandro Damián Nicanor Pérez Meyer]

forwarded 894476 https://bugreports.qt.io/browse/QTBUG-62511
thanks

Indeed, that's right. Thanks a lot Bernhard!

--
Existe un problema cultural, el que lleva -por ejemplo- a los padres a
comprar una computadora y ponerla en la habitación de los varones. La
computadora tiende a ser vista como un objeto "masculino", por lo cual
el número de mujeres interesadas es menor.
Margarita Manterola, Debian Developer, en un mail sobre las mujeres
en el software libre, LugFi.
http://listas.fi.uba.ar/pipermail/lug/2005-September/020266.html

Lisandro Damián Nicanor Pérez Meyer

http://perezmeyer.com.ar/
http://perezmeyer.blogspot.com/

[Vagrant Cascadian]

On 2018-07-11, Lisandro Damián Nicanor Pérez Meyer wrote:
> Version: 5.11.1+dfsg-1
>
> The bug upstream has been closed as invalid (see https://bugreports.qt.io/
> browse/QTBUG-62511) Non the less a workaround has been included in Qt 5.11,
> already in experimental. Setting QT_RCC_SOURCE_DATE_OVERRIDE should be enough
> to solve this issue.

The mentioned upstream bug report is simply marked as invalid without
explanation.

This is (currently) an empty search:

https://codesearch.debian.net/search?q=QT_RCC_SOURCE_DATE_OVERRIDE

Do you have a link for QT_RCC_SOURCE_DATE_OVERRIDE present in Debian and
upstream? This bug doesn't really seem resolved if that variable isn't
actually getting set by anything or consumed by anything...


Ideally QT_RCC_SOURCE_DATE_OVERRIDE would get set based on
SOURCE_DATE_EPOCH, otherwise build tools have an arbitrary growing
number of variables to keep track of for each library, toolchain,
language, etc. that implements their own way of doing this sort of
thing, which kind of defeats the purpose of SOURCE_DATE_EPOCH. *sigh*


live well,
vagrant

[Lisandro Damián Nicanor Pérez Meyer]

El mié., 11 de jul. de 2018 17:12, Vagrant Cascadian <vag...@debian.org> escribió:

On 2018-07-11, Lisandro Damián Nicanor Pérez Meyer wrote:
> Version: 5.11.1+dfsg-1
>
> The bug upstream has been closed as invalid (see https://bugreports.qt.io/
> browse/QTBUG-62511) Non the less a workaround has been included in Qt 5.11,
> already in experimental. Setting QT_RCC_SOURCE_DATE_OVERRIDE should be enough
> to solve this issue.

The mentioned upstream bug report is simply marked as invalid without
explanation.

The point is that stuff like CMake can change this value. See comments.

This is (currently) an empty search:

  https://codesearch.debian.net/search?q=QT_RCC_SOURCE_DATE_OVERRIDE

Because it did not scan experimental.

Do you have a link for QT_RCC_SOURCE_DATE_OVERRIDE present in Debian and
upstream?  This bug doesn't really seem resolved if that variable isn't
actually getting set by anything or consumed by anything...

Right now, in experimental. We hope to get the transition going soon.

Ideally QT_RCC_SOURCE_DATE_OVERRIDE would get set based on
SOURCE_DATE_EPOCH, otherwise build tools have an arbitrary growing
number of variables to keep track of for each library, toolchain,
language, etc. that implements their own way of doing this sort of
thing, which kind of defeats the purpose of SOURCE_DATE_EPOCH. *sigh*

That's sadly something we can only "fix" by making packages have the right value set. As per Qt policy the environment variable needs to be prefixed with QT, so no chance of directly using SOURCE_DATE_EPOCH.

Feel free to jump in in the upstream bug report if you feel like this is not enough.

Cheers, Lisandro.

[Bernhard M. Wiedemann]

> That's sadly something we can only "fix" by making packages have the right
> value set. As per Qt policy the environment variable needs to be prefixed
> with QT, so no chance of directly using SOURCE_DATE_EPOCH.

I added a downstream patch to a test-version [1] in openSUSE with

+ static const quint64 sourceDate2 = 1000 *
qgetenv("SOURCE_DATE_EPOCH").toULongLong();
+ if (sourceDate2 != 0)
+ lastmod = sourceDate2;

but somehow this did not help (e.g. with bitcoin's
bitcoin-0.16.1/src/qt/qrc_bitcoin_locale.cpp going into the bitcoin-qt
binary).
So maybe the patch is not the correct solution... or there are more
similar issues somewhere else in rcc.



[1]
https://build.opensuse.org/package/view_file/home:bmwiedemann:reproducible:test/libqt5-qtbase/reproducibletime.patch

[Sune Vuorela]

On Wednesday, July 11, 2018 10:08:23 PM CEST Vagrant Cascadian wrote:

> Ideally QT_RCC_SOURCE_DATE_OVERRIDE would get set based on
> SOURCE_DATE_EPOCH, otherwise build tools have an arbitrary growing

No. As explained, we need to look at each individual package to check if the
timestamp is actually used for anything. It can be. It probably is.

I think a better fix might be to specifically mark in the rcc file if the
dates are important or not. But it requires involvement upstream. (Or maybe if
the rcc file is autogenerated or not). I don't think it is a problem for non-
autogenerated rcc files.

[Thomas Preud'homme]

On jeudi 12 juillet 2018 18:28:21 BST Sune Vuorela wrote:
> On Wednesday, July 11, 2018 10:08:23 PM CEST Vagrant Cascadian wrote:
> > Ideally QT_RCC_SOURCE_DATE_OVERRIDE would get set based on
> > SOURCE_DATE_EPOCH, otherwise build tools have an arbitrary growing
>
> No. As explained, we need to look at each individual package to check if the
> timestamp is actually used for anything. It can be. It probably is.
>
> I think a better fix might be to specifically mark in the rcc file if the
> dates are important or not. But it requires involvement upstream. (Or maybe
> if the rcc file is autogenerated or not). I don't think it is a problem for
> non- autogenerated rcc files.

Agreed, it's only a problem for files autogenerated at build time. rcc on a
file that's part of the source tarball is gonna give a reproducible result.

Best regards,

Thomas

[Peter Wu]

Hi all,

To improve reproducibility for an upstream project that does not rely on
a valid QFileInfo::lastModified value, we used the --format-version 1
option to force the use of the older (pre-Qt 5.8) format that does not
include the timestamp.

This upstream project uses CMake with AUTORCC enabled, the patch was
pretty simple: https://code.wireshark.org/review/31593

For the benefit of other distributions (I also use Arch Linux) and
upstream projects, I have documented this here as well:
https://reproducible-builds.org/docs/deterministic-build-systems/#cmake-notes

If you feel that the recommendations are wrong, feel free to edit the
page or discuss it here or on IRC, I aimed at having the peculiarities
documented somewhere.
--
Kind regards,
Peter Wu
https://lekensteyn.nl