RFC: Hardening Qubes binary distribution process

331 views
Skip to first unread message

Joanna Rutkowska

unread,
May 20, 2015, 11:28:11 AM5/20/15
to qubes...@googlegroups.com, Marek Marczykowski
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

We're currently considering how to further protect Qubes binary production and
distribution process. As discussed recently in the Qubes 3.0-rc1 announcement
[1] we have already taken some steps to harden Qubes binary production process,
e.g. by introducing sandboxed builds using DispVMs for templates, and by using a
TorVM to fetch Fedora packages used by our builder to make it difficult to feed
us (people who build Qubes packages) with backdoored Fedora packages by an
hypothetical adversary who might have access to the Fedora signing key.

Still, we're quite concerned about the possibility of an adversary stealing our
binary signing keys. As one can imagine we have limited possibilities to protect
these keys against sophisticated _physical_ attacks without changing our
apartments into bunkers (which is not necessarily something we might be dreaming
of doing...).

An attacker who might have gained access to the Qubes binary signing keys might
be able to inject backdoored packages to _select_ users without others being
able to ever find out such an attack is going on. This would be very sad and
might have very unpleasant consequences to some of the users.

This attack is, of course, not specific to Qubes in any way -- it applies all
the same way to updates or ISO images downloaded from Fedora or any other
distribution.

In order to protect against such targeted attacks, the more paranoid users might
wish to use Tor for downloading of the updates. We're currently considering
whether we should make this a default option (or at least selectable during
installation) in a future Qubes release (3.1)?

Using Tor for downloading of all the updates for Qubes does not protect,
however, against a situation where the adversary (who managed to steal our
binary signing key) decides to inject maliciously backdoored packages to any and
all Qubes users. Again, there is nothing Qubes-specific in this attack scenario,
and any Linux distribution serving binary updates is just as vulnerable. Except
for the fact that the group of people who happen to be Qubes users might
constitute a somehow more interesting target group for such an attack (e.g.
because "normal Linux" users might be relatively easily attacked in a number of
other, perhaps simpler ways).

As already mentioned several times, the proper way to address this problem is to
introduce a multi-signature scheme in which every (binary) update is signed by
at least M out of N trusted keys, spread among N trusted people, residing in
different countries, having different sources of income, etc. Naturally, this
scheme requires binaries to build deterministically from the same sources, so
that each of these N people could 1) verify the sources, 2) build the binary and
see what is the hash, and 3) sign the hash with their key. Unfortunately
deterministic builds for still months or perhaps even years in front of our
times :/

Thus, in the meantime, I thought it might be reasonable to temporarily solve
this problem by not distributing binaries, but rather only source code and have
all the Qubes updates building on the users computers automatically. This would
allow to easily implement the multi-signature scheme immediately, because all
that would be needed was to check if the source code has been signed by at least
M keys from the group of N well known keys.

Yes, this would mean that updates installation would take significantly longer,
sometimes even a few hours, and would increase energy consumption and heat
production. But it will eliminate the need for Qubes binary signing keys. Yes,
an attacker who manages to steal e.g. Marek's source code signing key would be
able to inject some malicious commits, or modify existing ones, but such
backdooring should be significantly easier to detect by others (especially the
N-1 other trusted people) than the binary modifications.

I would like now to discuss the above idea and hear what people think about it.

One more problem: how to distribute the ISO, which would become the weakest link
in this scheme?

Thanks,
joanna.

[1] http://blog.invisiblethings.org/2015/04/23/qubes-30rc1-and-roadmap.html
-----BEGIN PGP SIGNATURE-----

iQIcBAEBAgAGBQJVXKgAAAoJEDOT2L8N3GcY4kIQANZpemsTeoWaGQMHcUiV70gX
OcAL8Wp1nmQpHeeUOmufVxVaYnVMiXjEM9bOEno+7EbowvaNuZtBiurnfHCOSSTi
Kmy234iCpC5xgu6qNgz9b6xuvA05xRdhccWlkDVZnhei2rEfKUxfciUEJtowtnnF
HaDf8caE4mTmJ7oBFXJRpMdTGxLtiog5ehpK5nhT/tu5XmSt+WYKxWeV5QWUNlAU
CXIrUAn8Vz56BiXi4VsnKdqAoomfFDqOKsw/+dBLGsm0N/dLiMQZfmyuiZb/tjHe
FboJdQHxn5VZaVxSTkHRCZMXPuJIiPgdbrvx8WKa6onnCXY56Tf1Jb9RGIEtUSQ7
Svk5DWcha3DfXGDKoV2aWeJ5q9v768aE9MtYuGe9lDl4CfFZWVEmmtU39NH4Zvzd
uEGqIfIEE6yZu4mrp2kbOEykY3oArfVVl1nU+uVyjz2vcqgwAQUHdyCOcM/uAlhj
SOIVst8fPdhs37z9XoBxG1W4xUXJDcYHUZAzGqPiOwYG36g6H8qnvJE0686g9lkp
/eLTN+RmrE04JzJ5AjzothvGsEwsWvFIbRaR4yGtTSNxq1JOa7dR8Qe0FvRu1HD1
ZJTBU1NJSI7rMXNwL3oMrdeB5Zt4nhVkNvcy7pjRwxUUtkyebbfNQyWEwhOT1LMH
ijEmMQXWRos9aieh1rqb
=rP+3
-----END PGP SIGNATURE-----

Franz

unread,
May 20, 2015, 3:30:36 PM5/20/15
to Joanna Rutkowska, qubes...@googlegroups.com, Marek Marczykowski
It may be possible to make an arrangement with some trust-able organization which may sell the numbered DVDs in the main cities of various countries. Price will cover all costs and perhaps leave also something for you.

Otherwise, this may also be a way to get Qubes known outside of the usual circle of software developer. For example let me suppose you are able to make an arrangement with some large financial institution that has branches in many countries. This may be an exclusivity agreement for a limited time.  This institution may be interested not for the money (in this case the DVD may probably be distributed free of charge), but because this improves their image because it shows they take security seriously supporting the more advanced (and more complex) open-source effort to face security risk. But this would obviously mean a lot for you too, because a lot of people that trust this financial institution will trust you too. It would also rise the problem of users who are unable to install it, understand it etc. Well, it is a real challenge.
Best
Fran

 
Thanks,
joanna.

[1] http://blog.invisiblethings.org/2015/04/23/qubes-30rc1-and-roadmap.html
-----BEGIN PGP SIGNATURE-----

iQIcBAEBAgAGBQJVXKgAAAoJEDOT2L8N3GcY4kIQANZpemsTeoWaGQMHcUiV70gX
OcAL8Wp1nmQpHeeUOmufVxVaYnVMiXjEM9bOEno+7EbowvaNuZtBiurnfHCOSSTi
Kmy234iCpC5xgu6qNgz9b6xuvA05xRdhccWlkDVZnhei2rEfKUxfciUEJtowtnnF
HaDf8caE4mTmJ7oBFXJRpMdTGxLtiog5ehpK5nhT/tu5XmSt+WYKxWeV5QWUNlAU
CXIrUAn8Vz56BiXi4VsnKdqAoomfFDqOKsw/+dBLGsm0N/dLiMQZfmyuiZb/tjHe
FboJdQHxn5VZaVxSTkHRCZMXPuJIiPgdbrvx8WKa6onnCXY56Tf1Jb9RGIEtUSQ7
Svk5DWcha3DfXGDKoV2aWeJ5q9v768aE9MtYuGe9lDl4CfFZWVEmmtU39NH4Zvzd
uEGqIfIEE6yZu4mrp2kbOEykY3oArfVVl1nU+uVyjz2vcqgwAQUHdyCOcM/uAlhj
SOIVst8fPdhs37z9XoBxG1W4xUXJDcYHUZAzGqPiOwYG36g6H8qnvJE0686g9lkp
/eLTN+RmrE04JzJ5AjzothvGsEwsWvFIbRaR4yGtTSNxq1JOa7dR8Qe0FvRu1HD1
ZJTBU1NJSI7rMXNwL3oMrdeB5Zt4nhVkNvcy7pjRwxUUtkyebbfNQyWEwhOT1LMH
ijEmMQXWRos9aieh1rqb
=rP+3
-----END PGP SIGNATURE-----

--
You received this message because you are subscribed to the Google Groups "qubes-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel...@googlegroups.com.
To post to this group, send email to qubes...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/20150520152801.GB25294%40work-mutt.
For more options, visit https://groups.google.com/d/optout.

Rusty Bird

unread,
May 20, 2015, 3:42:27 PM5/20/15
to qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi Joanna,

Have you considered starting an open mailing list for distribution
agnostic discussion about deterministic builds, and inviting all the
developers working in that field?

It seems like there's a lot going on with Gitian, Bitcoin Core, Tor
Browser, Debian, Fedora, Whonix, F-Droid, Bazel and probably many
more, but no central place to share new tricks, avoid dead ends, hatch
splendid plans...

Rusty
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJVXOOLAAoJEEadePR6ryrfB9EQAJZByCSi28f2CiYjLrrZ/Moz
rcc5FCOegJn4vS+xi/8/ctSNf8Q9Q9V4xIn0Gafe+f1+u5RzUDh70nyuSB2S+7/p
KJE33wp6jNpUtr/IeMCJbeGdgM1pr/rXmWRzaa4PaGD6a+KNS2D/OWxnmCDIjEAM
l38OCctjmLk1FQhXTkTq9frhrYX+Mwp1GfynKogXC4HCC0CCVgEWF5vGozMO1Jdh
kQG4mvMUojbL4sJ4+u+EHuEbVFKt/8ZpJsLRURjAGehlxT6h96ZzwPcFWEI+tmlU
irnX+i1agCYSJ67GTKSBheA5xotKDIg0fF8Ev80DDdP2CUy43DUWZEEgNsiks/2a
jFQSlE9aYxsTIDul7HWRWAukt6qXjGtGJxCMbRYJcA46xb+EoMB6HtMbXrlkU5Gk
2UXtY+kwFR7ZhxxRx41+gP+4ffSe5bIGZAJRNe8luc77vaQV2ijjlo6B6kQsmFgp
uMecS9M3oyfVK8kEW2kFFFdi2/mbmRLY4VMFJwjcqZ4aSwTk+cuUMwG3h6pl6aja
0zXkNbwhAKEB4lkhb4XQbDa/iVLpoHdqf1C4aGGpAJinCeNBmyU9r5EhqUn2svX4
xJrUBR+4OVm2odAfk8kx4UWwXZlJmnCVMlu+9F8P+Eexry+kLFHEENn6NZOjrCKf
hM/xWT95G/yDUK3aozxs
=HMO/
-----END PGP SIGNATURE-----

Jason M

unread,
May 20, 2015, 6:51:02 PM5/20/15
to qubes...@googlegroups.com, joa...@invisiblethingslab.com, marm...@invisiblethingslab.com

Personally I think it would be a great option, but please don't make it the default. I personally do not like any of the distributions that install by building source code since it takes so long, and is not always reliable.  Network issues, processor abilities tend to get in the way.  So if a user can decide to either build from source or download binary, that would be an acceptable solution for me.

I actually think time should rather be spent on making deterministic builds though over implementing such a feature since users can already choose to build from source now, and it is not difficult to update modules using same method; maybe just create a FAQ on how to do it.

That's my 2 cents worth :)
 

One more problem: how to distribute the ISO, which would become the weakest link
in this scheme?

Provide an ISO that builds another ISO? LOL

HW42

unread,
May 20, 2015, 8:49:38 PM5/20/15
to Joanna Rutkowska, qubes...@googlegroups.com, Marek Marczykowski
Joanna Rutkowska:
Why do you think that this is significantly easier than getting
reproducible builds?

I think building a build system which is as reliable as the normal
packet manager for _all_ the code used is also rather tricky.

Whether multisigned source or reproducible build you have rather similar
problems:

1) The Qubes specific code/parts.

That's the easy part. It may need some tweaks to get reproducible or
intergrated it into the multisigned build system but since we have full
control that should be easy to archive.

2) Distribution system.

In case of reproducible builds you need to modify the verification part
of the packet manager. With the multisigned source approach this is a
bit more complex but still manageable (the tool downloads the code;
verify it; calls qubes-builder; install the built packages).

3) The rest of the system.

I think thats where the most work is needed. You need to get _all_ used
packages to get successfully build by the update system or respectively
produce reproducible packages.

In case of multisigned source the most trickiest part is probably
getting the build dependencies build in the right order (I had cases in
which Debian packages had conflicting build deps).

In addition a question would be who sign the code of all the external
packages.

Here you could profit from efforts like the Debian reproducible builds
project [2]. (Does somebody now if fedora has a similar project?)


To be clear: I'm happy to see that you target this problem. And will try
to help regardless whether you decide in favor of the multisigned source
way or target reproducible builds directly.

>
> One more problem: how to distribute the ISO, which would become the weakest link
> in this scheme?

Without reproducible builds I think the best solution would be
distribute a signed ISO like now but additionally distribute the hash of
it in a public, verifiable, append only log (I currently don't have an
overview about such solutions so can't recommend any specific). This way
you can at least (nearly) guarantee that all users get the same ISO.
[2]: https://wiki.debian.org/ReproducibleBuilds

signature.asc

HW42

unread,
May 20, 2015, 9:03:45 PM5/20/15
to Joanna Rutkowska, qubes...@googlegroups.com, Marek Marczykowski
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

[ I think I have now finally configured my MUA to use Inline-PGP
automatically for the Qubes-MLs. Sorry for the noise]

HW42:
After writing the last mail I realized that I'm not sure if you are
talking only about the Qubes specific packages or also about the
fedora packages (like I assumed in the text below)?
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJVXS7YAAoJEOSsySeKZGgW148QAKBh4KnJpM5fyBPUxS4AyBbd
1jp3VnKRqn7160MyEWKPhQ6UKmc7hnfRrCrifQm1YtE0vtTZ/jI5Xj8hLhOFT6rY
GK2ZLGjraNrXEjxh+jgk4F25HmKs2HYC6qGfuc6OX+BMnJImLpPoBO+DsLybEcLr
57wkdzIaLyDvyhxlDdGt1w4JMsKZTincbh5JFhvwU0JqpXKzwB6YlINUscF8GuyX
fDSGVhyAeCqhKibxuDZnHroaX/xwQ9sEHYHkC8Q2GUoKXxGZenk0LmCSr/kxVTUY
N9kKaJc6hKZhyP2e2j9H4lPdjE63GfL3JfUWPHRgz4H0/BQ6mOeVtMbhCGQPCgzI
gTerv2D/N1DyTHpm4s1WW/OS3NAxKLcjfu83xuKVo+8aFlv1vQT9/KL5C8SK/XjY
7GiIhR7W70TMHWdn83UNEPuAciXMlSb5gEHdi8G8UWIgkURhGsa+p0/Auz/Nrge4
PP0RTUqdotAeQTrc3XOw4w3SW9ujHB5SbQwwFSDhPyBBcxspBj1Tn14VjBgkUPQX
C+LDDHFZAk6aJiL+dVfc4GiwMeU/rnGdgSkXz8CXXvRcCGlg4MDSqilXPSmPGla2
iDW0ZvC8ZPKmH+yKZVByBOJQsuRquHTAGPtAdHvXv4ElAn2LZyb/PnjhJtx1kvdI
aXh8OA75/c2KHlVYyM2Y
=+8gI
-----END PGP SIGNATURE-----

Alex Dubois

unread,
May 22, 2015, 4:56:10 PM5/22/15
to qubes...@googlegroups.com, joa...@invisiblethingslab.com, hw...@ipsumj.de, marm...@invisiblethingslab.com

Another option could be to identify a tool that build a hash graph (of all files) which ignore/strip files's timestamps (and other possible metadata that are not relevant from a security stand point but change the hash/signing). Is it not conceivable/easier or it the binary produced also changing depending on hardware used (assuming compile options are obviously the same)?

As for the original option it merge with the Gentoo spirit... which I prefer.

Marek Marczykowski

unread,
Jun 24, 2015, 8:51:51 PM6/24/15
to Joanna Rutkowska, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

While the above idea can solve some problems, IMO it still doesn't
eliminate need of trust in some binary packages - the ISO is one
example, but also Fedora/Debian/whatever updates, or template packages
(distributed as rpm, or some other format).

Also what about installing new packages (like qubes-specific addons)?
Also compile on your own? This would mean that we need to write some
subset of package manager to have it usable for "normal user". While
Qubes Builder does most of the things automatically, it still needs some
understanding how package build process is working (for example
correctly setup COMPONENTS, including proper dependencies). Jason has
done a good job here by writing 'setup' tool for selecting most common
configurations, but still it is far from tool for every user to build
selected packages (either new one, or updates).

Such source-based approach would require a lot of work on the update
tool to be usable for normal user, not only power users. For example
what about GUI for that? This would require at least writing some
PackageKit backend.

Maybe we should switch default distribution to something based on source
packages - Gentoo?

Regarding template packages - of course you can say that the user can
build it on his/shes machine. But given the amount of time(*) and disk
space needed(**), it doesn't sound like a practical solution. We want to
have system usable for average user in the end (otherwise we wouldn't
care about the ability to run all the legacy applications...).

(*) for example building Whonix template takes about 2h when starting
from clean environment
(**) about 10-20G

Generally I'm not convinced that implementing this temporary solution on
top of current Qubes OS (for example Fedora-based dom0), is
significantly easier than implementing proper deterministic builds for
the few packages that we have. Note that, while the update process needs
to be easy for every user, the build can require significantly more
knowledge (for example need to setup some additional VMs with strictly
controlled environment) - so require somehow less polishing, especially
in terms of user interface.

Some time ago I've done a simple test: built linux-utils component under
faketime environment. With a few minor tweaks I was able to build exactly
the same rpm package every time, on two different machines (one of them
was not even Qubes-based). So maybe deterministic builds are not as hard
(in our case) as you think? Maybe we should first try this approach and
when we fail (or not succeed in reasonable time), then think of some
alternative?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVi1BvAAoJENuP0xzK19cshs8IAJXcVgfth0sjjT3XbCT1OREz
cvM9eg8t3E6NnpqAJu/Il4/KTJmGJgXpLTZnaaIPAJsgGNMENO9VfdpNK8Z78jxM
+/c11XT/8HXGSsYoVPE/Y3xAMchrep0frpfdp+FR8CjiY968DnRixs8nP/RtlfB6
luK5ymIJKUcfe62ZqmDkAueMXwrTmkZPzEE63SqzkKj3xsI9IG1DAkjQQ6OishHK
nBic8w6ON/MT0Srvs2hRFiQ7bJXyPAVMKAhdhDrkkCjbqyiQh0pS2C9z33B01JKa
3bEhGlbfbSy4DRVMJUjcyp/Fg9dBvhcyXxURV7rzNcHDPyRE6XIk9r363N2eT6M=
=s3Vw
-----END PGP SIGNATURE-----

Zrubi

unread,
Jun 25, 2015, 12:02:14 PM6/25/15
to qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 06/25/2015 02:50 AM, Marek Marczykowski wrote:
> Maybe we should switch default distribution to something based on
> source packages - Gentoo?

:)

Just searched when I've suggested that at the first time:

https://groups.google.com/d/topic/qubes-devel/WIOZzdQ9oow/discussion

Still think that portage (gentoo) is the best for the task ;)


- --
Zrubi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVjCYAAAoJEACyhZsQIQUVldgP+wSZnDbqIA6+3DemTjM+9vK7
fH56f15GWf6JZvo1PzTdcoAK1J+sdmAH8HeXos4EcgB77HX0NWQsxro4ptcrAGzW
GpuaQ9Y4XBlYwfb+xOwE29vW4lv/LbTa1mRcuATFTrqZf2uPkeigI38mIcPZBROZ
CO3uOmx2JcALXLVi/Ok8XaFkesO2P4vN1C52OpIYJbwrHwM2mVkaZxmH/408jCVm
YWhj+L8lSW1M5LKIwAU4qmkU5H95SLf++G2zqS4rY/ldgmJvFTNW9ebN1eKvPkjU
UVjEJXMr/Nq0BS1wJediU2/JbP6dUA5pKHteblVzJ52uhYyGyosw+gE3VR0nK6B3
xo6x6mSz16U2T2xO4EWZUTxiaDPHi4LiZM9BktrqsZXg2PL3myF4b8glbe/Cwfo/
LlNxsTHYKGPmjb7I1HEbHMV2+0qWZ4K9MZUkG4EbDcTv64NMDdTXSBdJ9sM7s7Wu
4IE2XGV/13qIQZCLgIp8CoFrKzTDEfc9A8Uou8JniEDNxcXvoGvSBJKEjFTElsdw
wasWvNSqwaLNaXNJq3mExJNHyzhLvSqkdlBolUaCPEX3CJdVP+1+tret0FqdkW18
XCA9pFxinIWhbrfTzp6lff4nsnVf2vmxWORYyvIssqWCjF3Ezs7jW3uDpUiNpdiO
2seFx9O3vQlKGDhNy4RR
=16ln
-----END PGP SIGNATURE-----

coderman

unread,
Jun 25, 2015, 8:30:16 PM6/25/15
to Zrubi, qubes...@googlegroups.com
as much a fan of gentoo as i may be,
a more appropriate option might be Guix, GuixSD:
https://www.gnu.org/software/guix/about/

any of the reproducible builds / hardened distribution options will be
significant work. it's hard to provide solid recommendations when the
technologies are young and volatile...

best regards,

coderman

unread,
Jun 25, 2015, 8:36:35 PM6/25/15
to Jason M, qubes...@googlegroups.com, joa...@invisiblethingslab.com, marm...@invisiblethingslab.com
On 5/20/15, Jason M <nrg...@gmail.com> wrote:
> ...
> Provide an ISO that builds another ISO? LOL

back in 2007 i structured the Tor VM builds this way. all of the code
and Linux kernel part were built, and then prepared into an autorun
ISO image that was entirely self contained.

you boot a disconnected Windows host, mount the ISO, and at the end of
some hours you had windows packages.

never any network access. never any need to pull remote repos from
Windows. greatly reduced my concerns around Windows builds, since i
would always simply revert to good snapshot, and the VM itself was
never exposed to other networks.

i would post the sha256 of the build ISO, and while others might get
slightly different packages (see: reproducible builds) it was easy to
verify that the changes were due to artifacts of build, rather than
surreptitiously modified code.


but again, the future is reproducible builds. anything less is a stop gap...


best regards,

Manuel Amador (Rudd-O)

unread,
Jun 29, 2015, 4:33:45 AM6/29/15
to qubes...@googlegroups.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/24/2015 05:50 PM, Marek Marczykowski wrote:
> Generally I'm not convinced that implementing this temporary solution on
> top of current Qubes OS (for example Fedora-based dom0), is
> significantly easier than implementing proper deterministic builds for
> the few packages that we have

Have you considered implementing the builds as Nix / NixOS derivations?
They have already worked on the problem of deterministic builds and it's
fully automated these days. Maybe the full Nix system cannot be used,
but they certainly have algorithms and parts that can be reused to
produce deterministic builds. Plain RPM build does not produce
deterministic builds because of the dates/times issue -- both the host
and the date/time of the build will alter the signature value of the
RPM, which alters the hash value of the result -- as well as a ton of
other issues.

- --
Rudd-O
http://rudd-o.com/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJVkQLiAAoJEFmZwbV7vYQ2KIAP/3faDbbR14TKFxCMONhZJRLn
FoiTQxEshAGi366RMBTMRM/BfcGtpGBmzctYYHQOXzgs2zC4k0CcgbETbY57pTTN
hZl0EQJd26cMKTzHKG/nKX24X4T+p+matvEgOArh4f++/hplaukHUtHvn0U6/qgJ
Dn8fwor75jzJ/C08iNdobFt5I39gxOa12xTzIJ9MJG/pP09fhduu1CdtCJKXR48S
YcZCj6X/g2PiPbbkWJAu+mbNAAM9Ovn2SgdWuFsYjvt10LFLifZi2DVBJyMAN1Xl
C2k64YurntxUsWoJwrrJS+cjicFfaxjk26lFl/vj5imBfJdIJ8MOlwrIE4XgLru+
gwmT03HRLn05ZirW+P6finr+9w02TNDsCjtcU4tc3LqeSth6TibugtNuZTnenLtI
1ELxriHplwotv6Qo6UKEGhPvqViECOQA5RIg8kbJ02p5ljqUWY1/bzVq8oqoVJDG
35lAjeAiCBcA1JZ3YNUjtYk/aJVE3crZbC3OEBacDVK+0/MTOVfdgKkTkvTeXPVX
/Kd6UF1as/x6bdoHTGnHaI+3v8pt8uGnpFlFuwudVnG0zVkQ54dRebcruSd6t5f3
S2c+69UgLgP8n2O1pI0zM4bk2+m3Qmq4dSz7vxSul80Q0rpx75sZQAZxQTwMI2qT
gdusN7iNYsqDbocDNic7
=DQSm
-----END PGP SIGNATURE-----

Marek Marczykowski-Górecki

unread,
Jun 29, 2015, 5:45:36 AM6/29/15
to Manuel Amador (Rudd-O), qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Jun 29, 2015 at 01:33:39AM -0700, Manuel Amador (Rudd-O) wrote:
>
> On 06/24/2015 05:50 PM, Marek Marczykowski wrote:
> > Generally I'm not convinced that implementing this temporary solution on
> > top of current Qubes OS (for example Fedora-based dom0), is
> > significantly easier than implementing proper deterministic builds for
> > the few packages that we have
>
> Have you considered implementing the builds as Nix / NixOS derivations?
> They have already worked on the problem of deterministic builds and it's
> fully automated these days. Maybe the full Nix system cannot be used,
> but they certainly have algorithms and parts that can be reused to
> produce deterministic builds. Plain RPM build does not produce
> deterministic builds because of the dates/times issue -- both the host
> and the date/time of the build will alter the signature value of the
> RPM, which alters the hash value of the result -- as well as a ton of
> other issues.

Actually the timestamps are not a big problem - faketime solves most of
them. But that's only tip of the iceberg...
I'll look into NixOS how they solved that problem. Also Debian has done
some work on deterministic builds.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVkRO5AAoJENuP0xzK19csOUAH/Ru9vzYo/P+KICAAoi4y+uoY
Ym9g6nYf7kmpmy4mUGWyOrq87pSYC1M6ls3EeL7SpVkgByEjIcFjLXCmqwkA+v2O
/hU2VcPWMQSKidR3Kl3rMFT13pGDV3b2jK9czG7dK+XPlABDX+Wa9P5fEgH1ABJu
A7ubxSEQeKQTfAzHKMF+YQ0QqTUki6LIhSg+KV6yZ01oOmWQP8AgRiayfVcNw9hx
hPJcROXCpQVGv8LDviGQyhOY80VSDfbqTxWan8lOZZnhfYXgo2bSK8XNzRl2oeFA
1GNQps5bFgRv5tnrqen5u7MwGyK3D9um5FWMS8WwHF24nM1eyAkE9zGWAS1ccA4=
=VXwb
-----END PGP SIGNATURE-----

coderman

unread,
Jun 29, 2015, 11:31:43 PM6/29/15
to Marek Marczykowski-Górecki, Manuel Amador (Rudd-O), qubes...@googlegroups.com
On 6/29/15, Marek Marczykowski-Górecki <marm...@invisiblethingslab.com> wrote:
> ...
> I'll look into NixOS how they solved that problem. Also Debian has done
> some work on deterministic builds.

Debian deterministic builds also got some recent sponsorship, which
may improve their approach significantly over the summer.

end of summer to a better view of Guix, Nix, Debian reproducible builds status.

maybe this question deferred too long already... choose wisely! :)



best regards,

Manuel Amador (Rudd-O)

unread,
Oct 10, 2015, 9:52:50 PM10/10/15
to qubes...@googlegroups.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/29/2015 09:45 AM, Marek Marczykowski-Górecki wrote:
> Actually the timestamps are not a big problem - faketime solves most of > them. But that's only tip of the iceberg... > I'll look into NixOS
how they solved that problem.

Their build scripts eliminate many sources of non-determinism, and do
tons of work to make the builds hermetic. Hermeticism is what you want
if you want reproducibility and auditability. They have very
interesting mechanisms to enforce hermeticism and eliminate sources of
indeterminism, down to a special "tar" like command that they use to
convey files from place to place.

For this to work within Qubes, you'll have to port all of Nix's
improvements to the rpm-build environment used to build Qubes proper.

- --
Rudd-O
http://rudd-o.com/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJWGcDqAAoJEFmZwbV7vYQ2D3oP/RYc4ACdbgv55inUHCdqGBR5
tZLtwCCYGmI5rmHiB6FqerP1bOD/40DaaGnBolEgPwbfqiklCD+CigT2MOahxXhd
IOt9QBn6zu+OgKNKOhP0oxjTubs07vEu3jqvNM2QyaWOxm+ctWs+uajmoE0dp9Ms
d7/a8KGowfZ+s3sesQisqZNR1g+RPfOuJCTkVORVdUk861iqdj52Hlx+y8GvJWYL
bTffYH/9T3d9SMsicvdKO5TYlQahz09j+obAHWa0VhXUU5e7ic8b2RIslLXjP4p6
zmi45izFolDPwN1o3jL39Jh2ab7phDvBwFxtCExtBN5xiZKjP/pV0ex25DhOxPbB
8QfT6n+G+7/bKGnqSDPmeS31dSvd8bszssA94N4J4VfceO4mf3YIm2olE4JvbG7D
ZGPOwsg263Ct3r+dfwpx0gPUk7TJv8lob5zBOy+Mu9NooQSGsc14o9bbIBRd+J63
ibi/b08bqk6h5r1hVZTZn/iIenGxNOUMm/PDieht8Kr8IdBd8AFoSPJkqX2CSCUU
Q7rsMc5woIsJliomEL37YJ1yIcvtVldnsqmBmaWOTRSdrN+jtItAaYnwf0Y7Ad0M
MBQ2MKA9zfr4vn67FXrcomKvLlIIKBJtX1i177AYDAiSgOvJ9fpisPQyw+25yg/X
Anr3+GrOcNyahKiAvfJ+
=UKf3
-----END PGP SIGNATURE-----

Paul Harvey

unread,
Oct 11, 2015, 1:08:01 AM10/11/15
to Manuel Amador (Rudd-O), qubes...@googlegroups.com
I've been watching the Debian reproducible builds effort with great
interest. I have a project which tries to "measure" whether the
content of btrfs snapshots are unchanged over time and whenever
sent/received from host to host by "measuring" a sha512sum/PGP
signature of a reproducible tarball of the snapshot directory. Between
tar's file ordering issues and working around (still unresolved) btrfs
atime bugs, that was surprisingly harder than it sounds! But I found
some good advice on the Debian wiki.

If you scroll down to "specific issues", there's an impressive list of
the sorts of probems encountered that I wouldn't have thought of
without going through the motions myself (Eg. variability in
python/java/etc pre-complied bytecode, automatically generated
documentation files, ...):

https://wiki.debian.org/ReproducibleBuilds/Howto

They seem to have ~17,000 out of ~20,000 packages or so building
reproducibly (~85% IIRC).

At first glance my (completely unqualified/naive) impression is that
if Fedora's own efforts have been left to rot ( no updates on the
stuff linked from
https://securityblog.redhat.com/2013/09/18/reproducible-builds-for-fedora/
since 2013) it seems the Qubes team might have other, higher-impact
priorities to work on rather than shoe-horning reproducible builds
into a distro ecosystem that doesn't seem to be making much effort
itself (but perhaps I'm mistaken)...

Cheers
> --
> You received this message because you are subscribed to the Google Groups "qubes-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel...@googlegroups.com.
> To post to this group, send email to qubes...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/5619C0EB.3010806%40rudd-o.com.
Reply all
Reply to author
Forward
0 new messages