qubes-builder gpg verification security, check for rollback (downgrade) or indefinite freeze attacks

[Patrick Schleizer]

Hi!

Does qubes-builder check for rollback (downgrade) or indefinite freeze
attacks [1]?

Threat model:
- a user who builds from source code
- building user successfully verified Qubes' source code
- user doesn't manually ensure after build, that version numbers match,
doesn't read the build log [unless it stops and shows errors], and
relies that the verification chain is intact
- git hosting compromised [2]
- eventually targeting specific builders

function:
verify_tag

link:
https://github.com/QubesOS/qubes-builder/blob/7d21e6b7b0a5ab3a68e8acdbc3f540f2221b47c0/scripts/verify-git-tag#L38

code:
gpg --verify --status-fd=1 $temp_name/content.asc 2>/dev/null|grep -q
'^\[GNUPG:\] TRUST_\(FULLY\|ULTIMATE\)$'

It does not check freshness? So any older tag/signature would be
accepted, a rollback attack would succeed?

I am very much into file verification, gpg, wrote gpg-bash-lib [6] where
I'd appreciate feedback and sometimes report gpg usage security issues
in other projects. [non-exhaustive list [7]]

Having said that, do you have any other gpg verification code in other
files that I could look into?

Cheers,
Patrick

[1] "rollback (downgrade) or indefinite freeze attack"
Defined as per TUF: Attacks and Weaknesses:
- https://github.com/theupdateframework/tuf/blob/develop/SECURITY.md
- http://www.webcitation.org/6F7Io2ncN
[2]
* In case github gets hacked [3] again.
* Or in cases similar to:
* SSL CA's such as DigiNotar was hacked or [4]
* comodo resellers that got hacked. [5]
[3]
http://www.extremetech.com/computing/120981-github-hacked-millions-of-projects-at-risk-of-being-modified-or-deleted
[4] https://en.wikipedia.org/wiki/DigiNotar
[5]
http://www.scmagazine.com/two-more-comodo-resellers-owned-in-ssl-hack/article/199620/
[6] https://github.com/Whonix/gpg-bash-lib
[7] https://phabricator.whonix.org/T245

[Jason M]

I recently implemented some gpg verification of files, not github, in python.  Only added import key and verify.
https://github.com/nrgaway/qubes-app-salt-config/blob/develop-wip/srv/qubes-salt/_modules/gpg.py
https://github.com/nrgaway/qubes-app-salt-config/blob/develop-wip/srv/qubes-salt/_states/gpg.py
https://github.com/nrgaway/qubes-app-salt-config/blob/develop-wip/srv/qubes-salt/_renderers/verify.py

I just finished converting a setup script from bash to python and there is a small routine that verifies fingerprint matches one from keyserver.
https://github.com/nrgaway/qubes-builder/blob/nrgaway/setup#L927
https://github.com/nrgaway/qubes-builder/blob/nrgaway/.setup.data

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, Apr 02, 2015 at 12:11:03PM +0000, Patrick Schleizer wrote:
> Hi!
>
> Does qubes-builder check for rollback (downgrade) or indefinite freeze
> attacks [1]?
>
> Threat model:
> - a user who builds from source code
> - building user successfully verified Qubes' source code
> - user doesn't manually ensure after build, that version numbers match,
> doesn't read the build log [unless it stops and shows errors], and
> relies that the verification chain is intact
> - git hosting compromised [2]
> - eventually targeting specific builders
>
> function:
> verify_tag
>
> link:
> https://github.com/QubesOS/qubes-builder/blob/7d21e6b7b0a5ab3a68e8acdbc3f540f2221b47c0/scripts/verify-git-tag#L38
>
> code:
> gpg --verify --status-fd=1 $temp_name/content.asc 2>/dev/null|grep -q
> '^\[GNUPG:\] TRUST_\(FULLY\|ULTIMATE\)$'
>
> It does not check freshness? So any older tag/signature would be
> accepted, a rollback attack would succeed?

Yes, older tag/signature would be accepted in this function. But when
verified, it is *merged* into local sources, not *reseted*. So if you have
newer sources (in terms of git history) already downloaded it will not
downgrade it. Personally I use combo make prepare-merge + do-merge,
which additionally show me color coded info if the merge is
fast-forward or not (in addition to all the new commits to be merged).
Perhaps we should even introduce an option to fail on non-fast-forward
updates.

It doesn't protect you in any way against such type of attacks during
first time source download. But you can easily list version tags
(including release tags) - make show-vtags - and check if the versions
are what you've expected.

> I am very much into file verification, gpg, wrote gpg-bash-lib [6] where
> I'd appreciate feedback and sometimes report gpg usage security issues
> in other projects. [non-exhaustive list [7]]

In short we use git history to mitigate downgrades, but it can't be
easily used for file verification. Maybe you can setup git repo with a
file hash, then use signed tags there. But it's an overkill IMHO.

> Having said that, do you have any other gpg verification code in other
> files that I could look into?

Some components downloads additional sources - like kernel, KDE, etc. There
is also code to verify that downloads - just a simple gpg -v with
isolated keyring (one key in most cases). That keyring isolation is
provided by qubes-builder (scripts/get-sources). When upstream do not
provide signatures, we have file hashes committed into the git repo.
This code is in Makefiles in those repositories:
- linux-kernel
- desktop-linux-kde
- desktop-linux-xfce
- vmm-xen
- core-libvirt

> Cheers,
> Patrick
>
> [1] "rollback (downgrade) or indefinite freeze attack"
> Defined as per TUF: Attacks and Weaknesses:
> - https://github.com/theupdateframework/tuf/blob/develop/SECURITY.md
> - http://www.webcitation.org/6F7Io2ncN
> [2]
> * In case github gets hacked [3] again.
> * Or in cases similar to:
> * SSL CA's such as DigiNotar was hacked or [4]
> * comodo resellers that got hacked. [5]
> [3]
> http://www.extremetech.com/computing/120981-github-hacked-millions-of-projects-at-risk-of-being-modified-or-deleted
> [4] https://en.wikipedia.org/wiki/DigiNotar
> [5]
> http://www.scmagazine.com/two-more-comodo-resellers-owned-in-ssl-hack/article/199620/
> [6] https://github.com/Whonix/gpg-bash-lib
> [7] https://phabricator.whonix.org/T245
>

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVHcx5AAoJENuP0xzK19csDrwH/i5RxukrCuVqmxSe6x17Rpj9
LSiN5u+lsTM/7kIyBqx/86uXn4Lil7bUHklb6awssUrCDhl80ZYFBevdlDnPpemt
XpeHQaZHfeqmMq7CrO4yhshO2gjjR7EH55hHEyumpSx6V0oY40/MTGNPydvTwBm1
5GoFo3lKcHFA2kopNL35VQ9nZxa98ZyEp/zim6dzg6kqs/9E9zn5EuXa4y+hXwpA
MeEIVLI79ex5q09Rtk8wE+Dg0GqjcqUzUOy3oOFNg/OcMzvHyVjet0CR9M0/IaW8
v98sTBU45OagkoeNlIrdBV9lHM9HQEtO4LHEgYu23QmQvlA7z6+L3d/Qw4fubcA=
=6JYY
-----END PGP SIGNATURE-----

[Patrick Schleizer]

Marek Marczykowski-Górecki:

> On Thu, Apr 02, 2015 at 12:11:03PM +0000, Patrick Schleizer wrote:
>> function:
>> verify_tag
>
>> link:
>> https://github.com/QubesOS/qubes-builder/blob/7d21e6b7b0a5ab3a68e8acdbc3f540f2221b47c0/scripts/verify-git-tag#L38
>
>> code:
>> gpg --verify --status-fd=1 $temp_name/content.asc 2>/dev/null|grep -q
>> '^\[GNUPG:\] TRUST_\(FULLY\|ULTIMATE\)$'
>
>> It does not check freshness? So any older tag/signature would be
>> accepted, a rollback attack would succeed?
>
> Yes, older tag/signature would be accepted in this function. But when
> verified, it is *merged* into local sources, not *reseted*. So if you have
> newer sources (in terms of git history) already downloaded it will not
> downgrade it. Personally I use combo make prepare-merge + do-merge,
> which additionally show me color coded info if the merge is
> fast-forward or not (in addition to all the new commits to be merged).
> Perhaps we should even introduce an option to fail on non-fast-forward
> updates.
>
> It doesn't protect you in any way against such type of attacks during
> first time source download. But you can easily list version tags
> (including release tags) - make show-vtags - and check if the versions
> are what you've expected.

Git merge rather than reset is useful indeed. Prevents people who
already got the sources from being targetd for rollback (downgrade)
attacks. Wouldn't prevent indefinite freeze attacks, though.

Also users who get the source for the very first time are more at risk?
At that point, nothing protects them from rollback (downgrade) attacks?

That's my main target audience here. People who prefer to build from
source code to make sure they don't get backdoored by eventual backdoors
in the binary builds injected by the build machine during build. [Not
claiming any backdoor by the Qubes team. But build machines seem
worthwhile targets to me for powerful adversaries.] Building from source
code is a cheap way to prevent those. But if they buy that advantage of
cost of eventual targeted rollback (downgrade) attacks, that's non-ideal.

>> Having said that, do you have any other gpg verification code in other
>> files that I could look into?
>
> Some components downloads additional sources - like kernel, KDE, etc. There
> is also code to verify that downloads - just a simple gpg -v with
> isolated keyring (one key in most cases). That keyring isolation is
> provided by qubes-builder (scripts/get-sources). When upstream do not
> provide signatures, we have file hashes committed into the git repo.
> This code is in Makefiles in those repositories:
> - linux-kernel
> - desktop-linux-kde
> - desktop-linux-xfce
> - vmm-xen
> - core-libvirt

I see. Checked linux-kernel. Code:

@xzcat $(SRC_FILE) | gpg -q --verify $(SIGN_FILE) - 2>/dev/null

I don't think this is safe.

Quote Werner Koch (gnupg lead developer):
http://lists.gnupg.org/pipermail/gnupg-devel/2005-December/022559.html

"there is no clear distinction between the codes and for proper error
reporting you are advised to use the --status-fd messages."

Cheers,
Patrick

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

AFAIK preventing this problem would require refreshing some signature
periodically, right? This is also not ideal because of manual work
required for that. But maybe worth the effort? Maybe it is enough to add
another section in our warrant canaries (list of most recent tags on
source code)? I guess 3-month isn't frequent enough...

> >> Having said that, do you have any other gpg verification code in other
> >> files that I could look into?
> >
> > Some components downloads additional sources - like kernel, KDE, etc. There
> > is also code to verify that downloads - just a simple gpg -v with
> > isolated keyring (one key in most cases). That keyring isolation is
> > provided by qubes-builder (scripts/get-sources). When upstream do not
> > provide signatures, we have file hashes committed into the git repo.
> > This code is in Makefiles in those repositories:
> > - linux-kernel
> > - desktop-linux-kde
> > - desktop-linux-xfce
> > - vmm-xen
> > - core-libvirt
>
> I see. Checked linux-kernel. Code:
>
> @xzcat $(SRC_FILE) | gpg -q --verify $(SIGN_FILE) - 2>/dev/null
>
> I don't think this is safe.
>
> Quote Werner Koch (gnupg lead developer):
> http://lists.gnupg.org/pipermail/gnupg-devel/2005-December/022559.html
>
> "there is no clear distinction between the codes and for proper error
> reporting you are advised to use the --status-fd messages."

AFAIR this applies mostly to verifying key validity/trust. In our case
it shouldn't be a problem because of dedicated keyring with just the
key(s) to verify given component. But indeed it would be good idea to
switch to either --status-fd, or gpgv (or both).

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVHykaAAoJENuP0xzK19cs7lUH/2SBrkBdT4L3YoIwDpZOBfe7
cOENGYsb/U0rTK/cZil9vANdkwYbAY+mZSLJ8F3caUrsH8iWiak0JT4e7vKjsoYu
bxxJX45Z1OzFhLpV5Q3xVglw4QFWlhYpvWVbmN5CkeGOKRLxATo7/njNUVp2T23c
3hkxkdEj9e0w/GavIdGwhQ361eTDfrBrTRrZB6GhfXZIAjBZye1WQDyFuJM8cC8J
Gm9IJOGcLP1MrCjTG5phEB/9u/n6rn+6UClameq2IkzZ4JyPHFG0s7azU8w1Ayaa
Q6AmxWL+OmCfvIrgib7qb6udtPafMQpThbaF4Ycwn6wZLNDx4LODW0gnEfms4QA=
=FFf6
-----END PGP SIGNATURE-----

[Patrick Schleizer]

Marek Marczykowski-Górecki:

Similar to the Valid-Until field in Debian apt release files [and other
package managers that use similar fields]:
http://blog.ganneff.de/blog/2008/09/23/valid-until-field-in-release-f.html

Well, that would be ideal, but that would also be something that the one
who provides the file and signature would have to do, i.e. upstream. So
if you wanted to do this, you could only do this for your own signatures
and could only try to convince upstream of doing it as well. Worthwhile,
sure, but unlikely to happen for every upstream.

> This is also not ideal because of manual work
> required for that.

Yes.

> But maybe worth the effort?

Definitively.

> Maybe it is enough to add
> another section in our warrant canaries (list of most recent tags on
> source code)?

Dunno. Any place works as long as it's checked during build.

> I guess 3-month isn't frequent enough...

Yes.

>>>> Having said that, do you have any other gpg verification code in other
>>>> files that I could look into?
>>>
>>> Some components downloads additional sources - like kernel, KDE, etc. There
>>> is also code to verify that downloads - just a simple gpg -v with
>>> isolated keyring (one key in most cases). That keyring isolation is
>>> provided by qubes-builder (scripts/get-sources). When upstream do not
>>> provide signatures, we have file hashes committed into the git repo.
>>> This code is in Makefiles in those repositories:
>>> - linux-kernel
>>> - desktop-linux-kde
>>> - desktop-linux-xfce
>>> - vmm-xen
>>> - core-libvirt
>
>> I see. Checked linux-kernel. Code:
>
>> @xzcat $(SRC_FILE) | gpg -q --verify $(SIGN_FILE) - 2>/dev/null
>
>> I don't think this is safe.
>
>> Quote Werner Koch (gnupg lead developer):
>> http://lists.gnupg.org/pipermail/gnupg-devel/2005-December/022559.html
>
>> "there is no clear distinction between the codes and for proper error
>> reporting you are advised to use the --status-fd messages."
>
> AFAIR this applies mostly to verifying key validity/trust. In our case
> it shouldn't be a problem because of dedicated keyring with just the
> key(s) to verify given component.

Not sure. I specifically asked about this on gnupg-users mailing list.
'Are there cases where gpg --verify will exit 0, even if verification
failed?'
https://lists.gnupg.org/pipermail/gnupg-users/2015-January/052232.html

While I was not told any specific example as a great deterrence, my
conclusion is, that gpg exit codes are totally not to be relied on.

> But indeed it would be good idea to
> switch to either --status-fd, or gpgv (or both).

Yes.

Other options to consider (just consider, not arguing they are great,
better, whatsoever):
- gpg2
- gppv2

Another subtlety to consider, checksec.sh on Debian wheezy reports, that
gpg[v] has Partial RELRO and NO PIE vs gpg[v]2 that has FULL RELRO and PIE.

***

A related point I would like to raise is another common misconception or
unpopular fact about OpenPGP signatures. You might be aware of it, but I
would invite you to specifically think it through wtr to the Qubes file
[not git] verification code.

OpenPGP signatures do sign files. Not file names.

Names of files are not part of OpenPGP signatures. When file and
signature is renamed, the OpenPGP signature remains valid. So version
information taken from names of files is untrustworthy.

In other words, when using "gpg --armor --detach-sign
some-file-version-c" a file: some-file-version-c.asc will be created.

But an adversary position to arbitrarily change file names on a mirror
[for targeted attacks] or so could rename it to some-file-version-d and
some-file-version-d.asc. Or in other words, the adversary could have
changed the name of the files to some-file-version-d[.asc], while in
reality it was just some-file-version-c[.asc].

See also:
https://lists.gnupg.org/pipermail/gnupg-users/2015-January/052185.html

One could "use OpenPGP notations to include the legitimate names of
files to prevent file name tampering" as I suggested here in a slightly
different context:
https://trac.torproject.org/projects/tor/ticket/14187

Those OpenPGP notations are also included in status-fd.

But that's something the provider of the OpenPGP signature, i.e.
upstream would have to do.

Since convincing upstream to do this is time consuming, takes a while
until they implement it, and may not always work, I think the best that
could be done until then or as a stopgap, would be extracting the
signature creation date of the signature from status-fd and comparing it
with the expected value. (Signature creation date is part of the OpenPGP
signature and tamper resistant. [1])

Cheers,
Patrick

[1] https://lists.gnupg.org/pipermail/gnupg-users/2013-March/046218.html

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ok, I'll work on this, at least for our repos. But that would require
some rework of repos management scripts, so will take a while.

This isn't a problem for source tarballs, as the version is stored
inside archive (as a directory name). So if someone would provide
linux-3.12.11.tar.xz as linux-3.12.39.tar.xz, the build will fail
because there would be no linux-3.12.39 directory.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVJA0uAAoJENuP0xzK19cs4OAH/1HJEMts6G6vtG5dtlXKr9NM
86PmfBU10aIt01Oas9VoBsXfggu3D3XDlc+nEp04fAItnST7EdDdHWdh5t7zhzmv
kR0B7/8sbNTJpALD+U+itfttd7hH45IufG55s+GxbDP3cgPCcP3n0MjkVuyw4qHX
/230UfeF214J2anbuYVyckhTsChESOcYL7eFEvevPxwtYdnPcp0o6OJ6TbFW52Se
ieptjROQuEONKVTl5eorN9NqMgPNVZQQxtgqtBLcOSyGMEhZyXBY4MMaTQVbALD7
ktXNd3BNbQbW6RNgUG7OsKG3oLKWXqwWqxOhj5DAWNiuYYBd0waL6fsFDc2xGVw=
=JN75
-----END PGP SIGNATURE-----