Bugs & Issues I noticed during & after upgrading from R2 to R3 and fedora 20 to debian 8

[David Hobach]

Dear developers,

all in all I'm quite surprised at how well everything went, thanks for
that!

I'm also very happy with R3 & the debian template; however I wanted to
provide feedback about the following points I noticed:

1) Cloning an AppVM does not appear to clone all settings; in particular
the firewall rules are set to "allow all" for the clone by default. I'm
not sure whether this one already existed in R2.

2) Incorrect firewall settings (e.g. incorrect host name) will result in
a silent "deny all" rule. I think that's ok from a security perspective
and probably done by iptables and not by Qubes, but it can take users
quite some time to figure it out (in my case a host name was missing in
the /etc/hosts due to the migration). An error message would be nice.
Btw: That one already existed in R2; I just noticed it again the hard
way during the migration.

3) The R2 -> R3 in-place upgrade screwed my disposable template VM. Even
after updating it, it wouldn't start. I had to re-create it. This wasn't
mentioned in the wiki so it was unexpected for me.

4) dom0 updates through a debian 8 netvm are not possible. I noticed
that there's a related bug open, but it might make sense to mention it
in the wiki, if this is not going to be fixed soon.

5) The Qubes VM Manager hangs from time to time; in R2 it was much more
responsive. For example when stopping all my VMs incl. the netvm the app
can hang during VM shutdown or - when I start them again - during
startup. For qvm-shutdown --all --wait I can see all VMs going to
yellow, then it hangs and after a few seconds all of them are closed at
the same time. In R2 one after another VM did shut down and the whole
process was visible in the VM manager.

Sorry for only talking about issues...

Kind Regards
David

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, Jun 21, 2015 at 07:03:28PM +0200, David Hobach wrote:
> Dear developers,
>
> all in all I'm quite surprised at how well everything went, thanks for that!
>
> I'm also very happy with R3 & the debian template; however I wanted to
> provide feedback about the following points I noticed:
>
> 1) Cloning an AppVM does not appear to clone all settings; in particular the
> firewall rules are set to "allow all" for the clone by default. I'm not sure
> whether this one already existed in R2.

Created ticket here;
https://github.com/QubesOS/qubes-issues/issues/1032

> 2) Incorrect firewall settings (e.g. incorrect host name) will result in a
> silent "deny all" rule.

There should be tray notification for that... Can you check 'sudo
systemctl status qubes-firewall' in sys-firewall/firewallvm? If there
was a problem with displaying the notification, it should give an idea
why.

> I think that's ok from a security perspective and
> probably done by iptables and not by Qubes, but it can take users quite some
> time to figure it out (in my case a host name was missing in the /etc/hosts
> due to the migration). An error message would be nice.
> Btw: That one already existed in R2; I just noticed it again the hard way
> during the migration.
>
> 3) The R2 -> R3 in-place upgrade screwed my disposable template VM. Even
> after updating it, it wouldn't start. I had to re-create it. This wasn't
> mentioned in the wiki so it was unexpected for me.

What do you mean by re-creating? Just qvm-create-default-dvm call, or
removing the whole *-dvm VM?

> 4) dom0 updates through a debian 8 netvm are not possible. I noticed that
> there's a related bug open, but it might make sense to mention it in the
> wiki, if this is not going to be fixed soon.

Actually the fix is already committed, just needs some more testing.

> 5) The Qubes VM Manager hangs from time to time; in R2 it was much more
> responsive. For example when stopping all my VMs incl. the netvm the app can
> hang during VM shutdown or - when I start them again - during startup. For
> qvm-shutdown --all --wait I can see all VMs going to yellow, then it hangs
> and after a few seconds all of them are closed at the same time. In R2 one
> after another VM did shut down and the whole process was visible in the VM
> manager.

This is libvirt issue - every operation on a VM (including VM
startup/shutdown) blocks other operations - also just getting VM status.
I think it is already improved in new major libvirt version, but we'll
upgrade it for R3.1, not R3.0 (because of too many changes to make a
switch at this point). So this issue unfortunately will be present in
R3.0.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVhzREAAoJENuP0xzK19csrYQIAIheHYgalYHIuerbPnpz2J6s
4+oudAwU3K5N+gLOjhFq6p7BD0zhX6Iewi0wQcbDfADfHLnES+ctjJe/Pjms4t2b
r/H30VMIkO4r2Q/ACy7SJ3gHJ6iz0/SjHV2dYOA3L+6wE4v1K+45y7H1YTKaBqdZ
LBFPGeO/zA9qo+BV60AuLuW4WyUXXAoBvypzRgV5MmAqVbgTheUBd9sZ9ZDQ03AR
+lSSMkLVBF3NlGkwN4WpotvH+wtHE1fB8C6ollGNn4nIL/2pHSK0n0gEXp/BLix0
G4PEig4Sn943Wtpo8+iY/hAMepmigc6UleUvrNep3nu/OCwJV2zu8NcYrkx3GcI=
=MXrI
-----END PGP SIGNATURE-----

[David Hobach]

Hi Marek,

thanks for your answer! More below:

On 06/22/2015 12:01 AM, Marek Marczykowski-Górecki wrote:
> On Sun, Jun 21, 2015 at 07:03:28PM +0200, David Hobach wrote:
>> Dear developers,
>>
>> all in all I'm quite surprised at how well everything went, thanks for that!
>>
>> I'm also very happy with R3 & the debian template; however I wanted to
>> provide feedback about the following points I noticed:
>>
>> 1) Cloning an AppVM does not appear to clone all settings; in particular the
>> firewall rules are set to "allow all" for the clone by default. I'm not sure
>> whether this one already existed in R2.
>
> Created ticket here;
> https://github.com/QubesOS/qubes-issues/issues/1032
>
>> 2) Incorrect firewall settings (e.g. incorrect host name) will result in a
>> silent "deny all" rule.
>
> There should be tray notification for that... Can you check 'sudo
> systemctl status qubes-firewall' in sys-firewall/firewallvm? If there
> was a problem with displaying the notification, it should give an idea
> why.

Output is:
sudo systemctl status qubes-firewall
● qubes-firewall.service - Qubes firewall updater
Loaded: loaded (/lib/systemd/system/qubes-firewall.service; enabled)
Active: active (running) since Mon 2015-06-22 10:12:47 CEST; 5min ago
Main PID: 911 (qubes-firewall)
CGroup: /system.slice/qubes-firewall.service
├─ 911 /bin/sh /usr/sbin/qubes-firewall
└─1401 /usr/bin/qubesdb-watch /qubes-iptables

Jun 22 10:12:47 firewallvm systemd[1]: Starting Qubes firewall updater...
Jun 22 10:12:47 firewallvm systemd[1]: Started Qubes firewall updater.
Jun 22 10:12:49 firewallvm qubes-firewall[911]: /qubes-iptables
Jun 22 10:13:09 firewallvm qubes-firewall[911]: /qubes-iptables
Jun 22 10:13:33 firewallvm qubes-firewall[911]: /qubes-iptables

I also see other notifications (e.g. for dispvm being updated, VM
getting started), but I do have rather extreme settings wrt to window
focus (but it should still be in the background then, shouldn't it?).

>> I think that's ok from a security perspective and
>> probably done by iptables and not by Qubes, but it can take users quite some
>> time to figure it out (in my case a host name was missing in the /etc/hosts
>> due to the migration). An error message would be nice.
>> Btw: That one already existed in R2; I just noticed it again the hard way
>> during the migration.
>>
>> 3) The R2 -> R3 in-place upgrade screwed my disposable template VM. Even
>> after updating it, it wouldn't start. I had to re-create it. This wasn't
>> mentioned in the wiki so it was unexpected for me.
>
> What do you mean by re-creating? Just qvm-create-default-dvm call, or
> removing the whole *-dvm VM?

I had to remove the dvm template completely (btw it didn't work 100%
with the Qubes VM manager right click remove, I had to remove the
template files in dom0 at /var/lib/qubes/appvms) and then use
qvm-create-default-dvm to create a new one.

I guess it's because I ran qvm-create-default-dvm as root in the past,
but I also don't quite understand why qvm-create-default-dvm complains
about being run as root after it has run instead of before. It's nice to
know about potential problems, but it's usually better to avoid them in
the first place.

>> 4) dom0 updates through a debian 8 netvm are not possible. I noticed that
>> there's a related bug open, but it might make sense to mention it in the
>> wiki, if this is not going to be fixed soon.
>
> Actually the fix is already committed, just needs some more testing.

Awesome!

>> 5) The Qubes VM Manager hangs from time to time; in R2 it was much more
>> responsive. For example when stopping all my VMs incl. the netvm the app can
>> hang during VM shutdown or - when I start them again - during startup. For
>> qvm-shutdown --all --wait I can see all VMs going to yellow, then it hangs
>> and after a few seconds all of them are closed at the same time. In R2 one
>> after another VM did shut down and the whole process was visible in the VM
>> manager.
>
> This is libvirt issue - every operation on a VM (including VM
> startup/shutdown) blocks other operations - also just getting VM status.
> I think it is already improved in new major libvirt version, but we'll
> upgrade it for R3.1, not R3.0 (because of too many changes to make a
> switch at this point). So this issue unfortunately will be present in
> R3.0.

No problem; for a plus it seems that shutting down VMs is more faster in
R3 than R2.

Ah yes I also noticed that my AEM didn't complain after the upgrade to
R3... It just worked as before. I guess I have a really crappy TPM
implementation or was already owned... :-(

Kind Regards
David

[cprise]

On 06/22/2015 04:45 AM, David Hobach wrote:
>
> Ah yes I also noticed that my AEM didn't complain after the upgrade to
> R3... It just worked as before. I guess I have a really crappy TPM
> implementation or was already owned... :-(
>
> Kind Regards
> David
>

That's pretty alarming. Are you sure the updated Xen and kernel versions
are being loaded at boot time? What do 'xl info' and 'uname -a' say?

Maybe you only upgraded your HD boot partition instead of your removable
AEM drive (assuming you have one), and so your AEM boot drive is still
using the R2 versions of Xen and kernel...

[David Hobach]

>>> 1) Cloning an AppVM does not appear to clone all settings; in
>>> particular the
>>> firewall rules are set to "allow all" for the clone by default. I'm
>>> not sure
>>> whether this one already existed in R2.
>>
>> Created ticket here;
>> https://github.com/QubesOS/qubes-issues/issues/1032

Just noticed one more similar issue: I created a dvm from within a VM
with limited firewall rules - all rules were set to allowed all for the
dvm instead of being cloned from the original VM. :-(
The netvm settings were taken over now though (not as in R2).

[David Hobach]

I usually use
echo "My secret passphrase..." | tpm_sealdata -o
/boot/antievilmaid/sealed_secret.blob --pcr 0 --pcr 1 --pcr 2 --pcr 3
--pcr 4 --pcr 5 --pcr 6 --pcr 7 --pcr 8 --pcr 9 --pcr 12 --pcr 14 --pcr
17 --pcr 18 --pcr 19
for sealing; maybe there's also some pcr missing there. I'm not an
expert at what pcr is for what.

Other than that you can see from the attached logs that I didn't have to
re-seal (see timestamps).

It might really be a TPM issue though; once I also did some changes in
the BIOS that were not detected by AEM.

[cprise]

OK, you last sealed the secret on May 12, but you did the upgrade in
June? And the immediate result of the upgrade is that your secret phrase
is still printed on the screen during boot?

Maybe trousers or the TPM is just returning a constant dummy value each
time you seal the secret. That would cause the secret to be decoded no
matter what.

You may want to run 'tpm_version' and use that info to search for
problems between your TPM and trousers. Also keep in mind that your
system may have more than one TPM.

[David Hobach]

Exactly.

> Maybe trousers or the TPM is just returning a constant dummy value each
> time you seal the secret. That would cause the secret to be decoded no
> matter what.

Might be a reason; but it sometimes doesn't decode. In particular when I
a) supply an incorrect passphrase.
b) make certain BIOS changes (some don't matter, some do - I didn't do
these kind of changes for quite some time anymore though; maybe I'll
re-try to cause the TPM to become alerted)

> You may want to run 'tpm_version' and use that info to search for
> problems between your TPM and trousers. Also keep in mind that your
> system may have more than one TPM.

It says:
TPM 1.2 Version Info:
Chip Version: 1.2.3.91
Spec Level: 2
Errata Revision: 3
TPM Vendor ID: WEC
TPM Version: 01010000
Manufacturer Info: 57454300

I'm also pretty sure I only have one TPM as I had to explicitly buy it
and build it in my PC as the mainboard only provided an empty slot (it's
a desktop PC != laptop). Btw it's this one:
https://www.alternate.de/ASRock/TPM-Modul/html/product/1127156

[cprise]

So the point of failure may be that its not measuring the TCB software
after it loads; or maybe trousers is not including the software-related
aspect due to some bug in the way it processes arguments or talks to the
TPM. (As you indicated, it does appear to take BIOS settings measurement
into account, as well as your SRK. But I think the software is somehow
being ignored.)

The example usage in the AEM guide shows only 3 PCRs: 17, 18 and 19.
IIRC these are amalgams of other, individual measurements and could be
considered good enough for most uses. Maybe if you re-sealed with only
those 3 it could avoid manifestation of what appears to be a bug (either
in trousers or the TPM itself).

>
>> You may want to run 'tpm_version' and use that info to search for
>> problems between your TPM and trousers. Also keep in mind that your
>> system may have more than one TPM.
>
> It says:
> TPM 1.2 Version Info:
> Chip Version: 1.2.3.91
> Spec Level: 2
> Errata Revision: 3
> TPM Vendor ID: WEC
> TPM Version: 01010000
> Manufacturer Info: 57454300
>
> I'm also pretty sure I only have one TPM as I had to explicitly buy it
> and build it in my PC as the mainboard only provided an empty slot (it's
> a desktop PC != laptop). Btw it's this one:
> https://www.alternate.de/ASRock/TPM-Modul/html/product/1127156

If you describe the issue to the trousers-users mailing list maybe
someone could offer suggestions on how to troubleshoot the problem:

http://sourceforge.net/p/trousers/mailman/trousers-users/

[David Hobach]

Hmmm... Actually I tried to produce a TPM complaint by a) enabling a
serial port on my motherboard in the BIOS and b) by changing my
supervisor password. Unfortunately both didn't result in the TPM
complaining about anything (the secret passphrase was displayed). I
recall that it didn't show 2 or 3 times after changes in the past though.
Do you possibly know a foolproof test that should definitely make the
secret passphrase not show up?

> The example usage in the AEM guide shows only 3 PCRs: 17, 18 and 19.
> IIRC these are amalgams of other, individual measurements and could be
> considered good enough for most uses. Maybe if you re-sealed with only
> those 3 it could avoid manifestation of what appears to be a bug (either
> in trousers or the TPM itself).

Will try that soon as well.

>>
>>> You may want to run 'tpm_version' and use that info to search for
>>> problems between your TPM and trousers. Also keep in mind that your
>>> system may have more than one TPM.
>>
>> It says:
>> TPM 1.2 Version Info:
>> Chip Version: 1.2.3.91
>> Spec Level: 2
>> Errata Revision: 3
>> TPM Vendor ID: WEC
>> TPM Version: 01010000
>> Manufacturer Info: 57454300
>>
>> I'm also pretty sure I only have one TPM as I had to explicitly buy it
>> and build it in my PC as the mainboard only provided an empty slot (it's
>> a desktop PC != laptop). Btw it's this one:
>> https://www.alternate.de/ASRock/TPM-Modul/html/product/1127156
>
> If you describe the issue to the trousers-users mailing list maybe
> someone could offer suggestions on how to troubleshoot the problem:
>
> http://sourceforge.net/p/trousers/mailman/trousers-users/

Good idea, I'll wait until I have time to do some more tests though.

[cprise]

I think its just the BIOS settings relating to the boot conditions that
are measured by the TPM. Probably you should assume this aspect is
already working correctly.

The test should definitely involve changing of either xen or the linux
kernel, because those were the big changes in moving to R3rc1. If R3rc1
has an alternate kernel version available you could install it which
will add a choice to the boot menu which is not sealed by the TPM.
Switching between the default sealed kernel and the alternate unsealed
kernel should make a good TPM test.

[David Hobach]

Now I also changed the boot order by adding some redundant device -
nothing changed.

> The test should definitely involve changing of either xen or the linux
> kernel, because those were the big changes in moving to R3rc1. If R3rc1
> has an alternate kernel version available you could install it which
> will add a choice to the boot menu which is not sealed by the TPM.
> Switching between the default sealed kernel and the alternate unsealed
> kernel should make a good TPM test.

I just replaced the whole kernel, initramfs and the .map and .config
file by old versions. Unfortunately the TPM didn't complain (secret
passphrase did display).

>>> The example usage in the AEM guide shows only 3 PCRs: 17, 18 and 19.
>>> IIRC these are amalgams of other, individual measurements and could be
>>> considered good enough for most uses. Maybe if you re-sealed with only
>>> those 3 it could avoid manifestation of what appears to be a bug (either
>>> in trousers or the TPM itself).
>>
>> Will try that soon as well.

Also tried that one; no change.

>>>>
>>>>> You may want to run 'tpm_version' and use that info to search for
>>>>> problems between your TPM and trousers. Also keep in mind that your
>>>>> system may have more than one TPM.
>>>>
>>>> It says:
>>>> TPM 1.2 Version Info:
>>>> Chip Version: 1.2.3.91
>>>> Spec Level: 2
>>>> Errata Revision: 3
>>>> TPM Vendor ID: WEC
>>>> TPM Version: 01010000
>>>> Manufacturer Info: 57454300
>>>>
>>>> I'm also pretty sure I only have one TPM as I had to explicitly buy it
>>>> and build it in my PC as the mainboard only provided an empty slot
>>>> (it's
>>>> a desktop PC != laptop). Btw it's this one:
>>>> https://www.alternate.de/ASRock/TPM-Modul/html/product/1127156
>>>
>>> If you describe the issue to the trousers-users mailing list maybe
>>> someone could offer suggestions on how to troubleshoot the problem:
>>>
>>> http://sourceforge.net/p/trousers/mailman/trousers-users/
>>
>> Good idea, I'll wait until I have time to do some more tests though.

I guess I'll ask them about TPM debugging now...