Upgrade instructions for R3.2 and QSB37 patches

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi,

I'm testing upgrade process for QSB37 patches for R3.2[1]. And it
isn't straightforward, mostly because of major Xen upgrade (4.6->4.8).
What I have currently:

1. Execute `sudo qubes-dom0-update`, but when prompted for confirmation, abort (answer `n`).
2. Shutdown all the VMs
3. Run `sudo dnf update` in dom0. Note, after this step (until you restart the system), most qvm-* tools will stop working.
4. Restart the system

As you can see, after updating Xen but before restarting the system,
things are broken. This include inability to cleanly restart the system
if any VM remain running.

Having manual procedure may be an option for "security-testing"
repository, but IMO it would be bad for "current" repository, especially
for "stable" and "long term support" release. I think we shouldn't
assume that _every_ Qubes user read qubes-announce (or other announcement
channel) frequent enough. Such system breakage would be unpleasant
surprise for anyone just applying stable (non-testing) updates.

Currently, I'm trying to abort the upgrade if any VM is running. And
display this:

***** USER ACTION REQUIRED *****
Major Xen upgrade detected (4.6 -> 4.8) and some VMs are running.
Please shutdown all of them, then resume the process by executing 'sudo dnf update' from dom0 console

But still not sure if that's the right thing to do. Maybe we shouldn't
put such upgrades to the stable r3.2 repository at all, and require
users to manually initiate such upgrade? There was a suggestion to name
an updated version as r3.3 (which means separate repository). But this
technically would also break our promise to keep "R3.2" supported at
least 1 year after final 4.0 release. And I can see how people may be
afraid of "major" upgrade of a production system.

Related:
https://github.com/QubesOS/qubes-issues/issues/3430 "Mechanism to notify
users when critical action is required"

[1] https://github.com/QubesOS/qubes-issues/issues/3460

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlpb0M8ACgkQ24/THMrX
1yzHUgf/Qw9FNu2xHk+oYM3YLRjBFAk6nE2+rxf1/1ZdwywamgtfSmq7ssdbPz8c
qfMUsMhy7lFoMzRdqAsPfxFVAIfLwEZNx8JXlsrKvcOtUljNRXOEwKqDdC3i4Fwq
jlzzLRX002EVF/gCSepPc+1x7WfQKGlA7nyRfncrIsj48F4NgOxc1oMxsOWN16V5
TMTa9tYa3I33rmAwTL4Mj7yHdUsEjyl6925d5on2EzmaXW3EU7Dgf4P3S5QbQK+q
SuJ9zG6OvIkyk574apexLnkm/SPCBqgVMER5lay6Q7wtJ9LJ+4IbrnTg7NMG6KRq
qQ59SQljkCc+5/YXZhC46zua0jvfsQ==
=0W2e
-----END PGP SIGNATURE-----

[Andrew David Wong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I agree that:

1. We should keep our promise to support R3.2 for a full year after the
release of R4.0 stable.

2. We should not force R3.2 users to install an upgrade that may break
their stable installations.

3. We should not expect every Qubes user to be subscribed to the mailing
lists, website, or social media or regularly check them for
announcements. Users who test betas and release candidates are more
likely to be on the mailing lists and see announcements, but users
who choose to stay with a stable release may be less likely to
receive communications from us.

I think the best solution is to display the "user action required"
message (we can work on a more verbose version) whenever the user runs
qubes-dom0-update (at the very least), but do not take any action
pertaining to this special upgrade unless the user agrees. The user
should be able to see the message, decline the special upgrade (or
refrain from taking the steps required to initiate it), and complete
qubes-dom0-update normally. This is important, because even if the
user wants to perform the special upgrade, she must be allowed to
decline it initially so that she has a chance to save work and make
backups before any potentially breaking changes are introduced to the
system.

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEZQ7rCYX0j3henGH1203TvDlQMDAFAlpb2WoACgkQ203TvDlQ
MDDt+BAAx1Oty+yDbai93Jk1XnlJlHXd5oK7WvF71yIRWb+XSzlpG4/M+flMx/TT
T5Sm8bPDER72ZlFJMP9fuOp187Nrmrpp6zai9Z0l8O6j/Aqgq1t5WQ8DVSQPH9Ts
zl1HldrE9U+K0Yv8dqeZ7qT04dqf+xRoFCb2VnlWHnocZjrImCPMeRzCZG4AJl83
e8J5+r3ZvkT1NlDZDT1DDwY5rS/2bz50FOPfuGsKeISC5lT2fws8zuzwzTMPY6zq
QHZwtLOocjBIFTFPbxWUmEr/YxeTCVc9ukSdidXvj1raRejhn9NGWp+2PQUDlXbg
FnaRG28q4ww2UY5I1P1gWkC7BKMYk6XKYtbVQn34XolwR7Iy6lhNMEHnA4iS4bFb
LTa9k2g105H4TF8eIxN3uo8ouBGqo9nIkziisbGNit2zAr9/HSklDi6Ig013+0im
2fQqE6J8kJEMalOQs8KlSwmyFv5kAn9qmDi/8ZzhdFlTWfLWjIC3uxC/1EFFfugM
oUkj5QdxpfBO2WOr/4G+xmJW2wdYuxxlDwb9bB/xHFso2XCsakqIaIljrbv2BKpW
OMyJoWdYPOogd0EpumWELDX1RSIbJcckWjlGYJ1RkkQ2dtLDbtTlg6MAulj+MlO1
CCFd1eFdED8CeuLqnKLNrJnrKn1ua1g1Qtll7H+ivZA3Et4oXcE=
=cEF0
-----END PGP SIGNATURE-----

[Reg Tiangha]

My two cents: Because of the security craziness going on right now, I
don't think that anyone would think too badly if the one-year extended
support promise of R3.2 were broken in favour of R3.3. Or you could
retcon it to say that the R3.X series (whatever the latest version ends
up being) will be supported for one year after R4.0 is released, which
would open up the option of using the R3.3 moniker if that makes the
maintenance easier. At this point, I think it's better to have a secure
system that's easier to maintain rather than to be beholden to having to
stick with a version number just because. If setting up a separate
repository makes upgrading (and developer maintenance!) easier, then go
ahead and do it. These are extenuating circumstances, after all and I
don't think anyone anticipated having to backport another major Xen
version this late into the lifecycle.

[Zrubi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 01/14/2018 10:51 PM, Marek Marczykowski-Górecki wrote:
> There was a suggestion to name an updated version as r3.3 (which
> means separate repository). But this technically would also break
> our promise to keep "R3.2" supported at least 1 year after final
> 4.0 release. And I can see how people may be afraid of "major"
> upgrade of a production system.

I think the R3.3 would be a better solution, because:

Your promise about 3.2 is still fine. Just rename it to 3.x, where x
is the latest release from the R3 branch.

People are afraid of major upgrade for several good reasons. So please
do not break the current 3.2 status. (it has been broke because of new
major kernel versions before) So those people can decide to keep the
vulnerable 3.2 and/or TEST and jump to 3.3.

Jumping to a new version is much easier and much safer choice (as a
user) because of the clean install + restore from backup upgrade path,
instead of a one way upgrade "choice" that may leads to a bricked OS.

Major changes (like kernel and Xen version changes) in 3.2 will break
things for sure and much harder to test it.



- --
Zrubi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJaXFgkAAoJEBozWgtUjzdkfTcP/iqbTsr1jfG6KlqRErpO3Pzn
AzkP7YX2xP/sdf6AqFUaDFi52SdIHwwobTE9YKT6uD9SmoI0Bju3u8Ts/YcoO4S0
g3DpU1/dnUDljIXBB2mwERwBVOuUD3QwRcmXw4s05mglFRTsycKgO01Q91n7oxiB
Y8XuIwF3s0Bh2+OGIzsxDh9Y0XO4/DipCSh1ERHmpD1StE3g0GSnDtLN1VJnbyzk
/kzqCfCk719nSUDdfA9cJc3xa0CdAUOhAlg6k7fPSTgLE00WTv7BbvNiLBsjp20H
vez9+cSdCb2LCLb+In4VQTcmjP7ppEuWFmOK3s2IOodzjw60pCF4TOFKnka4Vk+q
x6JLSLHA9HRcBLmOmnsJH43UEtizOnaxFHCRShvsmqs1VERP+AbWDnGOLPOt0wcA
5OIN1ny02L9SHomXKXxe4gC3CCiP5yTVZJUbgW/dwcL3Sh6koWXg3daUrzrZSNBE
z7FBu8X8msKwXm2L7WxsVNFp0sDsJPOXmaiJNt1F9Hl8ywHrdFy6yUxdDT7UJ28s
QscikuE1eEEWAEZ5KfxDvjHLg4NNNNmGU9d9T6mD+RhSoW20iTd+/O1d+Idrriwg
Bsj9ATvljTH1wB9tJhshSqZ1ocUAxd4Rg9a5dN0ZinOYIGcN2wN8HYWVIs+SJ3mc
tlgW3Z2QSQ5CrSvQm7dp
=tnzz
-----END PGP SIGNATURE-----

[Ilpo Järvinen]

On Mon, 15 Jan 2018, Zrubi wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 01/14/2018 10:51 PM, Marek Marczykowski-Górecki wrote:
> > There was a suggestion to name an updated version as r3.3 (which
> > means separate repository). But this technically would also break
> > our promise to keep "R3.2" supported at least 1 year after final
> > 4.0 release. And I can see how people may be afraid of "major"
> > upgrade of a production system.
>
> I think the R3.3 would be a better solution, because:
>
> Your promise about 3.2 is still fine. Just rename it to 3.x, where x
> is the latest release from the R3 branch.
>
> People are afraid of major upgrade for several good reasons. So please
> do not break the current 3.2 status. (it has been broke because of new
> major kernel versions before) So those people can decide to keep the
> vulnerable 3.2 and/or TEST and jump to 3.3.
>
> Jumping to a new version is much easier and much safer choice (as a
> user) because of the clean install + restore from backup upgrade path,
> instead of a one way upgrade "choice" that may leads to a bricked OS.
>
> Major changes (like kernel and Xen version changes) in 3.2 will break
> things for sure and much harder to test it.

Since HW requirements also change due to the approach in the current fixes
(VT-x required), releasing them as R3.3 would make more sense, IMHO.

There might (will?) also be a fix for Xen that makes PVs meltdown
resistant eventually (there was already some early test patch at xen
devel last week). That, when ready, would naturally apply to R3.2.


--
i.

[Holger Levsen]

On Sun, Jan 14, 2018 at 04:28:49PM -0600, Andrew David Wong wrote:
> I agree that:
>
> 1. We should keep our promise to support R3.2 for a full year after the
> release of R4.0 stable.
>
> 2. We should not force R3.2 users to install an upgrade that may break
> their stable installations.

I kind of agree with these two, but then…

a.) if 3.2 suddenly gets xen 4.8 I'm not sure it's really 3.2 anymore.
I tend to think that Qubes 3.x with xen 4.8 should be called Qubes
3.3.
b.) if fixing spectre and meltdown really needs xen 4.8 I'm not sure
there is a point keeping 3.2 called "supported". I also don't think that
Qubes can be blamed for this, as until now it was sufficient to keep
the kernel up2date… also Qubes 3.3 would keep the promise "as far as
possible" while making clear that there are more changes than usual.
c.) continuing to support 3.2 with (such) known security issues is
probably pointless... if that's the case I think this should be
communicated clearly.

> 3. We should not expect every Qubes user to be subscribed to the mailing
> lists, website, or social media or regularly check them for
> announcements.

agreed.


--
cheers,
Holger

[bow...@gmail.com]

I second this line general line of thoughts with 3.3 and more precisely what Lazlo is saying about 3.2. However I suspect you'll decide to support 3.3 for 1 year but 3.2... I doubt users will stay long on it. If it is a prod system they should really have a staging system to test the upgrade and accelerate it.

[Vít Šesták]

I mostly agree with Axon. But I'd like to ask how is it supposed to behave if user declines the update?

a. User no longer gets any update for dom0 until they decide to accept the update.
b. User receives updates, but they are not protected from Meltdown/Spectre.

Also, it depends on what happens with users without VT-x/AMD-v. Such CPUs are probably rare, but still it might make sense. Also, such CPU are likely not to be vulnerable. If they can still work with newer Xen, just with all VMs in PV, it is OK. (I hope this is possible.) If not, then I'd like to follow the approach B.

If you are pretty confident about CPU support, then approach A will probably make more sense, because it requires less effort on support. (No need to support two parallel versions.)

On version number: Approach B is a candidate for version number increase (in order to distinguish from two versions). Approach A is not much a reason for version increase.

Just noting for the future: OSes like Fedora Atomic or NixOS could have made such update easier, for two reasons. First, it could update the Xen tools after reboot. Second, it could have made the rollback easier. I know it is too late now, but it might be good to use this in, say, Qubes 4.1.

Regards,
Vít Šesták

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, Jan 17, 2018 at 02:40:08PM -0800, Vít Šesták wrote:
> I mostly agree with Axon. But I'd like to ask how is it supposed to behave if user declines the update?
>
> a. User no longer gets any update for dom0 until they decide to accept the update.
> b. User receives updates, but they are not protected from Meltdown/Spectre.

Closer to "b", but not exactly. Most system components will be exactly
the same and can receive updates. This applies to all packages in VMs,
and most in dom0, except:
- xen
- core-admin
- probably libvirt

That's about technical side. It doesn't mean we will keep such parallel
"3.2" and "3.3". Because it also make testing harder (more
possible configurations).

> Also, it depends on what happens with users without VT-x/AMD-v. Such CPUs are probably rare, but still it might make sense. Also, such CPU are likely not to be vulnerable. If they can still work with newer Xen, just with all VMs in PV, it is OK. (I hope this is possible.) If not, then I'd like to follow the approach B.

Yes, PV will remain supported. And we'll switch to PVH automatically if
hardware allows for that.

Oh, there may be also third option: do not upgrade Xen version in 3.2
and apply Meltdown mitigation patches for PV on Xen 4.6. Apparently
development of those patches went much much faster than we (and Xen
Security Team) have anticipated. This is solution similar to what Linux
have done, but simplified version. Malicious VM could still use Meltdown
to read some of Xen memory, but it is largely limited (especially no
access to the whole host memory).
Similar to the Linux version - this will have noticeable performance
impact. The mitigation code can be disabled from Xen command line, but
that will make the system vulnerable to the attack again. We're testing
this approach right now.

PVH option (original plan for R3.2) do not have this problem,
if hardware allows (VT-x/AMD-v + HAP/EPT/SLAT/RVI)[1].

[1] https://github.com/QubesOS/qubes-core-admin/pull/178#issuecomment-357520110

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlpf2PAACgkQ24/THMrX
1yzsEggAlNvEfWfMh3Juf6/TvcBtX4RQHfzQycQHXUEFyA1BtP6VQUPnxkHWhJyM
49Qve48wsjGyUCjym6bv7u6Cp0zcU+UJwEoKaqnfD+bOiED/DISt3Ql5Po7XxpdG
ezEGW24/qzc4c2r9pjRNndyzPtBdbRdt1sLPDdlhgAgYRP0muNiLd7urytuFw1gV
WmKCL8NJiu8rN2fE0WhxoUdAEu3c+9IlMvB2TbCu0PrgKjymzAo3+5thwHGtBvbA
izPopmlaFnDsCHiXcqShQhxS52K5v311IhKCX0ddCF/isvNNV3530LI8AR1GxnlF
eSCMQymIAE3KbcdSWGO5I4twoknJEA==
=UNp/
-----END PGP SIGNATURE-----

[Blacklight447]

Do I understand correctly that PVH is ready for implementation? Last time I checked it was said on github that it wasnt.

[Vít Šesták]

On Thursday, January 18, 2018 at 12:15:18 AM UTC+1, Marek Marczykowski-Górecki wrote:
> Closer to "b", but not exactly. Most system components will be exactly
> the same and can receive updates. This applies to all packages in VMs,
> and most in dom0, except:
> - xen
> - core-admin
> - probably libvirt

This sounds like more close to option A, because Xen updates are IMHO the most important ones.

> Yes, PV will remain supported. And we'll switch to PVH automatically if
hardware allows for that.

Hmm, I thought that HVMs will be used, not PVHs.

> Oh, there may be also third option: do not upgrade Xen version in 3.2
> and apply Meltdown mitigation patches for PV on Xen 4.6.

So, to sum it up:

Pros:

* Works even without VT-x and SLAT.
* Conservative approach – not so likely to break existing installations.
* Easier update process for users.
* Maybe it is finally easier to deploy the patch. (Needs some work, but does not require to develop update process and agree on the right one.)

Cons:

* Performance hit.
* Not sure if Spectre is also addressed.
* No bonus like having PVHs. (But this is not a goal.)

Am I correct?

If Spectre can be also addressed in Q3.2 via the third option, then I believe that the third option is the right thing to do. While some users (those who notice) might complain it is slower, it is better than users complaining it does not boot at all after an update. (While this could be a problem of minority, all of such users would notice.)

Also, those who complain about performance can try Qubes 4. Well, it is just a RC and it has higher requirements (most notably VT-d), but it still can make the performance hit in Q3.2 less bad. After all, the selling point of Qubes is not performance, it is security.

Also, the people who don't want the performance hit and cannot upgrade to Q4 can:

a. Disable the patch. (Reasonable on CPUs that aren't affected.)
b. Maybe migrate to HVMs?

If Spectre cannot be addressed via the third option, then the conclusion is not so clear for me.

Regards,
Vít Šesták 'v6ak'

[Chris Laprise]

On 01/17/2018 06:14 PM, Marek Marczykowski-Górecki wrote:

> Oh, there may be also third option: do not upgrade Xen version in 3.2
> and apply Meltdown mitigation patches for PV on Xen 4.6. Apparently
> development of those patches went much much faster than we (and Xen
> Security Team) have anticipated. This is solution similar to what Linux
> have done, but simplified version. Malicious VM could still use Meltdown
> to read some of Xen memory, but it is largely limited (especially no
> access to the whole host memory).
> Similar to the Linux version - this will have noticeable performance
> impact. The mitigation code can be disabled from Xen command line, but
> that will make the system vulnerable to the attack again. We're testing
> this approach right now.

If this Xen 4.6 patch were more robust in protecting memory, I'd opt for
it instead of upgrading Xen.

As for extended 3.2 support, I think this should be commuted to mean
"latest release from the 3.x series". You could release an upgrade as
either 3.3 or 3.2.5 for example, signifying a large bug fix.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Vít Šesták]

On Thursday, January 18, 2018 at 11:25:11 AM UTC+1, Chris Laprise wrote:
> If this Xen 4.6 patch were more robust in protecting memory, I'd opt for
> it instead of upgrading Xen.

I believe that PVHs are more robust. But I'd prefer to stay conservative in Qubes 3.2 and progressive in Qubes 4.0.

> As for extended 3.2 support, I think this should be commuted to mean
> "latest release from the 3.x series". You could release an upgrade as
> either 3.3 or 3.2.5 for example, signifying a large bug fix.

This would make sense if 3.3 is released and 3.2 is supported for standard 6mo. But I see the reason against it: It requires more work and time that could have been spent elsehow.

On the other hand, I don't like an idea of creating something like 3.3 with users of 3.2 being forces to update. This goes against nature of Qubes. This can be acceptable when there is no other way.

Regards,
Vít Šesták 'v6ak'

[awokd]

On Wed, January 17, 2018 11:14 pm, Marek Marczykowski-Górecki wrote:

>
> Oh, there may be also third option: do not upgrade Xen version in 3.2
> and apply Meltdown mitigation patches for PV on Xen 4.6. Apparently
> development of those patches went much much faster than we (and Xen
> Security Team) have anticipated. This is solution similar to what Linux
> have done, but simplified version.

This sounds less prone to breakage. I could see keeping the 3.2(.1)
version number if this works. Any idea how soon you'll be able to make a
decision? I want to go through all the Qubes docs and split them out
between 3.x and 4.0 but it would be useful to know what x= before I do
that.

[Tai...@gmx.com]

On 01/18/2018 08:28 AM, Vít Šesták wrote:

> On the other hand, I don't like an idea of creating something like 3.3 with users of 3.2 being forces to update. This goes against nature of Qubes. This can be acceptable when there is no other way.

I am someone who thinks large OEM's should be forced to provide security
updates for 5 years, and I use my computers for 10 so bear this in mind
when I say that some forced updates are worth it and that we shouldn't
entertain the idea that a PC without IOMMU is safe and should continue
to be used.
It has been available on power user devices for 5 years and you can get
a laptop with it used for only $100.