How to rollback Dom0 updates?

[Simon]

Hello everybody,

Is there a way to rollback updates which corrupted a Qubes-OS system?

I checked DNF history, but it seems to have been disabled / bypassed for
all events following the OS installation back in September:

--------------------- 8< ----------------------

[user@dom0 ~]$ sudo dnf history
ID | Command line | Date and time | Action(s) |
Altered
-------------------------------------------------------------------------------
5 | --exclude=qubes-template | 2016-09-19 21:10 | Install |
1 <
4 | remove cairo-dock-plug-i | 2016-09-07 18:19 | Erase |
19 >
3 | --exclude=qubes-template | 2016-09-07 14:34 | Install |
14 <
2 | --exclude=qubes-template | 2016-09-07 14:24 | Install |
5 ><
1 | | 2016-09-04 17:57 | Install |
937 >E

--------------------- 8< ----------------------

Is there any equivalent feature allowing update rollback in Qubes-OS for
the Dom0 domain?

Thanks by advance,
Simon.

[Andrew David Wong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

That's strange. My dom0 dnf history shows all my updates, including recent ones.

Are you sure your dom0 has been getting updated? How are you performing updates?

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJYTPPUAAoJENtN07w5UDAwDr4P/3oO0gkaeH/x1grbCNe93Qa7
gzy50wk+8u1aF1KiJbJpDzJ2U3xsHC097paQU5nyP5jhmpfojB/V9M0VbSV5Kh3Q
r37l20z3cFL6xXjXENzH6ef5gNGwfLsJbWzBIOKJf8KWni5k/XMUetLUwm3Xv06S
/VxVZDkI5TYM1tpvU0MbNApJAwxGsEOm5+KUvqpWvpcrdBMDoygfSEXCoc+iEGfR
FbF+CIw3SgVY5l2NrfTQyRcrr9CYdBLtsByRQHOYhbymqSi1mL7WDMYmAxcoMphd
g2XCnlQUgrrsA3WuXXK2gM+i784uIdditvcKdhiBsMS6GcVt0XGYEHYnzGR8/A8h
asjwfxNTLVia3qt4zg5GTXDiCZPtK/04cN97lARMXleaf4zSDhfQ1hTqsqji+iRd
lx352Vby2KTvZE38eWJZG7Nc4X+OQxUfc5g2xpR7BJ14WZHejLK6iHimCKshDgNW
liF6sYGryPCFMtjTANh6lPS7LaN3tO/IJV2w27ayy1kOZtq+F8OigebaqIPhEy3s
22FsjXkn1+EhtclSvb/I4YwgI+Dg7STrIz54QIMR5yGDrd+V47tH2486+hBlgfTR
xddDdRhCFYtp452ie9orLZyBvFTC2Wh1XAVEka8EEoUANLkrEh/05DtZBdSC2owf
BA4wRJhNIFB4oKeXKK5a
=RT7P
-----END PGP SIGNATURE-----

[Simon]

Le 2016-12-11 07:36, Andrew David Wong a écrit :
> That's strange. My dom0 dnf history shows all my updates, including
> recent ones.
>
> Are you sure your dom0 has been getting updated? How are you performing
> updates?

Hello Andrew,

When I do `rpm -qa --last' I can list all installed and updated packages
with the appropriates dates, so I can confirm that dom0 gets correctly
updated:

---------------- 8< ------------------------

[user@dom0 ~]$ rpm -qa --last | head
perf-4.8.12-100.fc23.x86_64 Sat Dec 10 18:10:11 2016
pciutils-libs-3.5.2-1.fc23.x86_64 Sat Dec 10 18:10:11 2016
pciutils-3.5.2-1.fc23.x86_64 Sat Dec 10 18:10:11 2016
dnsmasq-2.76-2.fc23.x86_64 Thu Dec 8 23:33:14 2016
dmidecode-3.0-6.fc23.x86_64 Thu Dec 8 23:33:13 2016
qubes-input-proxy-1.0.8-1.fc23.x86_64 Thu Dec 8 23:33:12 2016
qubes-gpg-split-dom0-2.0.24-1.fc23.x86_64 Thu Dec 8 23:33:12 2016
perl-Time-Local-1.250-1.fc23.noarch Thu Dec 8 23:33:11 2016
kernel-qubes-vm-4.4.31-11.pvops.qubes.x86_64 Thu Dec 8 23:33:06 2016
kernel-4.4.31-11.pvops.qubes.x86_64 Thu Dec 8 23:32:57 2016

---------------- 8< ------------------------

To update, I simply use the Qubes VM Manager. The only things which may
be noticeable are the following:

- Usually I update all the templates and Dom0 simultaneously: I right
click on the AppVM template and click `Update VM', I do this for each
AppVM in a row (without waiting for the update of the previous AppVM to
terminate) and finally for Dom0 (since it locks access to the Qubes VM
Manager during the while Dom0 update process, which is the longest, see
below).

- I have the impression that Dom0 updates are downloaded twice, most
probably an issue around the proxy feature (there is no such issue with
the templates, updating Dom0 takes twice as much time as updating the
templates).

- I regularly have ghost updates (the icon announcing that updates are
available while there are none) but I think this is a known issue.

Best regards,
Simon.

[Andrew David Wong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

When you say "AppVM," do you actually mean TemplateVM? There should
normally be no reason to update AppVMs.

> - I have the impression that Dom0 updates are downloaded twice, most
> probably an issue around the proxy feature (there is no such issue with
> the templates, updating Dom0 takes twice as much time as updating the
> templates).
>

Hm, that would be odd. The normal dom0 update process is for the updates
to b downloaded by the UpdateVM (default sys-firewall), then transferred
to dom0, where the signatures are checked, and the updates are installed.
This might have the appearance of the updates being downloaded twice,
but they're really only downloaded once.

> - I regularly have ghost updates (the icon announcing that updates are
> available while there are none) but I think this is a known issue.
>

Yes, that's a known issue:

https://github.com/QubesOS/qubes-issues/issues/2086

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJYTbrxAAoJENtN07w5UDAwhlgP/3zvrF174QxmCIPKM0CO1biT
R74ZdPm67yttn3ONy6YhuBpcVTtggKCjFBsc95lN8YopbqtImUyTIRC180u6RQ1x
EFVtdiGlFijtyBHxFCvkahMXpAdxJl3ZLVB26Ji+M3gr74wdAMESPe5tajQbuw5j
cmPb7q14+zOa5ei84J1b+OD53IryPxe89ZSwN3smIlq/DdhgWw38RpdM8nJbEJe7
rlRi1he52sx+E7aJqlk8oEuRxUFPg6nEBe7ijD+EW4GjPQqb2KS5oC6P1/1kyvFq
0SAnHpxIulW3mULRO7MM3SOhZ7I7H7mWYq7NsMfsVqf4Vl64WkJUgHC7cv58JPLW
b6/NmF+Aes5kre7qVcVTZdXU8gDLA/40NNZr3HRMqMJhWEVCVVAjyZg/ZwhR1B1m
Hr+OIdidMYyy8PoSgqkbB8KW+roZI3may7yXWJ7bDocO4i+ou+GwyX13mtgOGoth
dbvN5g+pPiK9t2XPRRyBxC9HRuoJB4dmfJKa2jK/tSR2p4i9f1/lsZwqGJfckkxc
el0N1VyAes5gL1iV5nQP9bpziKLVaWYcTNt4h21m3zXQYWd6//Ci+lhLuiO36CAK
YTHzWtT45gRCsZVTZRjBQXtgxWq0jf83l2q11fF8P+rqOZG9pK0xxJAP36pnQCvX
qGkjDxzqMf8NT0XvS0tV
=PK+K
-----END PGP SIGNATURE-----

[Simon]

Hi Andrew,

Le 2016-12-11 21:45, Andrew David Wong a écrit :
>> - Usually I update all the templates and Dom0 simultaneously: I right
>> click on the AppVM template and click `Update VM', I do this for each
>> AppVM in a row (without waiting for the update of the previous AppVM
>> to
>> terminate) and finally for Dom0 (since it locks access to the Qubes VM
>> Manager during the while Dom0 update process, which is the longest,
>> see
>> below).
>
> When you say "AppVM," do you actually mean TemplateVM? There should
> normally be no reason to update AppVMs.
>

Yes sorry, that's indeed what I mean, I do this for each TemplateVM and
not AppVM (usually when I update most AppVM are shut down).

>> - I have the impression that Dom0 updates are downloaded twice, most
>> probably an issue around the proxy feature (there is no such issue
>> with
>> the templates, updating Dom0 takes twice as much time as updating the
>> templates).
>>
>
> Hm, that would be odd. The normal dom0 update process is for the
> updates
> to b downloaded by the UpdateVM (default sys-firewall), then
> transferred
> to dom0, where the signatures are checked, and the updates are
> installed.
> This might have the appearance of the updates being downloaded twice,
> but they're really only downloaded once.

Well, this morning again there was a new ghost update : I launched Dom0
update, it was processing for 14 minutes and downloading about half of
the time, before finally concluding that there is actually no update
available. Odd and suboptimal indeed...

While trying to investigate a bit further, I stumbled on this
interesting "property" of Fedora which randomly selects a "nearby"
source to download the update. There was indeed updates currently
available for both my Debian and Fedora packages:

- Debian update took less than a minute,
- Fedora is still ongoing, with a download speed hardly reaching 80
KB/s.

I think I know understand why updating Dom0 seems so slow, thank you
Fedora for allowing even the crappiest server to act as an update source
as long as it is among the closest ones geographically speaking :( ...

I already stumbled on this before and had to cancel an update and try
later in order to update my Fedora template due to such poor performance
and an estimated time of completion counted in hours. A few time later,
a decent server offering a download speed counted in MB instead of KB
and the update was done in less than a minute too.

If I have some time, maybe I should try to find the culprit(s) and
blacklist it/them somehow. By the way I'm a bit surprised to see the
/var/log/tinyproxy to remain empty even after all those tests.

And to end-up on a more positive note I find it great that the template
VM now shut down themselves automatically once the update is done :) !

Have a nice day,
Simon.

[Manuel Amador (Rudd-O)]

The reliable way to do rollbacks is to:

1. Install Qubes OS on a btrfs file system.
2. Install the dnf / yum plugin that will snapshot your system right
before upgrades.

Slightly less convenient:

1. Migrate your Qubes OS to ZFS.
2. Manually ZFS snapshot your Qubes OS before dom0 / template upgrades.


--
Rudd-O
http://rudd-o.com/