feedback for todays kernel-qubes-vm update (4.4.55-11)

[Joonas Lehtonen]

Hi,

just a quick notice about todays kernel update.

After upgrading, the new kernel 4.4.55 became the new default for all
VMs that previously used the default kernel, but
VMs would no longer boot because they claim that an old kernel the one
that got removed during the upgrade (4.4.11?) is no longer present even
though the VM was configured to boot the default (4.4.55).

Easy workaround:
configure the VM to boot 4.4.38 and save.
reopen the preferences and configure it to boot the latest default
-> boot the vm

regards,
Joonas





Installed: 1000:kernel-qubes-vm-4.4.55-11.pvops.qubes.x86_64
Updated: qubes-gui-dom0-3.2.10-1.fc23.x86_64
Installed: 1000:kernel-4.4.55-11.pvops.qubes.x86_64
Updated: qubes-mgmt-salt-dom0-qvm-3.2.1-1.fc23.noarch
[..]
Updated: qubes-manager-3.2.11-1.fc23.x86_64

[Joonas Lehtonen]

Joonas Lehtonen:

> Hi,
>
> just a quick notice about todays kernel update.
>
> After upgrading, the new kernel 4.4.55 became the new default for all
> VMs that previously used the default kernel, but
> VMs would no longer boot because they claim that an old kernel the one
> that got removed during the upgrade (4.4.11?) is no longer present even
> though the VM was configured to boot the default (4.4.55).

This was mainly an UI thing. qvm-ls -k displayed it correctly. These
affected VMs used to have the now-removed kernel version 4.4.14-11.
Qubes Manager just can not display not installed kernels.

[Michael Carbone]

Joonas Lehtonen:

> Hi,
>
> just a quick notice about todays kernel update.
>
> After upgrading, the new kernel 4.4.55 became the new default for all
> VMs that previously used the default kernel, but
> VMs would no longer boot because they claim that an old kernel the one
> that got removed during the upgrade (4.4.11?) is no longer present even
> though the VM was configured to boot the default (4.4.55).
>
> Easy workaround:
> configure the VM to boot 4.4.38 and save.
> reopen the preferences and configure it to boot the latest default
> -> boot the vm

Encountered the same issues on update. There was also a broken pipe
error during the update process.

Workaround works, needs to be done manually for each affected VM (which
qvm-ls -k lists).

--
Michael Carbone

Qubes OS | https://www.qubes-os.org
@QubesOS <https://www.twitter.com/QubesOS>

PGP fingerprint: D3D8 BEBF ECE8 91AC 46A7 30DE 63FC 4D26 84A7 33B4

[Marek Marczykowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Apr 18, 2017 at 05:33:00PM +0000, Michael Carbone wrote:
> Joonas Lehtonen:
> > Hi,
> >
> > just a quick notice about todays kernel update.
> >
> > After upgrading, the new kernel 4.4.55 became the new default for all
> > VMs that previously used the default kernel, but
> > VMs would no longer boot because they claim that an old kernel the one
> > that got removed during the upgrade (4.4.11?) is no longer present even
> > though the VM was configured to boot the default (4.4.55).
> >
> > Easy workaround:
> > configure the VM to boot 4.4.38 and save.
> > reopen the preferences and configure it to boot the latest default
> > -> boot the vm
>
> Encountered the same issues on update. There was also a broken pipe
> error during the update process.
>
> Workaround works, needs to be done manually for each affected VM (which
> qvm-ls -k lists).

Do we have an issue for this on github? I can't find one.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJY9k+wAAoJENuP0xzK19csWoEIAIvoGFgAddPiuiRvT49/lBGe
7CioOBXuScjhdwKpkbtHnohqKByQfV87eGCEettt4na5SwDuzX27ltMuqRTgRUbp
wtPVrfNUTLrnUFkznKbYeli7x0O0WUc3KYe1psGzFr5vrxRKFFIsPnhsLU7R3uhW
v9uU4iywnbbWabE4qVlHh/NhtY3STzJpdfTqV1oarJCSGhoz3nG6co94uPiIsI7H
TP4ShcLxca5n7pWBTgKJENONi1LmgPRA+L/qzsOTM6Pkemdx6YJq0iaLHE75WgbS
+Zbhnv2KVEOGq73RN5J1v1GOYnfIPLI/kt+oN+vNj3NicZFUd7LSwXp4Ui4HK3E=
=CY/B
-----END PGP SIGNATURE-----

[Michael Carbone]

Marek Marczykowski:

> On Tue, Apr 18, 2017 at 05:33:00PM +0000, Michael Carbone wrote:
>> Joonas Lehtonen:
>>> Hi,
>>>
>>> just a quick notice about todays kernel update.
>>>
>>> After upgrading, the new kernel 4.4.55 became the new default for all
>>> VMs that previously used the default kernel, but
>>> VMs would no longer boot because they claim that an old kernel the one
>>> that got removed during the upgrade (4.4.11?) is no longer present even
>>> though the VM was configured to boot the default (4.4.55).
>>>
>>> Easy workaround:
>>> configure the VM to boot 4.4.38 and save.
>>> reopen the preferences and configure it to boot the latest default
>>> -> boot the vm
>
>> Encountered the same issues on update. There was also a broken pipe
>> error during the update process.
>
>> Workaround works, needs to be done manually for each affected VM (which
>> qvm-ls -k lists).
>
> Do we have an issue for this on github? I can't find one.

Just made one:

https://github.com/QubesOS/qubes-issues/issues/2757

[Marek Marczykowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Apr 18, 2017 at 07:41:03PM +0200, Marek Marczykowski wrote:
> On Tue, Apr 18, 2017 at 05:33:00PM +0000, Michael Carbone wrote:
> > Joonas Lehtonen:
> > > Hi,
> > >
> > > just a quick notice about todays kernel update.
> > >
> > > After upgrading, the new kernel 4.4.55 became the new default for all
> > > VMs that previously used the default kernel, but
> > > VMs would no longer boot because they claim that an old kernel the one
> > > that got removed during the upgrade (4.4.11?) is no longer present even
> > > though the VM was configured to boot the default (4.4.55).
> > >
> > > Easy workaround:
> > > configure the VM to boot 4.4.38 and save.
> > > reopen the preferences and configure it to boot the latest default
> > > -> boot the vm
> >
> > Encountered the same issues on update. There was also a broken pipe
> > error during the update process.
> >
> > Workaround works, needs to be done manually for each affected VM (which
> > qvm-ls -k lists).
>
> Do we have an issue for this on github? I can't find one.

Are you sure those VMs are really configured to use default kernel?
You can check like this:

[marmarek@dom0 ~]$ qvm-prefs testvm|grep kernel
kernel : 4.4.55-11 (default)
kernelopts : nopat (default)

The important part is "(default)" word.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJY9lkvAAoJENuP0xzK19cs/OIIAI+7CM8j6CtUq+N5IVwS9TS3
gRoU1UNkp9ecKx1KPHtcnYT7yG078r1zGQfc8IQs2QsFPEEzxWL6DCewYG+0cRLx
QcPn1rkiw1qQNHHGrv7CQxUSJrQl6CHv/0ctzhXZsNUBZXFBlOwTxtanuD2Xjqbc
0/9f1+KLJIV6qnSRnuM939p80Rz1a3Dm4kddnECgLmCZC8FnFafgK15EUwtQt6Sk
nxd4jETd6XSuvWni3OznSnLkH62ZOB8HOFDHkEEs6nrJFEdfFYRHSrCmOn01rGs4
pPcwuGyec+H2P3dTGVE0gHkzNF1UFgjfpWZEAxILCAPdlNySOgvN6c9IBid5RdQ=
=Nxb3
-----END PGP SIGNATURE-----

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

So, Qubes Manager shows still old kernel? Have you tried restarting it
(Qubes Manager)?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJY9lm9AAoJENuP0xzK19csoA8H/jNou25t/3/ebbOHPjWTCmEd
FT6fouXFm7tN4oDw+SlUx6Hf+h/gzVhQuJc2SZIGgFT1Xk1Isj2aQvEHJvI0WLfG
XCjoe3EzYRuT6WIo77Zcbbl5+oJbeQ2Hu4Kf0kT1oWx0fL+CfzY05sIhwZrgiMLV
EnHEdOVra2aVuV67BU830rK6iu1H7TOw56ckIhrwPEdeKNamveYlh9a8Uvw7LTqo
RhVZX8wuA49/8ZJP8F31ftZoz5YDWFjxJ/jNt4V1ycAdMh5AV8BTQzf50JAlWOiy
ov1ZOKFYtNrrbJ893pTq70qYIIanK/r8MZhjhyLiaR5id05MAx90F1Ml7MzPDWs=
=Bqzx
-----END PGP SIGNATURE-----

[Reg Tiangha]

I've posted my experiences up on the GitHub issue, but here it is too:

I've actually noticed this in my kernel testing, but I never reported it
because I figured it had something to do with me going off script in
compiling my own kernels.

Anyway, what I've noticed:

It only really occurs whenever dnf removes an older version of
kernel-qubes-vm whenever dnf's installonly_limit threshold is exceeded.
If you up that number higher or don't reach it, it doesn't occur and VMs
set to use the default kernel get updated to use the new default kernel
whenever a newer version of kernel-qubes-vm is installed.

Specifically, a VM that's set to use the "default" kernel doesn't change
properly to a new default whenever an older kernel-qubes-vm package is
removed, even if qvm-prefs shows that VM is set to use the kernel
currently set to default in Qubes Manager. If that VM had been set to
use a specific kernel version prior to installation (i.e. not set to
default or to pvgrub2), it's fine and the setting stays after
uninstallation (although I never tested what happens if it's set to use
a specific kernel version and then you uninstall that version).

So I think it might have to do with the kernel-qubes-vm uninstall
scripts not cleaning things up properly, however, fixing that may not
necessarily fix any VMs that are currently misconfigured (in which case,
a user would need to toggle switching between kernels in Qubes Manager
in order for it to stick again, which is how I've been working around it).

[Andrew David Wong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 2017-04-18 10:33, Michael Carbone wrote:
> Joonas Lehtonen:
>> Hi,
>>
>> just a quick notice about todays kernel update.
>>
>> After upgrading, the new kernel 4.4.55 became the new default for
>> all VMs that previously used the default kernel, but VMs would no
>> longer boot because they claim that an old kernel the one that
>> got removed during the upgrade (4.4.11?) is no longer present
>> even though the VM was configured to boot the default (4.4.55).
>>
>> Easy workaround: configure the VM to boot 4.4.38 and save. reopen
>> the preferences and configure it to boot the latest default ->
>> boot the vm
>
> Encountered the same issues on update. There was also a broken
> pipe error during the update process.
>

Separate issue for the broken pipe and other errors:

https://github.com/QubesOS/qubes-issues/issues/2760

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJY9p+5AAoJENtN07w5UDAwi/AP/2ZRhctqU4K9ORvSjB4ZX+RR
GgyZJpgxR+8PqdTHZsZi5VWWFHALaPQXLON6JDbHqkQTgM9F5oqhdWUenabZSYGR
NNBU1993ZBjNlnOKHHLjfACE4iHhg/8RqV1fV3JNSv44S3l1DZKOUJPRQNsKHU3X
+Wy+6xjyWoLA4hjVF8yyRu6kXv4Fv4a7SOa0YRSkEnGl5q9q+sPymD2RMEmxZpE7
C/PbLcw3PK+1Evw893cx/iBU0Nn9EbL/hs558XP983JpridO41ULYjY2/YdyxohU
iYo+TqTXWlR5wBgPt3NZGayJeTC99b/Pzx8TRDDqk9rvRxVikgPuLN135+YC9ugL
VwALbifkF+SagbboNLhYhsb5X/5qzLHHswpZd7CfGKtL8ojm6MNabetkx2ZHOFm9
w0uVuSjIBOJ8WVvHEsVfrh3tRrSrt4L1amez/yzAiCPR08YHPMWRRpHSLQH2RSS/
CnjOm3P8B+TS2MTpvWbAQ+z17hDHVF85V0hPWhig1R22fdzauQZy1dZkmM9BJ+Jc
vy0vVtYrcN/5S5wTgDq91Sh7ygnuEN+wvRPQi/k8q/dg7NPESzd3o7bCmLL5dhkU
TghXNB4yPRn/PaJj7QEA4qKi32vmLFPhWzeLepyGty/cLW8VIOVylpnku6UjjWd1
pbo///3Cl8/fQqQBu5rY
=rbqn
-----END PGP SIGNATURE-----

[cooloutac]

How the heck did I not find this thread earlier.

I'm curious, did you have any untrusted vms affected? The only vms that were not affected for me, were all 4 of my untrusted ones. untrusted, dispvm, anon-whonix, and another untrusted.

Every single one of my other vms was affected.

[Reg Tiangha]

That has nothing to do with it. I've actually been dealing with this
issue for a while now (last December, really), but that's because I
compile my own kernels. I never reported it because I thought it was
just a problem on my end, but I now regret not doing so. It's the sort
of thing that wouldn't have shown up for the average Qubes user,
especially if they've done a fresh R3.2 install from scratch, because
the 4.4.55 kernel that was recently released is the 4th kernel in the
series, which would have only now triggered dnf to start uninstalling
older kernels, which is why most people are only noticing this now.

As for what VMs get affected, that's purely a coincidence on your end
(in my opinion). There is a pattern to the VMs that get affected but I
haven't totally nailed that down yet. I think part of it is if you've
been running nothing but the 'default' kernel, then when they were
created probably plays a factor. For example, if one VM was created when
4.4.11 was the main dom0 kernel, and another when the 4.4.31 kernel was
the main one, I think the VM that was created during the 4.4.11 era is
probably affected and it'll take a few more kernel upgrades before the
VM created during the 4.4.31 era gets hit too. But I'd need to do more
testing to see if that's really the case. I just haven't been paying
close attention to which VMs get hit when, and have just worked around
it as I've continued on in my kernel testing.

[cooloutac]

well at first I thought it had to do with the template but that wasn't the case. Then I also thought it had to do with what vms were newly created vs those I had restored from a previous backup, but I don't think thats the case either because I'm pretty sure my fedora clone and my debian clone were both restored from the same backup. and only my fedora clone template was affected. The other templates and the sys templates and dispvm, anon-whonix, and untrusted are indeed newly created though. I'm positive about that cause this is the type of thing that would happen in those vms and I remove them immediately. So I could be wrong...

[Joonas Lehtonen]

Marek Marczykowski-Górecki:

> On Tue, Apr 18, 2017 at 10:54:00AM +0000, Joonas Lehtonen wrote:
>
>
>> Joonas Lehtonen:
>>> Hi,
>>>
>>> just a quick notice about todays kernel update.
>>>
>>> After upgrading, the new kernel 4.4.55 became the new default for all
>>> VMs that previously used the default kernel, but
>>> VMs would no longer boot because they claim that an old kernel the one
>>> that got removed during the upgrade (4.4.11?) is no longer present even
>>> though the VM was configured to boot the default (4.4.55).
>
>> This was mainly an UI thing. qvm-ls -k displayed it correctly. These
>> affected VMs used to have the now-removed kernel version 4.4.14-11.
>> Qubes Manager just can not display not installed kernels.
>
> So, Qubes Manager shows still old kernel? Have you tried restarting it
> (Qubes Manager)?

Qubes Manager showed the new kernel (4.4.55-11) while qvm-ls -k showed
the old (removed) kernel.

Qubes Manager got restarted automatically during updating dom0.

[cooloutac]

I'm thinking more its what Reg said I have a felling all my affected vms were created at the same time. My debian clone is newer then my fedora clone. Obviously I don't back up the sys vms or untrusteds. and those were only ones not affected.

[Foppe de Haan]

this also seems to be affecting qvm-trim-template.