Bug#1020787: linux-image-5.19.0-2-amd64: After updating to 5.19 kernel the VMs are started without XSAVE CPU flags

[patrick]

Package: src:linux
Version: 5.19.11-1
Severity: important

Dear Maintainer,

I have a XEN hypervisor running with some VMs (everything based on Debian). After updating to the Kernel to 5.19 (from 5.18.0-4), some
binaries are falling to execute. If the VMs are started with Kernel 5.18.0-4 they are running fine.
One example is wget which results in this kernel log:
[ 366.744374] traps: wget[3327] trap invalid opcode ip:7fcabc94fb0a sp:7fffdd212c80 error:0 in libgnutls.so.30.34.1[7fcabc837000+12f000]

After googling a little I found this stackoverflow problem report:
https://stackoverflow.com/questions/65676095/illegal-instruction-core-dumped-after-packages-upgrade-on-centos-8

And indeed there is some difference in /proc/cpuinfo:
5.19:
flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp fsgsbase bmi1 bmi2 erms rdseed adx clflushopt md_clear

5.18:
flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp fsgsbase bmi1 avx2 bmi2 erms rdseed adx clflushopt xsaveopt xsavec xgetbv1 md_clear

So the flags for "fma xsave avx2 bmi2 xsaveopt xsavec xgetbv1 md_clear" are missing, which might result in gnutls failures.

regards

Patrick


-- Package-specific info:
** Version:
Linux version 5.19.0-2-amd64 (debian...@lists.debian.org) (gcc-11 (Debian 11.3.0-6) 11.3.0, GNU ld (GNU Binutils for Debian) 2.38.90.20220713) #1 SMP PREEMPT_DYNAMIC Debian 5.19.11-1 (2022-09-24)

** Command line:
root=/dev/xvda1 lockd.nlm_tcpport=61053 lockd.nlm_udpport=61053 ipv6.disable=1 net.ifnames=0 xen_blkfront.max_queues=3

** Tainted: W (512)
* kernel issued warning

** Kernel log:
[ 366.744374] traps: wget[3327] trap invalid opcode ip:7fcabc94fb0a sp:7fffdd212c80 error:0 in libgnutls.so.30.34.1[7fcabc837000+12f000]

** Model information

** Loaded modules:
tls
nft_chain_nat
xt_MASQUERADE
nf_nat
ipt_REJECT
nf_reject_ipv4
xt_owner
xt_mac
xt_conntrack
nf_conntrack
nf_defrag_ipv6
nf_defrag_ipv4
xt_multiport
nft_limit
xt_tcpudp
xt_LOG
nf_log_syslog
xt_limit
nft_compat
nf_tables
libcrc32c
nfnetlink
sunrpc
binfmt_misc
intel_rapl_msr
intel_rapl_common
intel_pmc_core_pltdrv
intel_pmc_core
ghash_clmulni_intel
aesni_intel
crypto_simd
cryptd
cfg80211
rfkill
evdev
pcspkr
drm
fuse
configfs
ip_tables
x_tables
autofs4
ext4
crc16
mbcache
jbd2
crc32c_generic
tun
xen_netfront
xen_blkfront
crct10dif_pclmul
crct10dif_common
crc32_pclmul
crc32c_intel

** PCI devices:

** USB devices:
not available


-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.19.0-2-amd64 (SMP w/4 CPU threads; PREEMPT)
Kernel taint flags: TAINT_WARN
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages linux-image-5.19.0-2-amd64 depends on:
ii initramfs-tools [linux-initramfs-tool] 0.142
ii kmod 30+20220630-3
ii linux-base 4.9

Versions of packages linux-image-5.19.0-2-amd64 recommends:
ii apparmor 3.0.7-1
ii firmware-linux-free 20200122-1

Versions of packages linux-image-5.19.0-2-amd64 suggests:
pn debian-kernel-handbook <none>
pn grub-pc | grub-efi-amd64 | extlinux <none>
pn linux-doc-5.19 <none>

Versions of packages linux-image-5.19.0-2-amd64 is related to:
ii firmware-amd-graphics 20210818-1
pn firmware-atheros <none>
pn firmware-bnx2 <none>
pn firmware-bnx2x <none>
pn firmware-brcm80211 <none>
pn firmware-cavium <none>
pn firmware-intel-sound <none>
pn firmware-intelwimax <none>
pn firmware-ipw2x00 <none>
pn firmware-ivtv <none>
pn firmware-iwlwifi <none>
pn firmware-libertas <none>
ii firmware-linux-nonfree 20210818-1
ii firmware-misc-nonfree 20210818-1
pn firmware-myricom <none>
pn firmware-netxen <none>
pn firmware-qlogic <none>
pn firmware-realtek <none>
pn firmware-samsung <none>
pn firmware-siano <none>
pn firmware-ti-connectivity <none>
pn xen-hypervisor <none>

-- no debconf information

[Debian Bug Tracking System]

Processing control commands:

> tag -1 moreinfo
Bug #1020787 [src:linux] linux-image-5.19.0-2-amd64: After updating to 5.19 kernel the VMs are started without XSAVE CPU flags
Added tag(s) moreinfo.

--
1020787: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1020787
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

[Diederik de Haas]

Control: tag -1 moreinfo

On Monday, 26 September 2022 20:04:10 CEST patrick wrote:
> I have a XEN hypervisor running with some VMs (everything based on Debian).

Which version of Xen are you using?

> After updating to the Kernel to 5.19 (from 5.18.0-4), some binaries are
> falling to execute. If the VMs are started with Kernel 5.18.0-4 they are
> running fine.

Is this all about the dom0 kernel or is it all/some about using 5.19 as
domU kernel? Are the issues happing on dom0 or inside domU?

If the issues happen inside (a) domU, can you share a (minimal) domU
configuration file so it becomes easier to replicate?

> And indeed there is some difference in /proc/cpuinfo:

> The flags for "fma xsave avx2 bmi2 xsaveopt xsavec xgetbv1 md_clear" are

> missing, which might result in gnutls failures.

In kernel 5.19 the following commits were added under ``arch/x86/kernel/fpu/``:

b91c0922bf1ed15b67a6faa404bc64e3ed532ec2 x86/fpu: Cleanup variable shadowing
8ad7e8f696951f192c6629a0cbda9ac94c773159 x86/fpu/xsave: Support XSAVEC in the kernel
f5c0b4f30416c670408a77be94703d04d22b57df x86/prctl: Remove pointless task argument

Of these, the first 2 seem like possible candidates that caused the issue.
https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s4.2.2
describes a way to apply a simple patch to a kernel.
What you could try, is creating a patch from reverting one of the earlier
mentioned commits and use that with 'test-patches'.

It's probably also useful to know what CPU(s) are in the machine (dom0).

On Monday, 26 September 2022 20:31:17 CEST Ps Ps wrote:
> On Xen Hypervisor I just found this logs:

So this is on dom0? In which log file did you find it?

Generally: be as specific as you can be and describe *exactly* what you did
and the exact results (if any). Also try to make it as easy as possible for
others to reproduce what you're experiencing.

FTR: I did not see this issue on my dom0 (Xen 4.16.2-1; kernel 5.19.11-1):

root@dom0:~# dmesg
[ 0.000000] Linux version 5.19.0-2-amd64 (debian...@lists.debian.org) (gcc-11 (Debian 11.3.0-6) 11.3.0, GNU ld (GNU Binutils for Debian) 2.38.90.20220713) #1 SMP PREEMPT_DYNAMIC Debian 5.19.11-1 (2022-09-24)
[ 0.000000] Command line: placeholder root=UUID=8008723b-668f-43f6-b432-8c56ed53f48a ro quiet net.ifnames=0
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[ 0.000000] signal: max sigframe size: 1776
[ 0.000000] Released 0 page(s)

root@dom0:~# grep flag /proc/cpuinfo | uniq
flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp fsgsbase bmi1 avx2 bmi2 erms rtm rdseed adx xsaveopt md_clear


Also found this patch which should make the error msg more informative ...
https://lore.kernel.org/all/20220810221909.127...@citrix.com/
Even though I haven't experienced it (yet?), the language of this patch
seems to indicate you're not alone with it.

[Diederik de Haas]

Hi Marek,

On dinsdag 27 september 2022 02:16:31 CEST Marek Marczykowski-Górecki wrote:

> On Tue, Sep 27, 2022 at 01:39:46AM +0200, Diederik de Haas wrote:
> > In kernel 5.19 the following commits were added under
> > ``arch/x86/kernel/fpu/``:
> >
> > b91c0922bf1ed15b67a6faa404bc64e3ed532ec2 x86/fpu: Cleanup variable
> > shadowing
> > 8ad7e8f696951f192c6629a0cbda9ac94c773159 x86/fpu/xsave: Support
> > XSAVEC in the kernel
> > f5c0b4f30416c670408a77be94703d04d22b57df x86/prctl:
> > Remove pointless task argument

8ad7e8f696951f192c6629a0cbda9ac94c773159 seems to be the 'offending' commit ...

> > Also found this patch which should make the error msg more informative ...

> > https://lore.kernel.org/all/20220810221909.12768-1-andrew.cooper3@citrix.c

> > om/ Even though I haven't experienced it (yet?), the language of this
> > patch seems to indicate you're not alone with it.
>

> There is also Xen patch fixing the issue:
> https://lore.kernel.org/xen-devel/a4ec41e6-16cd-4452-19c1-5d6d9e3bddf8@suse.
> com/

http://xenbits.xen.org/gitweb/?
p=xen.git;a=commit;h=c3bd0b83ea5b7c0da6542687436042eeea1e7909 is where it's
committed in Xen's master branch. I haven't seen a backport to 4.16 (yet?)

Looking at the patch thread you linked, I get the impression that the change
in the Linux kernel exposed the problem on the Xen side, but the Linux kernel
change didn't introduce a bug (Andrew's patch seems to make it more
informative, not correct a bug).

But I'm not sure that assesment is correct. What's your take on this?

Cheers,
Diederik

[Andy Smith]

Hi,

On Tue, Sep 27, 2022 at 01:59:42PM +0200, Diederik de Haas wrote:
> http://xenbits.xen.org/gitweb/?
> p=xen.git;a=commit;h=c3bd0b83ea5b7c0da6542687436042eeea1e7909 is where it's
> committed in Xen's master branch. I haven't seen a backport to 4.16 (yet?)

Does this mean that all versions of hypervisor need this patch
backported in order to support Linux kernel 5.19+ as dom0 or domU?

Cheers,
Andy

[Diederik de Haas]

Control: reassign -1 src:xen
Control: retitle -1 Xen: After updating to 5.19 kernel the VMs are started without XSAVE CPU flags
Control: tag -1 -moreinfo patch upstream fixed-upstream

On dinsdag 27 september 2022 14:05:59 CEST Marek Marczykowski-Górecki wrote:
> On Tue, Sep 27, 2022 at 01:59:42PM +0200, Diederik de Haas wrote:

> > On Tue, Sep 27, 2022 at 01:39:46AM +0200, Diederik de Haas wrote:
> > > In kernel 5.19 the following commits were added under
> > > ``arch/x86/kernel/fpu/``:
> > >
> > > b91c0922bf1ed15b67a6faa404bc64e3ed532ec2 x86/fpu: Cleanup variable
> > > shadowing
> > > 8ad7e8f696951f192c6629a0cbda9ac94c773159 x86/fpu/xsave: Support
> > > XSAVEC in the kernel
> > > f5c0b4f30416c670408a77be94703d04d22b57df x86/prctl:
> > > Remove pointless task argument
> >
> > 8ad7e8f696951f192c6629a0cbda9ac94c773159 seems to be the 'offending' commit
> >

> > http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;

> > h=c3bd0b83ea5b7c0da6542687436042eeea1e7909 is where it's committed
> > in Xen's master branch. I haven't seen a backport to 4.16 (yet?)

This would be the patch, but it needs to be backported.

> > Looking at the patch thread you linked, I get the impression that the
> > change in the Linux kernel exposed the problem on the Xen side, but the
> > Linux kernel change didn't introduce a bug (Andrew's patch seems to make
> > it more informative, not correct a bug).
> >
> > But I'm not sure that assesment is correct. What's your take on this?
>

> Yes, that seems correct.

Thanks, reassigning accordingly.

On dinsdag 27 september 2022 14:20:29 CEST Marek Marczykowski-Górecki wrote:

> On Tue, Sep 27, 2022 at 12:08:58PM +0000, Andy Smith wrote:
> > On Tue, Sep 27, 2022 at 01:59:42PM +0200, Diederik de Haas wrote:

> > > http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;

> > > h=c3bd0b83ea5b7c0da6542687436042eeea1e7909 is where it's committed
> > > in Xen's master branch. I haven't seen a backport to 4.16 (yet?)
> >
> > Does this mean that all versions of hypervisor need this patch
> > backported in order to support Linux kernel 5.19+ as dom0 or domU?
>

> Yes, I think so. I've seen the issue on Xen 4.14 too.

Agreed.

AFAIC, the question that remains is about its impact.
On my Xen server I did not see the mentioned problem, so it looks like it is
hardware dependent? (I did not do an extensive test though)

And for systems that are affected is how bad the consequences are. What I've
seen in this bug report is an error message in the log wrt gnutls. I don't know
whether this makes every application using gnutls not working at all anymore.
I also don't know whether other applications and how many are affected and to
what extend.
Marek and Patrick, can you shed some light on this?

FTR: it 'still' is a bug that should be fixed; the question is about urgency.

[Debian Bug Tracking System]

Processing control commands:

> reassign -1 src:xen
Bug #1020787 [src:linux] linux-image-5.19.0-2-amd64: After updating to 5.19 kernel the VMs are started without XSAVE CPU flags
Bug reassigned from package 'src:linux' to 'src:xen'.
No longer marked as found in versions linux/5.19.11-1.
Ignoring request to alter fixed versions of bug #1020787 to the same values previously set

> retitle -1 Xen: After updating to 5.19 kernel the VMs are started without XSAVE CPU flags

Bug #1020787 [src:xen] linux-image-5.19.0-2-amd64: After updating to 5.19 kernel the VMs are started without XSAVE CPU flags
Changed Bug title to 'Xen: After updating to 5.19 kernel the VMs are started without XSAVE CPU flags' from 'linux-image-5.19.0-2-amd64: After updating to 5.19 kernel the VMs are started without XSAVE CPU flags'.

> tag -1 -moreinfo patch upstream fixed-upstream

Bug #1020787 [src:xen] Xen: After updating to 5.19 kernel the VMs are started without XSAVE CPU flags
Removed tag(s) moreinfo.