general protection fault in vmx_vcpu_run (2)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: aa2b8820 Add linux-next specific files for 20210205
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=13d27b54d00000
kernel config: https://syzkaller.appspot.com/x/.config?x=15c41e44a64aa1a5
dashboard link: https://syzkaller.appspot.com/bug?extid=42a71c84ef04577f1aef

Unfortunately, I don't have any reproducer for this issue yet.

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+42a71c...@syzkaller.appspotmail.com

general protection fault, probably for non-canonical address 0xdffffc0000001e26: 0000 [#1] PREEMPT SMP KASAN
KASAN: probably user-memory-access in range [0x000000000000f130-0x000000000000f137]
CPU: 0 PID: 18290 Comm: syz-executor.0 Not tainted 5.11.0-rc6-next-20210205-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:atomic_switch_perf_msrs arch/x86/kvm/vmx/vmx.c:6527 [inline]
RIP: 0010:vmx_vcpu_run+0x538/0x2740 arch/x86/kvm/vmx/vmx.c:6698
Code: 8a 55 00 39 eb 0f 8d fd 00 00 00 e8 42 85 55 00 48 8b 0c 24 48 63 c3 48 8d 04 40 48 8d 2c c1 48 8d 7d 08 48 89 f8 48 c1 e8 03 <42> 80 3c 38 00 0f 85 05 1d 00 00 48 8d 7d 10 4c 8b 6d 08 48 89 f8
RSP: 0018:ffffc9000238fb00 EFLAGS: 00010003
RAX: 0000000000001e26 RBX: 0000000000000000 RCX: 000000000000f12e
RDX: 0000000000040000 RSI: ffffffff811d679e RDI: 000000000000f136
RBP: 000000000000f12e R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff811d675e R11: 0000000000000000 R12: ffff88806d8ba4d0
R13: ffff88806d8ba520 R14: ffff88806d8b8000 R15: dffffc0000000000
FS: 00007f1a30eaf700(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1a30ece6b8 CR3: 000000001c387000 CR4: 00000000001526f0
Call Trace:
vcpu_enter_guest+0x103d/0x3f90 arch/x86/kvm/x86.c:9015
vcpu_run arch/x86/kvm/x86.c:9155 [inline]
kvm_arch_vcpu_ioctl_run+0x440/0x1980 arch/x86/kvm/x86.c:9382
kvm_vcpu_ioctl+0x467/0xd90 arch/x86/kvm/../../../virt/kvm/kvm_main.c:3283
vfs_ioctl fs/ioctl.c:48 [inline]
__do_sys_ioctl fs/ioctl.c:753 [inline]
__se_sys_ioctl fs/ioctl.c:739 [inline]
__x64_sys_ioctl+0x193/0x200 fs/ioctl.c:739
do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x465b09
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f1a30eaf188 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 000000000056c008 RCX: 0000000000465b09
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000007
RBP: 00000000004b069f R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056c008
R13: 00007ffde3d7a22f R14: 00007f1a30eaf300 R15: 0000000000022000
Modules linked in:
---[ end trace 7085899e9678fd16 ]---
RIP: 0010:atomic_switch_perf_msrs arch/x86/kvm/vmx/vmx.c:6527 [inline]
RIP: 0010:vmx_vcpu_run+0x538/0x2740 arch/x86/kvm/vmx/vmx.c:6698
Code: 8a 55 00 39 eb 0f 8d fd 00 00 00 e8 42 85 55 00 48 8b 0c 24 48 63 c3 48 8d 04 40 48 8d 2c c1 48 8d 7d 08 48 89 f8 48 c1 e8 03 <42> 80 3c 38 00 0f 85 05 1d 00 00 48 8d 7d 10 4c 8b 6d 08 48 89 f8
RSP: 0018:ffffc9000238fb00 EFLAGS: 00010003
RAX: 0000000000001e26 RBX: 0000000000000000 RCX: 000000000000f12e
RDX: 0000000000040000 RSI: ffffffff811d679e RDI: 000000000000f136
RBP: 000000000000f12e R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff811d675e R11: 0000000000000000 R12: ffff88806d8ba4d0
R13: ffff88806d8ba520 R14: ffff88806d8b8000 R15: dffffc0000000000
FS: 00007f1a30eaf700(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1a30ece6b8 CR3: 000000001c387000 CR4: 00000000001526f0


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: a99163e9 Merge tag 'devicetree-for-5.12' of git://git.kern..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=15cd357f500000
kernel config: https://syzkaller.appspot.com/x/.config?x=49116074dd53b631
dashboard link: https://syzkaller.appspot.com/bug?extid=42a71c84ef04577f1aef
compiler: Debian clang version 11.0.1-2
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=12c7f8a8d00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=137fc232d00000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+42a71c...@syzkaller.appspotmail.com

RBP: 0000000000402ed0 R08: 0000000000400488 R09: 0000000000400488
R10: 0000000000400488 R11: 0000000000000246 R12: 0000000000402f60
R13: 0000000000000000 R14: 00000000004ac018 R15: 0000000000400488
==================================================================
BUG: KASAN: global-out-of-bounds in atomic_switch_perf_msrs arch/x86/kvm/vmx/vmx.c:6604 [inline]
BUG: KASAN: global-out-of-bounds in vmx_vcpu_run+0x4f1/0x13f0 arch/x86/kvm/vmx/vmx.c:6771
Read of size 8 at addr ffffffff89a000e9 by task syz-executor198/8346

CPU: 0 PID: 8346 Comm: syz-executor198 Not tainted 5.11.0-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011

Call Trace:
__dump_stack lib/dump_stack.c:79 [inline]
dump_stack+0x125/0x19e lib/dump_stack.c:120
print_address_description+0x5f/0x3a0 mm/kasan/report.c:230
__kasan_report mm/kasan/report.c:396 [inline]
kasan_report+0x15e/0x200 mm/kasan/report.c:413
atomic_switch_perf_msrs arch/x86/kvm/vmx/vmx.c:6604 [inline]
vmx_vcpu_run+0x4f1/0x13f0 arch/x86/kvm/vmx/vmx.c:6771
vcpu_enter_guest+0x2ed9/0x8f10 arch/x86/kvm/x86.c:9074
vcpu_run+0x316/0xb70 arch/x86/kvm/x86.c:9225
kvm_arch_vcpu_ioctl_run+0x4e8/0xa40 arch/x86/kvm/x86.c:9453
kvm_vcpu_ioctl+0x62a/0xa30 arch/x86/kvm/../../../virt/kvm/kvm_main.c:3295

vfs_ioctl fs/ioctl.c:48 [inline]
__do_sys_ioctl fs/ioctl.c:753 [inline]

__se_sys_ioctl+0xfb/0x170 fs/ioctl.c:739
do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x43eee9
Code: 28 c3 e8 2a 14 00 00 66 2e 0f 1f 84 00 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 c0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffe7ad00d38 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000400488 RCX: 000000000043eee9
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000005
RBP: 0000000000402ed0 R08: 0000000000400488 R09: 0000000000400488
R10: 0000000000400488 R11: 0000000000000246 R12: 0000000000402f60
R13: 0000000000000000 R14: 00000000004ac018 R15: 0000000000400488

The buggy address belongs to the variable:
str__initcall__trace_system_name+0x9/0x40

Memory state around the buggy address:
ffffffff899fff80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffffffff89a00000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffffffff89a00080: 00 00 00 00 00 00 00 00 00 00 00 00 00 01 f9 f9
^
ffffffff89a00100: f9 f9 f9 f9 07 f9 f9 f9 f9 f9 f9 f9 00 03 f9 f9
ffffffff89a00180: f9 f9 f9 f9 00 06 f9 f9 f9 f9 f9 f9 00 00 00 00
==================================================================

[syzbot]

syzbot has bisected this issue to:

commit 167dcfc08b0b1f964ea95d410aa496fd78adf475
Author: Lorenzo Stoakes <lsto...@gmail.com>
Date: Tue Dec 15 20:56:41 2020 +0000

x86/mm: Increase pgt_buf size for 5-level page tables

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=13fe3ea8d00000
start commit: a99163e9 Merge tag 'devicetree-for-5.12' of git://git.kern..
git tree: upstream
final oops: https://syzkaller.appspot.com/x/report.txt?x=10013ea8d00000
console output: https://syzkaller.appspot.com/x/log.txt?x=17fe3ea8d00000

kernel config: https://syzkaller.appspot.com/x/.config?x=49116074dd53b631
dashboard link: https://syzkaller.appspot.com/bug?extid=42a71c84ef04577f1aef

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=141f3f04d00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=17de4f12d00000

Reported-by: syzbot+42a71c...@syzkaller.appspotmail.com
Fixes: 167dcfc08b0b ("x86/mm: Increase pgt_buf size for 5-level page tables")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[Borislav Petkov]

On Tue, Feb 23, 2021 at 03:17:07PM -0800, syzbot wrote:
> syzbot has bisected this issue to:
>
> commit 167dcfc08b0b1f964ea95d410aa496fd78adf475
> Author: Lorenzo Stoakes <lsto...@gmail.com>
> Date: Tue Dec 15 20:56:41 2020 +0000
>
> x86/mm: Increase pgt_buf size for 5-level page tables
>
> bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=13fe3ea8d00000
> start commit: a99163e9 Merge tag 'devicetree-for-5.12' of git://git.kern..
> git tree: upstream
> final oops: https://syzkaller.appspot.com/x/report.txt?x=10013ea8d00000

No oops here.

> console output: https://syzkaller.appspot.com/x/log.txt?x=17fe3ea8d00000

Nothing special here too.

> kernel config: https://syzkaller.appspot.com/x/.config?x=49116074dd53b631

Tried this on two boxes, the Intel one doesn't even boot with that
config - and it is pretty standard one - and on the AMD one the
reproducer doesn't trigger anything. It probably won't because the GP
is in vmx_vcpu_run() but since the ioctls were doing something with
IRQCHIP, I thought it is probably vendor-agnostic.

So, all in all, I could use some more info on how you're reproducing and
maybe you could show the oops too.

Thx.

--
Regards/Gruss,
Boris.

https://people.kernel.org/tglx/notes-about-netiquette

[Dmitry Vyukov]

Hi Boris,

Looking at the bisection log, the bisection was distracted by something else.
You can always find the original reported issue over the dashboard link:
https://syzkaller.appspot.com/bug?extid=42a71c84ef04577f1aef
or on lore:
https://lore.kernel.org/lkml/0000000000007f...@google.com/

[Borislav Petkov]

Hi Dmitry,

On Wed, Feb 24, 2021 at 06:12:57PM +0100, Dmitry Vyukov wrote:
> Looking at the bisection log, the bisection was distracted by something else.

Meaning the bisection result:

167dcfc08b0b ("x86/mm: Increase pgt_buf size for 5-level page tables")

is bogus?

> You can always find the original reported issue over the dashboard link:
> https://syzkaller.appspot.com/bug?extid=42a71c84ef04577f1aef
> or on lore:
> https://lore.kernel.org/lkml/0000000000007f...@google.com/

Ok, so this looks like this is trying to run kvm ioctls *in* a guest,
i.e., nested. Right?

[Sean Christopherson]

On Wed, Feb 24, 2021, Borislav Petkov wrote:
> Hi Dmitry,
>
> On Wed, Feb 24, 2021 at 06:12:57PM +0100, Dmitry Vyukov wrote:
> > Looking at the bisection log, the bisection was distracted by something else.
>
> Meaning the bisection result:
>
> 167dcfc08b0b ("x86/mm: Increase pgt_buf size for 5-level page tables")
>
> is bogus?

Ya, looks 100% bogus.

> > You can always find the original reported issue over the dashboard link:
> > https://syzkaller.appspot.com/bug?extid=42a71c84ef04577f1aef
> > or on lore:
> > https://lore.kernel.org/lkml/0000000000007f...@google.com/
>
> Ok, so this looks like this is trying to run kvm ioctls *in* a guest,
> i.e., nested. Right?

Yep. I tried to run the reproducer yesterday, but the kernel config wouldn't
boot my VM. I haven't had time to dig in. Anyways, I think you can safely
assume this is a KVM issue unless more data comes along that says otherwise.

[Dmitry Vyukov]

On Wed, Feb 24, 2021 at 6:49 PM Borislav Petkov <b...@alien8.de> wrote:
>
> Hi Dmitry,
>
> On Wed, Feb 24, 2021 at 06:12:57PM +0100, Dmitry Vyukov wrote:
> > Looking at the bisection log, the bisection was distracted by something else.
>
> Meaning the bisection result:
>
> 167dcfc08b0b ("x86/mm: Increase pgt_buf size for 5-level page tables")
>
> is bogus?
>
> > You can always find the original reported issue over the dashboard link:
> > https://syzkaller.appspot.com/bug?extid=42a71c84ef04577f1aef
> > or on lore:
> > https://lore.kernel.org/lkml/0000000000007f...@google.com/
>
> Ok, so this looks like this is trying to run kvm ioctls *in* a guest,
> i.e., nested. Right?

Yes, testing happens in VM. But the kernel that crashes is the one
that receives the ioctls.

[Dmitry Vyukov]

Interesting. What happens? Does the kernel crash? Userspace crash?
Rootfs is not mounted? Or something else?

[Sean Christopherson]

Not sure, it ended up in the EFI shell instead of the kernel (running with QEMU's
-kernel). My QEMU+KVM setup does a variety of shenanigans, I'm guessing it's an
incompatibility in my setup.

[Sean Christopherson]

I haven't been able to reproduce this, and based on the super simple reproducer
and the fact that AFAICT this hasn't been hit in 2+ years, I suspect whatever
was broken has long since been fixed.

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
SYZFAIL: wrong response packet

Warning: Permanently added '10.128.0.15' (ECDSA) to the list of known hosts.
2023/07/10 22:43:25 ignoring optional flag "sandboxArg"="0"
2023/07/10 22:43:25 parsed 1 programs
2023/07/10 22:43:25 executed programs: 0
2023/07/10 22:43:28 result: hanged=false err=executor 0: failed to write control pipe: write |1: broken pipe
SYZFAIL: wrong response packet
(errno 16: Device or resource busy)
loop exited with status 67
2023/07/10 22:43:30 executed programs: 2
2023/07/10 22:43:35 executed programs: 55


Tested on:

commit: 3f01e9fe Merge tag 'linux-watchdog-6.5-rc2' of git://w..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=15ed6c5aa80000
kernel config: https://syzkaller.appspot.com/x/.config?x=5837d74dc9cc112b
dashboard link: https://syzkaller.appspot.com/bug?extid=42a71c84ef04577f1aef
compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2

Note: no patches were applied.

[Sean Christopherson]

On Mon, Jul 10, 2023, syzbot wrote:
> Hello,
>
> syzbot has tested the proposed patch but the reproducer is still triggering an issue:
> SYZFAIL: wrong response packet

Heh, well that wasn't helpful. I'm going to close this, worst case scenario
syzbot will provide a fresh new reproducer.

#syz invalid