[syzbot] [kvm?] WARNING in kvm_tdp_mmu_invalidate_all_roots

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: d3e1ee0e67e7 Add linux-next specific files for 20230421
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=16ac3de0280000
kernel config: https://syzkaller.appspot.com/x/.config?x=53c789efbcc06cf6
dashboard link: https://syzkaller.appspot.com/bug?extid=094a74444165dbcd3a54
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/c558a9e1fe6a/disk-d3e1ee0e.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/2ec100a34c4c/vmlinux-d3e1ee0e.xz
kernel image: https://storage.googleapis.com/syzbot-assets/1afcd9936dc1/bzImage-d3e1ee0e.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+094a74...@syzkaller.appspotmail.com

------------[ cut here ]------------
WARNING: CPU: 0 PID: 12623 at arch/x86/kvm/mmu/tdp_mmu.c:943 kvm_tdp_mmu_invalidate_all_roots+0x2bd/0x370 arch/x86/kvm/mmu/tdp_mmu.c:943
Modules linked in:
CPU: 0 PID: 12623 Comm: syz-executor.3 Not tainted 6.3.0-rc7-next-20230421-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/30/2023
RIP: 0010:kvm_tdp_mmu_invalidate_all_roots+0x2bd/0x370 arch/x86/kvm/mmu/tdp_mmu.c:943
Code: 00 e8 a7 67 64 00 49 8d 7c 24 18 31 f6 e8 eb 23 ec 08 31 ff 89 c3 89 c6 e8 b0 63 64 00 85 db 0f 85 00 fe ff ff e8 83 67 64 00 <0f> 0b e9 f4 fd ff ff 48 89 ef e8 94 46 b7 00 e9 88 fd ff ff e8 4a
RSP: 0018:ffffc90005f7fc80 EFLAGS: 00010212
RAX: 0000000000015eaf RBX: 0000000000000000 RCX: ffffc9000e5cd000
RDX: 0000000000040000 RSI: ffffffff811f7dcd RDI: 0000000000000005
RBP: ffffc900062ba788 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900062b1000
R13: 0000000000000000 R14: 0000000000000006 R15: ffffc900062b19d8
FS: 00007f9c99c7c700(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa626bad988 CR3: 000000001cd68000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
kvm_mmu_uninit_tdp_mmu+0x16/0x100 arch/x86/kvm/mmu/tdp_mmu.c:48
kvm_mmu_uninit_vm+0x6a/0x70 arch/x86/kvm/mmu/mmu.c:6239
kvm_arch_destroy_vm+0x369/0x490 arch/x86/kvm/x86.c:12465
kvm_create_vm arch/x86/kvm/../../../virt/kvm/kvm_main.c:1245 [inline]
kvm_dev_ioctl_create_vm arch/x86/kvm/../../../virt/kvm/kvm_main.c:5017 [inline]
kvm_dev_ioctl+0x11be/0x1bb0 arch/x86/kvm/../../../virt/kvm/kvm_main.c:5059
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:870 [inline]
__se_sys_ioctl fs/ioctl.c:856 [inline]
__x64_sys_ioctl+0x197/0x210 fs/ioctl.c:856
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f9c98e8c169
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 f1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f9c99c7c168 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f9c98fac120 RCX: 00007f9c98e8c169
RDX: 0000000000000000 RSI: 000000000000ae01 RDI: 0000000000000005
RBP: 00007f9c98ee7ca1 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007fff0f7a772f R14: 00007f9c99c7c300 R15: 0000000000022000
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[Sean Christopherson]

On Fri, Apr 21, 2023, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: d3e1ee0e67e7 Add linux-next specific files for 20230421
> git tree: linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=16ac3de0280000
> kernel config: https://syzkaller.appspot.com/x/.config?x=53c789efbcc06cf6
> dashboard link: https://syzkaller.appspot.com/bug?extid=094a74444165dbcd3a54
> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/c558a9e1fe6a/disk-d3e1ee0e.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/2ec100a34c4c/vmlinux-d3e1ee0e.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/1afcd9936dc1/bzImage-d3e1ee0e.xz
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+094a74...@syzkaller.appspotmail.com
>
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 12623 at arch/x86/kvm/mmu/tdp_mmu.c:943 kvm_tdp_mmu_invalidate_all_roots+0x2bd/0x370 arch/x86/kvm/mmu/tdp_mmu.c:943
> Modules linked in:
> CPU: 0 PID: 12623 Comm: syz-executor.3 Not tainted 6.3.0-rc7-next-20230421-syzkaller #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/30/2023
> RIP: 0010:kvm_tdp_mmu_invalidate_all_roots+0x2bd/0x370 arch/x86/kvm/mmu/tdp_mmu.c:943

> Call Trace:
> <TASK>
> kvm_mmu_uninit_tdp_mmu+0x16/0x100 arch/x86/kvm/mmu/tdp_mmu.c:48
> kvm_mmu_uninit_vm+0x6a/0x70 arch/x86/kvm/mmu/mmu.c:6239
> kvm_arch_destroy_vm+0x369/0x490 arch/x86/kvm/x86.c:12465
> kvm_create_vm arch/x86/kvm/../../../virt/kvm/kvm_main.c:1245 [inline]
> kvm_dev_ioctl_create_vm arch/x86/kvm/../../../virt/kvm/kvm_main.c:5017 [inline]
> kvm_dev_ioctl+0x11be/0x1bb0 arch/x86/kvm/../../../virt/kvm/kvm_main.c:5059
> vfs_ioctl fs/ioctl.c:51 [inline]
> __do_sys_ioctl fs/ioctl.c:870 [inline]
> __se_sys_ioctl fs/ioctl.c:856 [inline]
> __x64_sys_ioctl+0x197/0x210 fs/ioctl.c:856
> do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
> entry_SYSCALL_64_after_hwframe+0x63/0xcd

Gah, kvm->users_count is elevated when the VM is destroyed if creation fails,
but mmu_lock isn't held in that case, and lockdep yells. This particular bug is
effectively a false positive, nothing else holds a reference to the VM.

However, syzbot found another bug that _is_ a real problem (hasn't been reported
upstream yet for whatever reason). Not holding mmu_lock during "real" VM destruction
is problematic because walking the list can race with the asynchronous worker
deleting and freeing from the list.

Posted a fixed version[*] that should resolve both issues by protecting the walk
with RCU.

[*] https://lore.kernel.org/all/20230421214946....@google.com

==================================================================
BUG: KASAN: slab-use-after-free in kvm_tdp_mmu_invalidate_all_roots+0x2e3/0x370 arch/x86/kvm/mmu/tdp_mmu.c:945
Read of size 8 at addr ffff88801c8cf948 by task syz-executor.1/17012

CPU: 0 PID: 17012 Comm: syz-executor.1 Not tainted 6.3.0-rc7-next-20230421-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/30/2023

Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0xd9/0x150 lib/dump_stack.c:106
print_address_description.constprop.0+0x2c/0x3c0 mm/kasan/report.c:351
print_report mm/kasan/report.c:462 [inline]
kasan_report+0x11c/0x130 mm/kasan/report.c:572
kvm_tdp_mmu_invalidate_all_roots+0x2e3/0x370 arch/x86/kvm/mmu/tdp_mmu.c:945

kvm_mmu_uninit_tdp_mmu+0x16/0x100 arch/x86/kvm/mmu/tdp_mmu.c:48
kvm_mmu_uninit_vm+0x6a/0x70 arch/x86/kvm/mmu/mmu.c:6239
kvm_arch_destroy_vm+0x369/0x490 arch/x86/kvm/x86.c:12465

kvm_destroy_vm arch/x86/kvm/../../../virt/kvm/kvm_main.c:1313 [inline]
kvm_put_kvm+0x4da/0xae0 arch/x86/kvm/../../../virt/kvm/kvm_main.c:1347
kvm_vcpu_release+0x51/0x70 arch/x86/kvm/../../../virt/kvm/kvm_main.c:3846
__fput+0x27c/0xa90 fs/file_table.c:321
task_work_run+0x16f/0x270 kernel/task_work.c:179
resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
exit_to_user_mode_prepare+0x210/0x240 kernel/entry/common.c:204
__syscall_exit_to_user_mode_work kernel/entry/common.c:286 [inline]
syscall_exit_to_user_mode+0x1d/0x50 kernel/entry/common.c:297
do_syscall_64+0x46/0xb0 arch/x86/entry/common.c:86
entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7fd7a583e01b
Code: 0f 05 48 3d 00 f0 ff ff 77 45 c3 0f 1f 40 00 48 83 ec 18 89 7c 24 0c e8 63 fc ff ff 8b 7c 24 0c 41 89 c0 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 0c e8 a1 fc ff ff 8b 44
RSP: 002b:00007ffc821a2fb0 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
RAX: 0000000000000000 RBX: 000000000000000a RCX: 00007fd7a583e01b
RDX: 0000000000000000 RSI: 00007fd7a5400000 RDI: 0000000000000009
RBP: 00007fd7a59ad980 R08: 0000000000000000 R09: 0000000071612a4e
R10: 0000000000000000 R11: 0000000000000293 R12: 000000000014ffb4
R13: 00007ffc821a30b0 R14: 00007fd7a59ac050 R15: 0000000000000032
</TASK>

[Sean Christopherson]

On Fri, Apr 21, 2023, syzbot wrote:

> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: d3e1ee0e67e7 Add linux-next specific files for 20230421
> git tree: linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=16ac3de0280000
> kernel config: https://syzkaller.appspot.com/x/.config?x=53c789efbcc06cf6
> dashboard link: https://syzkaller.appspot.com/bug?extid=094a74444165dbcd3a54
> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/c558a9e1fe6a/disk-d3e1ee0e.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/2ec100a34c4c/vmlinux-d3e1ee0e.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/1afcd9936dc1/bzImage-d3e1ee0e.xz
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+094a74...@syzkaller.appspotmail.com

Time to see if syzbot gets confused by a fix that was squashed into the guilty
commit.

#syz fix: KVM: x86: Preserve TDP MMU roots until they are explicitly invalidated

[Dmitry Vyukov]

It was staged in the moderation queue for manual moderation:
https://github.com/google/syzkaller/blob/master/docs/syzbot.md#moderation-queue

here:
https://groups.google.com/g/syzkaller-upstream-moderation/c/pTGA87FeSIE

I've pushed it upstream since you say it's worth fixing.

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/ZEMFpqxvOHd2kZiu%40google.com.

[Sean Christopherson]

Ah, thanks! For future reference, does "#syz upstream" also work from our internal
bug tracker, or do I have to wait for the report to hit the moderation queue first?