[syzbot] [io-uring?] general protection fault in io_sqpoll_wq_cpu_affinity

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 626932085009 Add linux-next specific files for 20230825
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=12a97797a80000
kernel config: https://syzkaller.appspot.com/x/.config?x=8a8c992a790e5073
dashboard link: https://syzkaller.appspot.com/bug?extid=c74fea926a78b8a91042
compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/46ec18b3c2fb/disk-62693208.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/b4ea0cb78498/vmlinux-62693208.xz
kernel image: https://storage.googleapis.com/syzbot-assets/5fb3938c7272/bzImage-62693208.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+c74fea...@syzkaller.appspotmail.com

general protection fault, probably for non-canonical address 0xdffffc000000011d: 0000 [#1] PREEMPT SMP KASAN
KASAN: null-ptr-deref in range [0x00000000000008e8-0x00000000000008ef]
CPU: 1 PID: 27342 Comm: syz-executor.5 Not tainted 6.5.0-rc7-next-20230825-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023
RIP: 0010:io_sqpoll_wq_cpu_affinity+0x8c/0xe0 io_uring/sqpoll.c:433
Code: 48 c1 ea 03 80 3c 02 00 75 64 4c 8b a3 a8 00 00 00 48 b8 00 00 00 00 00 fc ff df 49 8d bc 24 e8 08 00 00 48 89 fa 48 c1 ea 03 <80> 3c 02 00 75 37 49 8b bc 24 e8 08 00 00 48 89 ee e8 7e ac 02 00
RSP: 0018:ffffc900051dfe00 EFLAGS: 00010206
RAX: dffffc0000000000 RBX: ffff888056c72400 RCX: 0000000000000000
RDX: 000000000000011d RSI: ffffffff841f12be RDI: 00000000000008e8
RBP: ffffc900051dfec8 R08: 0000000000000000 R09: ffffed100ad8e482
R10: ffffc900051dfde8 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000011 R15: ffffc900051dfec8
FS: 00007f1f3e2bc6c0(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffe2c9b3010 CR3: 0000000029652000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
__io_register_iowq_aff io_uring/io_uring.c:4207 [inline]
__io_register_iowq_aff+0xa7/0xe0 io_uring/io_uring.c:4198
io_register_iowq_aff io_uring/io_uring.c:4240 [inline]
__io_uring_register io_uring/io_uring.c:4447 [inline]
__do_sys_io_uring_register+0xf58/0x2250 io_uring/io_uring.c:4539
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x38/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f1f3d47cae9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f1f3e2bc0c8 EFLAGS: 00000246 ORIG_RAX: 00000000000001ab
RAX: ffffffffffffffda RBX: 00007f1f3d59bf80 RCX: 00007f1f3d47cae9
RDX: 0000000020000140 RSI: 0000000000000011 RDI: 0000000000000003
RBP: 00007f1f3d4c847a R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f1f3d59bf80 R15: 00007fffb91ba9f8
</TASK>
Modules linked in:
---[ end trace 0000000000000000 ]---
RIP: 0010:io_sqpoll_wq_cpu_affinity+0x8c/0xe0 io_uring/sqpoll.c:433
Code: 48 c1 ea 03 80 3c 02 00 75 64 4c 8b a3 a8 00 00 00 48 b8 00 00 00 00 00 fc ff df 49 8d bc 24 e8 08 00 00 48 89 fa 48 c1 ea 03 <80> 3c 02 00 75 37 49 8b bc 24 e8 08 00 00 48 89 ee e8 7e ac 02 00
RSP: 0018:ffffc900051dfe00 EFLAGS: 00010206
RAX: dffffc0000000000 RBX: ffff888056c72400 RCX: 0000000000000000
RDX: 000000000000011d RSI: ffffffff841f12be RDI: 00000000000008e8
RBP: ffffc900051dfec8 R08: 0000000000000000 R09: ffffed100ad8e482
R10: ffffc900051dfde8 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000011 R15: ffffc900051dfec8
FS: 00007f1f3e2bc6c0(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6671b75198 CR3: 0000000029652000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
----------------
Code disassembly (best guess):
0: 48 c1 ea 03 shr $0x3,%rdx
4: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
8: 75 64 jne 0x6e
a: 4c 8b a3 a8 00 00 00 mov 0xa8(%rbx),%r12
11: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
18: fc ff df
1b: 49 8d bc 24 e8 08 00 lea 0x8e8(%r12),%rdi
22: 00
23: 48 89 fa mov %rdi,%rdx
26: 48 c1 ea 03 shr $0x3,%rdx
* 2a: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1) <-- trapping instruction
2e: 75 37 jne 0x67
30: 49 8b bc 24 e8 08 00 mov 0x8e8(%r12),%rdi
37: 00
38: 48 89 ee mov %rbp,%rsi
3b: e8 7e ac 02 00 call 0x2acbe


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the bug is already fixed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite bug's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the bug is a duplicate of another bug, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[Gabriel Krisman Bertazi]

syzbot <syzbot+c74fea...@syzkaller.appspotmail.com> writes:

> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: 626932085009 Add linux-next specific files for 20230825
> git tree: linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=12a97797a80000
> kernel config: https://syzkaller.appspot.com/x/.config?x=8a8c992a790e5073
> dashboard link: https://syzkaller.appspot.com/bug?extid=c74fea926a78b8a91042
> compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/46ec18b3c2fb/disk-62693208.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/b4ea0cb78498/vmlinux-62693208.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/5fb3938c7272/bzImage-62693208.xz
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+c74fea...@syzkaller.appspotmail.com
>
> general protection fault, probably for non-canonical address 0xdffffc000000011d: 0000 [#1] PREEMPT SMP KASAN
> KASAN: null-ptr-deref in range [0x00000000000008e8-0x00000000000008ef]
> CPU: 1 PID: 27342 Comm: syz-executor.5 Not tainted 6.5.0-rc7-next-20230825-syzkaller #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023
> RIP: 0010:io_sqpoll_wq_cpu_affinity+0x8c/0xe0 io_uring/sqpoll.c:433

Jens,

I'm not sure I got the whole story on this one, but it seems fairly
trivial to reproduce and I can't see another way it could be
triggered. What do you think?

Thanks,

-- >8 --
Subject: [PATCH] io_uring: Don't set affinity on a dying sqpoll thread

Syzbot reported a null-ptr-deref of sqd->thread inside
io_sqpoll_wq_cpu_affinity. It turns out the sqd->thread can go away
from under us during io_uring_register, in case the process gets a
fatal signal during io_uring_register.

It is not particularly hard to hit the race, and while I am not sure
this is the exact case hit by syzbot, it solves it. Finally, checking
->thread is enough to close the race because we locked sqd while
"parking" the thread, thus preventing it from going away.

I reproduced it fairly consistently with a program that does:

int main(void) {
...
io_uring_queue_init(RING_LEN, &ring1, IORING_SETUP_SQPOLL);
while (1) {
io_uring_register_iowq_aff(ring, 1, &mask);
}
}

Executed in a loop with timeout to trigger SIGTERM:
while true; do timeout 1 /a.out ; done

This will hit the following BUG() in very few attempts.

BUG: kernel NULL pointer dereference, address: 00000000000007a8
PGD 800000010e949067 P4D 800000010e949067 PUD 10e46e067 PMD 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 15715 Comm: dead-sqpoll Not tainted 6.5.0-rc7-next-20230825-g193296236fa0-dirty #23
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
RIP: 0010:io_sqpoll_wq_cpu_affinity+0x27/0x70
Code: 90 90 90 0f 1f 44 00 00 55 53 48 8b 9f 98 03 00 00 48 85 db 74 4f
48 89 df 48 89 f5 e8 e2 f8 ff ff 48 8b 43 38 48 85 c0 74 22 <48> 8b b8
a8 07 00 00 48 89 ee e8 ba b1 00 00 48 89 df 89 c5 e8 70
RSP: 0018:ffffb04040ea7e70 EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff93c010749e40 RCX: 0000000000000001
RDX: 0000000000000000 RSI: ffffffffa7653331 RDI: 00000000ffffffff
RBP: ffffb04040ea7eb8 R08: 0000000000000000 R09: c0000000ffffdfff
R10: ffff93c01141b600 R11: ffffb04040ea7d18 R12: ffff93c00ea74840
R13: 0000000000000011 R14: 0000000000000000 R15: ffff93c00ea74800
FS: 00007fb7c276ab80(0000) GS:ffff93c36f200000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00000000000007a8 CR3: 0000000111634003 CR4: 0000000000370ef0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>

? __die_body+0x1a/0x60
? page_fault_oops+0x154/0x440
? do_user_addr_fault+0x174/0x7b0
? exc_page_fault+0x63/0x140
? asm_exc_page_fault+0x22/0x30
? io_sqpoll_wq_cpu_affinity+0x27/0x70
__io_register_iowq_aff+0x2b/0x60
__io_uring_register+0x614/0xa70
__x64_sys_io_uring_register+0xaa/0x1a0
do_syscall_64+0x3a/0x90
entry_SYSCALL_64_after_hwframe+0x6e/0xd8
RIP: 0033:0x7fb7c226fec9
Code: 2e 00 b8 ca 00 00 00 0f 05 eb a5 66 0f 1f 44 00 00 48 89 f8 48 89

f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01

f0 ff ff 73 01 c3 48 8b 0d 97 7f 2d 00 f7 d8 64 89 01 48
RSP: 002b:00007ffe2c0674f8 EFLAGS: 00000246 ORIG_RAX: 00000000000001ab
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb7c226fec9
RDX: 00007ffe2c067530 RSI: 0000000000000011 RDI: 0000000000000003
RBP: 00007ffe2c0675d0 R08: 00007ffe2c067550 R09: 00007ffe2c067550

R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000

R13: 00007ffe2c067750 R14: 0000000000000000 R15: 0000000000000000
</TASK>
Modules linked in:
CR2: 00000000000007a8

---[ end trace 0000000000000000 ]---

Reported-by: syzbot+c74fea...@syzkaller.appspotmail.com
Fixes: ebdfefc09c6d ("io_uring/sqpoll: fix io-wq affinity when IORING_SETUP_SQPOLL is used")
Signed-off-by: Gabriel Krisman Bertazi <kri...@suse.de>
---
io_uring/sqpoll.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index ee2d2c687fda..bd6c2c7959a5 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -430,7 +430,9 @@ __cold int io_sqpoll_wq_cpu_affinity(struct io_ring_ctx *ctx,

if (sqd) {
io_sq_thread_park(sqd);
- ret = io_wq_cpu_affinity(sqd->thread->io_uring, mask);
+ /* Don't set affinity for a dying thread */
+ if (sqd->thread)
+ ret = io_wq_cpu_affinity(sqd->thread->io_uring, mask);
io_sq_thread_unpark(sqd);
}

--
2.41.0

[Pavel Begunkov]

...

> Jens,
>
> I'm not sure I got the whole story on this one, but it seems fairly
> trivial to reproduce and I can't see another way it could be
> triggered. What do you think?

Right, it can be null and that should be checked. I haven't taken
a look at the report but the patch looks good regardless, thanks

Pavel Begunkov

[Jens Axboe]

Yep looks like the right fix, we should check the thread after parking.
I'll get this added to the queue, thanks.

--
Jens Axboe

[Jens Axboe]

On Mon, 28 Aug 2023 19:42:49 -0400, Gabriel Krisman Bertazi wrote:
> Syzbot reported a null-ptr-deref of sqd->thread inside
> io_sqpoll_wq_cpu_affinity. It turns out the sqd->thread can go away
> from under us during io_uring_register, in case the process gets a
> fatal signal during io_uring_register.
>
> It is not particularly hard to hit the race, and while I am not sure
> this is the exact case hit by syzbot, it solves it. Finally, checking
> ->thread is enough to close the race because we locked sqd while
> "parking" the thread, thus preventing it from going away.
>

> [...]

Applied, thanks!

[1/1] io_uring: Don't set affinity on a dying sqpoll thread
commit: 37b1499057f61de370dbd8f0c296274ab7186605

Best regards,
--
Jens Axboe