[syzbot] [io-uring?] KCSAN: data-race in io_wq_activate_free_worker / io_wq_worker_running

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: f97e18a3f2fb Merge tag 'gpio-updates-for-v6.6' of git://gi..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=12864667a80000
kernel config: https://syzkaller.appspot.com/x/.config?x=fe440f256d065d3b
dashboard link: https://syzkaller.appspot.com/bug?extid=a36975231499dc24df44
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/b1781aaff038/disk-f97e18a3.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/5b915468fd6d/vmlinux-f97e18a3.xz
kernel image: https://storage.googleapis.com/syzbot-assets/abc8ece931f3/bzImage-f97e18a3.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+a36975...@syzkaller.appspotmail.com

==================================================================
BUG: KCSAN: data-race in io_wq_activate_free_worker / io_wq_worker_running

write to 0xffff888127f736c4 of 4 bytes by task 4731 on cpu 1:
io_wq_worker_running+0x64/0xa0 io_uring/io-wq.c:668
schedule_timeout+0xcc/0x230 kernel/time/timer.c:2167
io_wq_worker+0x4b2/0x840 io_uring/io-wq.c:633
ret_from_fork+0x2e/0x40 arch/x86/kernel/process.c:145
ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:304

read to 0xffff888127f736c4 of 4 bytes by task 4719 on cpu 0:
io_wq_get_acct io_uring/io-wq.c:168 [inline]
io_wq_activate_free_worker+0xfa/0x280 io_uring/io-wq.c:267
io_wq_enqueue+0x262/0x450 io_uring/io-wq.c:914
io_queue_iowq+0x1d1/0x310 io_uring/io_uring.c:514
io_queue_sqe_fallback+0x82/0xe0 io_uring/io_uring.c:2084
io_submit_sqe io_uring/io_uring.c:2305 [inline]
io_submit_sqes+0xbd3/0xfb0 io_uring/io_uring.c:2420
__do_sys_io_uring_enter io_uring/io_uring.c:3628 [inline]
__se_sys_io_uring_enter+0x1f8/0x1c10 io_uring/io_uring.c:3562
__x64_sys_io_uring_enter+0x78/0x90 io_uring/io_uring.c:3562
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd

value changed: 0x0000000d -> 0x0000000b

Reported by Kernel Concurrency Sanitizer on:
CPU: 0 PID: 4719 Comm: syz-executor.1 Not tainted 6.5.0-syzkaller-01810-gf97e18a3f2fb #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023
==================================================================


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the bug is already fixed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite bug's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the bug is a duplicate of another bug, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[Pavel Begunkov]

1) the worst case scenario we'll choose a wrong type of
worker, which is inconsequential.

2) we're changing the IO_WORKER_F_RUNNING bit, but checking
for IO_WORKER_F_BOUND. The latter one is set at the very
beginning, it would require compiler to be super inventive
to actually hit the problem.

I don't believe it's a problem, but it'll nice to attribute
it properly, READ_ONCE?, or split IO_WORKER_F_BOUND out into
a separate field.

--
Pavel Begunkov

[Marco Elver]

It's a simple bit flag set & read, I'd go for READ_ONCE() (and
WRITE_ONCE() - but up to you, these bitflag sets & reads have been ok
with just the READ_ONCE(), and KCSAN currently doesn't care if there's
a WRITE_ONCE() or not).

> value changed: 0x0000000d -> 0x0000000b

This is interesting though - it says that it observed 2 bits being
flipped. We don't see where IO_WORKER_F_FREE was unset though.

[Edward AD]

please test un write data race

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git f97e18a3f2fb

diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index 62f345587df5..e70b89d32cc1 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -165,7 +165,7 @@ static inline struct io_wq_acct *io_work_get_acct(struct io_wq *wq,

static inline struct io_wq_acct *io_wq_get_acct(struct io_worker *worker)
{
- return io_get_acct(worker->wq, worker->flags & IO_WORKER_F_BOUND);
+ return io_get_acct(worker->wq, READ_ONCE(worker->flags) & IO_WORKER_F_BOUND);
}

static void io_worker_ref_put(struct io_wq *wq)
@@ -685,11 +685,11 @@ void io_wq_worker_running(struct task_struct *tsk)

if (!worker)
return;
- if (!(worker->flags & IO_WORKER_F_UP))
+ if (!(READ_ONCE(worker->flags) & IO_WORKER_F_UP))
return;
- if (worker->flags & IO_WORKER_F_RUNNING)
+ if (READ_ONCE(worker->flags) & IO_WORKER_F_RUNNING)
return;
- worker->flags |= IO_WORKER_F_RUNNING;
+ WRITE_ONCE(worker->flags, READ_ONCE(worker->flags) | IO_WORKER_F_RUNNING);
io_wq_inc_running(worker);
}

[syzbot]

> please test un write data race
>
> #syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git f97e18a3f2fb

This crash does not have a reproducer. I cannot test it.

[Pavel Begunkov]

__io_worker_busy() clears it, should be it. I assume syz just
missed another false data race with this one. After init only
the worker thread should be changing the flags AFAIR

--
Pavel Begunkov

[Marco Elver]

The data races reported are very real, i.e. it only reports if it
actually observes _real_ concurrency. I guess the question is if these
are benign or not. If benign, you can choose to annotate with
READ/WRITE_ONCE [1], data_race, or leave as is (ignoring this report
should not make it re-report any time soon).

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/memory-model/Documentation/access-marking.txt

[Pavel Begunkov]

That's what I'm saying, I assume that syz is not completely
analytical and triggering a race is subject to execution
randomness, and races with IO_WORKER_F_FREE are harder to hit
for syzkaller.

> are benign or not. If benign, you can choose to annotate with

Yes, it is, just like the one in the report

> READ/WRITE_ONCE [1], data_race, or leave as is (ignoring this report
> should not make it re-report any time soon).
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/memory-model/Documentation/access-marking.txt

--
Pavel Begunkov

[syzbot]

Auto-closing this bug as obsolete.
Crashes did not happen for a while, no reproducer and no activity.