[syzbot] WARNING in io_wq_put

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: a38fd874 Linux 5.12-rc2
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1276fd0ad00000
kernel config: https://syzkaller.appspot.com/x/.config?x=9008fb06fa15d749
dashboard link: https://syzkaller.appspot.com/bug?extid=77a738a6bc947bf639ca
compiler: Debian clang version 11.0.1-2

Unfortunately, I don't have any reproducer for this issue yet.

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+77a738...@syzkaller.appspotmail.com

------------[ cut here ]------------
WARNING: CPU: 1 PID: 12346 at fs/io-wq.c:1061 io_wq_destroy fs/io-wq.c:1061 [inline]
WARNING: CPU: 1 PID: 12346 at fs/io-wq.c:1061 io_wq_put+0x153/0x260 fs/io-wq.c:1072
Modules linked in:
CPU: 1 PID: 12346 Comm: syz-executor.5 Not tainted 5.12.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:io_wq_destroy fs/io-wq.c:1061 [inline]
RIP: 0010:io_wq_put+0x153/0x260 fs/io-wq.c:1072
Code: 8d e8 71 90 ea 01 49 89 c4 41 83 fc 40 7d 4f e8 33 4d 97 ff 42 80 7c 2d 00 00 0f 85 77 ff ff ff e9 7a ff ff ff e8 1d 4d 97 ff <0f> 0b eb b9 8d 6b ff 89 ee 09 de bf ff ff ff ff e8 18 51 97 ff 09
RSP: 0018:ffffc90001ebfb08 EFLAGS: 00010293
RAX: ffffffff81e16083 RBX: ffff888019038040 RCX: ffff88801e86b780
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000040
RBP: 1ffff1100b2f8a80 R08: ffffffff81e15fce R09: ffffed100b2f8a82
R10: ffffed100b2f8a82 R11: 0000000000000000 R12: 0000000000000000
R13: dffffc0000000000 R14: ffff8880597c5400 R15: ffff888019038000
FS: 00007f8dcd89c700(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055e9a054e160 CR3: 000000001dfb8000 CR4: 00000000001506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
io_uring_clean_tctx+0x1b7/0x210 fs/io_uring.c:8802
__io_uring_files_cancel+0x13c/0x170 fs/io_uring.c:8820
io_uring_files_cancel include/linux/io_uring.h:47 [inline]
do_exit+0x258/0x2340 kernel/exit.c:780
do_group_exit+0x168/0x2d0 kernel/exit.c:922
get_signal+0x1734/0x1ef0 kernel/signal.c:2773
arch_do_signal_or_restart+0x3c/0x610 arch/x86/kernel/signal.c:811
handle_signal_work kernel/entry/common.c:147 [inline]
exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
exit_to_user_mode_prepare+0xac/0x1e0 kernel/entry/common.c:208
__syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
syscall_exit_to_user_mode+0x48/0x180 kernel/entry/common.c:301
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x465f69
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f8dcd89c188 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
RAX: 0000000000000302 RBX: 000000000056bf60 RCX: 0000000000465f69
RDX: 0000000000000000 RSI: 0000000000000302 RDI: 0000000000000005
RBP: 00000000004bfa67 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf60
R13: 00007ffd520357bf R14: 00007f8dcd89c300 R15: 0000000000022000


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[Jens Axboe]

On 3/9/21 8:33 AM, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: a38fd874 Linux 5.12-rc2
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=1276fd0ad00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=9008fb06fa15d749
> dashboard link: https://syzkaller.appspot.com/bug?extid=77a738a6bc947bf639ca
> compiler: Debian clang version 11.0.1-2
>
> Unfortunately, I don't have any reproducer for this issue yet.

Should be fixed in io_uring-5.12 as well.

--
Jens Axboe

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: 1c273e10 Merge tag 'zonefs-5.12-rc4' of git://git.kernel.o..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=13853506d00000
kernel config: https://syzkaller.appspot.com/x/.config?x=c51293a9ca630f6d

dashboard link: https://syzkaller.appspot.com/bug?extid=77a738a6bc947bf639ca
compiler: Debian clang version 11.0.1-2

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=11ec259ed00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13acfa62d00000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+77a738...@syzkaller.appspotmail.com

------------[ cut here ]------------

WARNING: CPU: 1 PID: 12892 at fs/io-wq.c:1064 io_wq_destroy fs/io-wq.c:1064 [inline]
WARNING: CPU: 1 PID: 12892 at fs/io-wq.c:1064 io_wq_put+0x153/0x260 fs/io-wq.c:1075
Modules linked in:
CPU: 1 PID: 12892 Comm: syz-executor789 Not tainted 5.12.0-rc3-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011

RIP: 0010:io_wq_destroy fs/io-wq.c:1064 [inline]
RIP: 0010:io_wq_put+0x153/0x260 fs/io-wq.c:1075
Code: 8d e8 f1 5f f2 01 49 89 c4 41 83 fc 40 7d 4f e8 a3 e9 97 ff 42 80 7c 2d 00 00 0f 85 77 ff ff ff e9 7a ff ff ff e8 8d e9 97 ff <0f> 0b eb b9 8d 6b ff 89 ee 09 de bf ff ff ff ff e8 88 ed 97 ff 09
RSP: 0018:ffffc9000a5efaa0 EFLAGS: 00010293
RAX: ffffffff81e11103 RBX: ffff888021873040 RCX: ffff88801fbb3880

RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000040

RBP: 1ffff11004b3e480 R08: ffffffff81e1104e R09: ffffed1004b3e482
R10: ffffed1004b3e482 R11: 0000000000000000 R12: 0000000000000000
R13: dffffc0000000000 R14: ffff8880259f2400 R15: ffff888021873000
FS: 00007f2407204700(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 0000000000480510 CR3: 000000002d9a1000 CR4: 00000000001506f0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:

io_uring_clean_tctx fs/io_uring.c:8874 [inline]
__io_uring_files_cancel+0xe11/0xe60 fs/io_uring.c:8939
io_uring_files_cancel include/linux/io_uring.h:22 [inline]

do_exit+0x258/0x2340 kernel/exit.c:780
do_group_exit+0x168/0x2d0 kernel/exit.c:922
get_signal+0x1734/0x1ef0 kernel/signal.c:2773
arch_do_signal_or_restart+0x3c/0x610 arch/x86/kernel/signal.c:811
handle_signal_work kernel/entry/common.c:147 [inline]
exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
exit_to_user_mode_prepare+0xac/0x1e0 kernel/entry/common.c:208
__syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]

syscall_exit_to_user_mode+0x26/0x70 kernel/entry/common.c:301
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x448c49
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 d1 15 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f24072041f8 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
RAX: 0000000000000302 RBX: 00000000004cd508 RCX: 0000000000448c49

RDX: 0000000000000000 RSI: 0000000000000302 RDI: 0000000000000005

RBP: 00000000004cd500 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000004cd50c
R13: 00007ffc6fa9d1bf R14: 00007f2407204300 R15: 0000000000022000

[Jens Axboe]

On 3/20/21 6:44 PM, syzbot wrote:
> syzbot has found a reproducer for the following issue on:
>
> HEAD commit: 1c273e10 Merge tag 'zonefs-5.12-rc4' of git://git.kernel.o..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=13853506d00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=c51293a9ca630f6d
> dashboard link: https://syzkaller.appspot.com/bug?extid=77a738a6bc947bf639ca
> compiler: Debian clang version 11.0.1-2
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=11ec259ed00000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13acfa62d00000
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+77a738...@syzkaller.appspotmail.com

#syz test: git://git.kernel.dk/linux-block wq-no-manager

--
Jens Axboe

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+77a738...@syzkaller.appspotmail.com

Tested on:

commit: c95a47c2 io-wq: eliminate the need for a manager thread
git tree: git://git.kernel.dk/linux-block wq-no-manager
kernel config: https://syzkaller.appspot.com/x/.config?x=175bf2d0517d3b04

dashboard link: https://syzkaller.appspot.com/bug?extid=77a738a6bc947bf639ca
compiler: Debian clang version 11.0.1-2

Note: testing is done by a robot and is best-effort only.

[Jens Axboe]

On 3/20/21 6:44 PM, syzbot wrote:

> syzbot has found a reproducer for the following issue on:
>
> HEAD commit: 1c273e10 Merge tag 'zonefs-5.12-rc4' of git://git.kernel.o..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=13853506d00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=c51293a9ca630f6d
> dashboard link: https://syzkaller.appspot.com/bug?extid=77a738a6bc947bf639ca
> compiler: Debian clang version 11.0.1-2
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=11ec259ed00000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13acfa62d00000
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+77a738...@syzkaller.appspotmail.com

#syz test: git://git.kernel.dk/linux-block io_uring-5.12

--
Jens Axboe

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
WARNING in kvm_wait

------------[ cut here ]------------
raw_local_irq_restore() called with IRQs enabled
WARNING: CPU: 1 PID: 191 at kernel/locking/irqflag-debug.c:10 warn_bogus_irq_restore+0x1f/0x30 kernel/locking/irqflag-debug.c:10
Modules linked in:
CPU: 1 PID: 191 Comm: kworker/u4:4 Not tainted 5.12.0-rc2-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011

Workqueue: 0x0 (events_unbound)
RIP: 0010:warn_bogus_irq_restore+0x1f/0x30 kernel/locking/irqflag-debug.c:10
Code: cc cc cc cc cc cc cc cc cc cc cc 80 3d c7 af b1 03 00 74 01 c3 c6 05 bd af b1 03 01 48 c7 c7 c0 5f ae 89 31 c0 e8 d1 dd f6 f7 <0f> 0b c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc 41 56 53 48 83
RSP: 0018:ffffc90000dc0a08 EFLAGS: 00010246
RAX: 6a712abdc5855100 RBX: ffffffff8f982d60 RCX: ffff8880118bb880
RDX: 0000000000000103 RSI: 0000000000000103 RDI: 0000000000000000
RBP: 1ffff920001b8142 R08: ffffffff81609502 R09: ffffed10173e5fe8
R10: ffffed10173e5fe8 R11: 0000000000000000 R12: 0000000000000003
R13: ffff88823ffe6880 R14: 0000000000000246 R15: dffffc0000000000
FS: 0000000000000000(0000) GS:ffff8880b9f00000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00007f44d117d718 CR3: 000000001340a000 CR4: 00000000001506e0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:

<IRQ>
kvm_wait+0x10e/0x160 arch/x86/kernel/kvm.c:860
pv_wait arch/x86/include/asm/paravirt.h:564 [inline]
pv_wait_head_or_lock kernel/locking/qspinlock_paravirt.h:470 [inline]
__pv_queued_spin_lock_slowpath+0x6b5/0xb90 kernel/locking/qspinlock.c:508
pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:554 [inline]
queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline]
queued_spin_lock include/asm-generic/qspinlock.h:85 [inline]
do_raw_spin_lock+0x430/0x810 kernel/locking/spinlock_debug.c:113
spin_lock include/linux/spinlock.h:354 [inline]
mac80211_hwsim_tx_frame_no_nl+0x60e/0x1860 drivers/net/wireless/mac80211_hwsim.c:1514
mac80211_hwsim_tx_frame+0x143/0x180 drivers/net/wireless/mac80211_hwsim.c:1775
mac80211_hwsim_beacon_tx+0x4b9/0x870 drivers/net/wireless/mac80211_hwsim.c:1829
__iterate_interfaces+0x23e/0x4b0 net/mac80211/util.c:793
ieee80211_iterate_active_interfaces_atomic+0x9b/0x120 net/mac80211/util.c:829
mac80211_hwsim_beacon+0xa4/0x180 drivers/net/wireless/mac80211_hwsim.c:1852
__run_hrtimer kernel/time/hrtimer.c:1519 [inline]
__hrtimer_run_queues+0x4c9/0xa00 kernel/time/hrtimer.c:1583
hrtimer_run_softirq+0x176/0x1e0 kernel/time/hrtimer.c:1600
__do_softirq+0x318/0x714 kernel/softirq.c:345
invoke_softirq kernel/softirq.c:221 [inline]
__irq_exit_rcu+0x1d8/0x200 kernel/softirq.c:422
irq_exit_rcu+0x5/0x20 kernel/softirq.c:434
sysvec_apic_timer_interrupt+0x91/0xb0 arch/x86/kernel/apic/apic.c:1100
</IRQ>
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:632
RIP: 0010:__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
RIP: 0010:_raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:199
Code: b4 fd ff 66 90 53 48 89 fb 48 83 c7 18 48 8b 74 24 08 e8 0e 56 09 f8 48 89 df e8 56 2b 0b f8 e8 41 9f 2b f8 fb bf 01 00 00 00 <e8> c6 3b ff f7 65 8b 05 c7 9f ae 76 85 c0 74 02 5b c3 e8 7b fb ac
RSP: 0018:ffffc9000143fca0 EFLAGS: 00000286
RAX: 6a712abdc5855100 RBX: ffff8880b9f34c40 RCX: ffffffff8f59cb03
RDX: 0000000040000000 RSI: 0000000000000002 RDI: 0000000000000001
RBP: ffffc9000143fd00 R08: ffffffff817eef20 R09: ffffed10173e6989
R10: ffffed10173e6989 R11: 0000000000000000 R12: ffff8880b9f34c40
R13: ffff8880118bb880 R14: dffffc0000000000 R15: 0000000000000000
finish_task_switch+0x145/0x620 kernel/sched/core.c:4193
context_switch kernel/sched/core.c:4327 [inline]
__schedule+0x9a1/0xe70 kernel/sched/core.c:5075
schedule+0x14b/0x200 kernel/sched/core.c:5154
worker_thread+0xfe6/0x1300 kernel/workqueue.c:2442
kthread+0x39a/0x3c0 kernel/kthread.c:292
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294


Tested on:

commit: f5d2d23b io-wq: fix race around pending work on teardown
git tree: git://git.kernel.dk/linux-block io_uring-5.12
console output: https://syzkaller.appspot.com/x/log.txt?x=12cad621d00000
kernel config: https://syzkaller.appspot.com/x/.config?x=9fdcf055a7409ee0

[Jens Axboe]

This is a different issue, not related to this code at all.

--
Jens Axboe