[syzbot] possible deadlock in throtl_pending_timer_fn

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: cb7cbaae7fd9 Merge tag 'drm-next-2022-03-25' of git://anon..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=17ef8b43700000
kernel config: https://syzkaller.appspot.com/x/.config?x=7094767cefc58fb9
dashboard link: https://syzkaller.appspot.com/bug?extid=934ebb67352c8a490bf3
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

Unfortunately, I don't have any reproducer for this issue yet.

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+934ebb...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
5.17.0-syzkaller-10734-gcb7cbaae7fd9 #0 Not tainted
------------------------------------------------------
syz-executor.2/3698 is trying to acquire lock:
ffff888014034fc8 (&q->queue_lock){..-.}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:374 [inline]
ffff888014034fc8 (&q->queue_lock){..-.}-{2:2}, at: throtl_pending_timer_fn+0xf7/0x1690 block/blk-throttle.c:1152

but task is already holding lock:
ffffc900007f8d70 ((&sq->pending_timer)){+.-.}-{0:0}, at: lockdep_copy_map include/linux/lockdep.h:35 [inline]
ffffc900007f8d70 ((&sq->pending_timer)){+.-.}-{0:0}, at: call_timer_fn+0xd5/0x6b0 kernel/time/timer.c:1411

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 ((&sq->pending_timer)){+.-.}-{0:0}:
del_timer_sync+0x5b/0x1b0 kernel/time/timer.c:1364
throtl_pd_free+0x15/0x40 block/blk-throttle.c:500
blkg_free.part.0+0xb2/0x1f0 block/blk-cgroup.c:83
blkg_free block/blk-cgroup.c:207 [inline]
blkg_alloc+0x507/0x690 block/blk-cgroup.c:206
blkg_create+0xac4/0x1030 block/blk-cgroup.c:262
blkg_lookup_create block/blk-cgroup.c:370 [inline]
blkg_tryget_closest block/blk-cgroup.c:1821 [inline]
bio_associate_blkg_from_css+0xa7a/0xee0 block/blk-cgroup.c:1856
bio_associate_blkg+0xcd/0x3c0 block/blk-cgroup.c:1884
bio_init+0x2ae/0x5a0 block/bio.c:272
bio_alloc_bioset+0x233/0x4e0 block/bio.c:517
bio_alloc include/linux/bio.h:423 [inline]
submit_bh_wbc+0x276/0x6b0 fs/buffer.c:3028
submit_bh fs/buffer.c:3055 [inline]
block_read_full_page+0xe88/0x18f0 fs/buffer.c:2336
do_read_cache_folio+0x454/0x760 mm/filemap.c:3515
do_read_cache_page mm/filemap.c:3594 [inline]
read_cache_page_gfp+0x27/0x270 mm/filemap.c:3627
btrfs_read_disk_super fs/btrfs/volumes.c:1296 [inline]
btrfs_scan_one_device+0x295/0x5f0 fs/btrfs/volumes.c:1365
btrfs_mount_root+0x4b5/0xc10 fs/btrfs/super.c:1758
legacy_get_tree+0x105/0x220 fs/fs_context.c:610
vfs_get_tree+0x89/0x2f0 fs/super.c:1497
fc_mount fs/namespace.c:1043 [inline]
vfs_kern_mount.part.0+0xd3/0x170 fs/namespace.c:1073
vfs_kern_mount+0x3c/0x60 fs/namespace.c:1060
btrfs_mount+0x234/0xa60 fs/btrfs/super.c:1856
legacy_get_tree+0x105/0x220 fs/fs_context.c:610
vfs_get_tree+0x89/0x2f0 fs/super.c:1497
do_new_mount fs/namespace.c:3039 [inline]
path_mount+0x1320/0x1fa0 fs/namespace.c:3369
do_mount fs/namespace.c:3382 [inline]
__do_sys_mount fs/namespace.c:3590 [inline]
__se_sys_mount fs/namespace.c:3567 [inline]
__x64_sys_mount+0x27f/0x300 fs/namespace.c:3567
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x44/0xae

-> #0 (&q->queue_lock){..-.}-{2:2}:
check_prev_add kernel/locking/lockdep.c:3065 [inline]
check_prevs_add kernel/locking/lockdep.c:3188 [inline]
validate_chain kernel/locking/lockdep.c:3803 [inline]
__lock_acquire+0x2ac6/0x56c0 kernel/locking/lockdep.c:5029
lock_acquire kernel/locking/lockdep.c:5641 [inline]
lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5606
__raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
_raw_spin_lock_irq+0x32/0x50 kernel/locking/spinlock.c:170
spin_lock_irq include/linux/spinlock.h:374 [inline]
throtl_pending_timer_fn+0xf7/0x1690 block/blk-throttle.c:1152
call_timer_fn+0x1a5/0x6b0 kernel/time/timer.c:1421
expire_timers kernel/time/timer.c:1466 [inline]
__run_timers.part.0+0x67c/0xa30 kernel/time/timer.c:1734
__run_timers kernel/time/timer.c:1715 [inline]
run_timer_softirq+0xb3/0x1d0 kernel/time/timer.c:1747
__do_softirq+0x29b/0x9c2 kernel/softirq.c:558
invoke_softirq kernel/softirq.c:432 [inline]
__irq_exit_rcu+0x123/0x180 kernel/softirq.c:637
irq_exit_rcu+0x5/0x20 kernel/softirq.c:649
sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1097
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
lock_is_held_type+0xe/0x140 kernel/locking/lockdep.c:5669
lock_is_held include/linux/lockdep.h:283 [inline]
rcu_read_lock_sched_held+0x3a/0x70 kernel/rcu/update.c:125
trace_lock_release include/trace/events/lock.h:58 [inline]
lock_release+0x522/0x720 kernel/locking/lockdep.c:5652
rcu_lock_release include/linux/rcupdate.h:273 [inline]
rcu_read_unlock include/linux/rcupdate.h:727 [inline]
ext4_get_group_info+0x211/0x3d0 fs/ext4/ext4.h:3369
ext4_validate_block_bitmap.part.0+0x2c/0xd50 fs/ext4/balloc.c:376
ext4_validate_block_bitmap fs/ext4/balloc.c:373 [inline]
ext4_read_block_bitmap_nowait+0xb33/0x1ad0 fs/ext4/balloc.c:509
ext4_read_block_bitmap+0x1d/0xa0 fs/ext4/balloc.c:550
ext4_mb_clear_bb fs/ext4/mballoc.c:5928 [inline]
ext4_free_blocks+0x60e/0x2060 fs/ext4/mballoc.c:6164
ext4_remove_blocks fs/ext4/extents.c:2538 [inline]
ext4_ext_rm_leaf fs/ext4/extents.c:2704 [inline]
ext4_ext_remove_space+0x1f4f/0x4520 fs/ext4/extents.c:2952
ext4_ext_truncate+0x33f/0x400 fs/ext4/extents.c:4412
ext4_truncate+0xecc/0x1440 fs/ext4/inode.c:4224
ext4_evict_inode+0xa73/0x1950 fs/ext4/inode.c:286
evict+0x2ed/0x6b0 fs/inode.c:664
iput_final fs/inode.c:1744 [inline]
iput.part.0+0x562/0x820 fs/inode.c:1770
iput+0x58/0x70 fs/inode.c:1760
d_delete_notify include/linux/fsnotify.h:261 [inline]
vfs_rmdir.part.0+0x496/0x5a0 fs/namei.c:4032
vfs_rmdir fs/namei.c:4004 [inline]
do_rmdir+0x3a6/0x430 fs/namei.c:4080
__do_sys_rmdir fs/namei.c:4099 [inline]
__se_sys_rmdir fs/namei.c:4097 [inline]
__x64_sys_rmdir+0xc6/0x110 fs/namei.c:4097
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x44/0xae

other info that might help us debug this:

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock((&sq->pending_timer));
lock(&q->queue_lock);
lock((&sq->pending_timer));
lock(&q->queue_lock);

*** DEADLOCK ***

7 locks held by syz-executor.2/3698:
#0: ffff888018e6a460 (sb_writers#5){.+.+}-{0:0}, at: do_rmdir+0x1d0/0x430 fs/namei.c:4063
#1: ffff8880470a35c0 (&type->i_mutex_dir_key#3/1){+.+.}-{3:3}, at: inode_lock_nested include/linux/fs.h:813 [inline]
#1: ffff8880470a35c0 (&type->i_mutex_dir_key#3/1){+.+.}-{3:3}, at: do_rmdir+0x21b/0x430 fs/namei.c:4067
#2: ffff888018e6a650 (sb_internal){.+.+}-{0:0}, at: evict+0x2ed/0x6b0 fs/inode.c:664
#3: ffff8880437d8990 (jbd2_handle){++++}-{0:0}, at: start_this_handle+0xfb4/0x14a0 fs/jbd2/transaction.c:461
#4: ffff88804737f408 (&ei->i_data_sem){++++}-{3:3}, at: ext4_truncate+0x631/0x1440 fs/ext4/inode.c:4219
#5: ffffffff8bd7ea60 (rcu_read_lock){....}-{1:2}, at: ext4_get_group_info+0xf2/0x3d0 fs/ext4/ext4.h:3368
#6: ffffc900007f8d70 ((&sq->pending_timer)){+.-.}-{0:0}, at: lockdep_copy_map include/linux/lockdep.h:35 [inline]
#6: ffffc900007f8d70 ((&sq->pending_timer)){+.-.}-{0:0}, at: call_timer_fn+0xd5/0x6b0 kernel/time/timer.c:1411

stack backtrace:
CPU: 3 PID: 3698 Comm: syz-executor.2 Not tainted 5.17.0-syzkaller-10734-gcb7cbaae7fd9 #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2145
check_prev_add kernel/locking/lockdep.c:3065 [inline]
check_prevs_add kernel/locking/lockdep.c:3188 [inline]
validate_chain kernel/locking/lockdep.c:3803 [inline]
__lock_acquire+0x2ac6/0x56c0 kernel/locking/lockdep.c:5029
lock_acquire kernel/locking/lockdep.c:5641 [inline]
lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5606
__raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
_raw_spin_lock_irq+0x32/0x50 kernel/locking/spinlock.c:170
spin_lock_irq include/linux/spinlock.h:374 [inline]
throtl_pending_timer_fn+0xf7/0x1690 block/blk-throttle.c:1152
call_timer_fn+0x1a5/0x6b0 kernel/time/timer.c:1421
expire_timers kernel/time/timer.c:1466 [inline]
__run_timers.part.0+0x67c/0xa30 kernel/time/timer.c:1734
__run_timers kernel/time/timer.c:1715 [inline]
run_timer_softirq+0xb3/0x1d0 kernel/time/timer.c:1747
__do_softirq+0x29b/0x9c2 kernel/softirq.c:558
invoke_softirq kernel/softirq.c:432 [inline]
__irq_exit_rcu+0x123/0x180 kernel/softirq.c:637
irq_exit_rcu+0x5/0x20 kernel/softirq.c:649
sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1097
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:lock_is_held_type+0xe/0x140 kernel/locking/lockdep.c:5669
Code: 89 44 24 04 e8 33 8d bb ff 0f 0b 8b 44 24 04 eb bd 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 41 56 41 55 41 bd ff ff ff ff 41 54 <55> 53 48 83 ec 08 8b 0d 12 7a 56 04 85 c9 0f 84 d9 00 00 00 65 8b
RSP: 0018:ffffc9000229f540 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 0000000000000003 RCX: ffffffff815ce081
RDX: 000000000000000f RSI: 00000000ffffffff RDI: ffffffff8bd7e9a0
RBP: 1ffff92000453eb1 R08: 0000000000000000 R09: ffffffff8db49557
R10: fffffbfff1b692aa R11: 0000000000000000 R12: 0000000000000001
R13: 00000000ffffffff R14: 0000000000000001 R15: 0000000000000006
lock_is_held include/linux/lockdep.h:283 [inline]
rcu_read_lock_sched_held+0x3a/0x70 kernel/rcu/update.c:125
trace_lock_release include/trace/events/lock.h:58 [inline]
lock_release+0x522/0x720 kernel/locking/lockdep.c:5652
rcu_lock_release include/linux/rcupdate.h:273 [inline]
rcu_read_unlock include/linux/rcupdate.h:727 [inline]
ext4_get_group_info+0x211/0x3d0 fs/ext4/ext4.h:3369
ext4_validate_block_bitmap.part.0+0x2c/0xd50 fs/ext4/balloc.c:376
ext4_validate_block_bitmap fs/ext4/balloc.c:373 [inline]
ext4_read_block_bitmap_nowait+0xb33/0x1ad0 fs/ext4/balloc.c:509
ext4_read_block_bitmap+0x1d/0xa0 fs/ext4/balloc.c:550
ext4_mb_clear_bb fs/ext4/mballoc.c:5928 [inline]
ext4_free_blocks+0x60e/0x2060 fs/ext4/mballoc.c:6164
ext4_remove_blocks fs/ext4/extents.c:2538 [inline]
ext4_ext_rm_leaf fs/ext4/extents.c:2704 [inline]
ext4_ext_remove_space+0x1f4f/0x4520 fs/ext4/extents.c:2952
ext4_ext_truncate+0x33f/0x400 fs/ext4/extents.c:4412
ext4_truncate+0xecc/0x1440 fs/ext4/inode.c:4224
ext4_evict_inode+0xa73/0x1950 fs/ext4/inode.c:286
evict+0x2ed/0x6b0 fs/inode.c:664
iput_final fs/inode.c:1744 [inline]
iput.part.0+0x562/0x820 fs/inode.c:1770
iput+0x58/0x70 fs/inode.c:1760
d_delete_notify include/linux/fsnotify.h:261 [inline]
vfs_rmdir.part.0+0x496/0x5a0 fs/namei.c:4032
vfs_rmdir fs/namei.c:4004 [inline]
do_rmdir+0x3a6/0x430 fs/namei.c:4080
__do_sys_rmdir fs/namei.c:4099 [inline]
__se_sys_rmdir fs/namei.c:4097 [inline]
__x64_sys_rmdir+0xc6/0x110 fs/namei.c:4097
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f93106889b7
Code: 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 54 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffc98d5ac78 EFLAGS: 00000207 ORIG_RAX: 0000000000000054
RAX: ffffffffffffffda RBX: 0000000000000065 RCX: 00007f93106889b7
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007ffc98d5be00
RBP: 00007ffc98d5ad40 R08: 0000000000000000 R09: 00007ffc98d5ab10
R10: 0000555555c118e3 R11: 0000000000000207 R12: 00007f93106e21ea
R13: 00007ffc98d5be00 R14: 0000555555c11810 R15: 00007ffc98d5be40
</TASK>
BUG: sleeping function called from invalid context at block/blk-sysfs.c:766
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 4095, name: kworker/u16:5
preempt_count: 101, expected: 0
RCU nest depth: 0, expected: 0
INFO: lockdep is turned off.
Preemption disabled at:
[<0000000000000000>] 0x0
CPU: 3 PID: 4095 Comm: kworker/u16:5 Not tainted 5.17.0-syzkaller-10734-gcb7cbaae7fd9 #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014
Workqueue: netns cleanup_net
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
__might_resched.cold+0x222/0x26b kernel/sched/core.c:9733
blk_release_queue+0x1f/0x320 block/blk-sysfs.c:766
kobject_cleanup lib/kobject.c:705 [inline]
kobject_release lib/kobject.c:736 [inline]
kref_put include/linux/kref.h:65 [inline]
kobject_put+0x1c8/0x540 lib/kobject.c:753
blkg_free.part.0+0x112/0x1f0 block/blk-cgroup.c:86
blkg_free block/blk-cgroup.c:78 [inline]
__blkg_release+0x105/0x160 block/blk-cgroup.c:102
rcu_do_batch kernel/rcu/tree.c:2535 [inline]
rcu_core+0x7b1/0x1880 kernel/rcu/tree.c:2786
__do_softirq+0x29b/0x9c2 kernel/softirq.c:558
do_softirq.part.0+0xde/0x130 kernel/softirq.c:459
</IRQ>
<TASK>
do_softirq kernel/softirq.c:451 [inline]
__local_bh_enable_ip+0x102/0x120 kernel/softirq.c:383
pneigh_ifdown_and_unlock net/core/neighbour.c:835 [inline]
__neigh_ifdown.isra.0+0x15e/0x330 net/core/neighbour.c:387
neigh_ifdown+0x1b/0x30 net/core/neighbour.c:403
addrconf_ifdown.isra.0+0x133a/0x16b0 net/ipv6/addrconf.c:3887
addrconf_notify+0xeb/0x1ba0 net/ipv6/addrconf.c:3662
notifier_call_chain+0xb5/0x200 kernel/notifier.c:84
call_netdevice_notifiers_info+0xb5/0x130 net/core/dev.c:1938
call_netdevice_notifiers_extack net/core/dev.c:1976 [inline]
call_netdevice_notifiers net/core/dev.c:1990 [inline]
unregister_netdevice_many+0x92e/0x18a0 net/core/dev.c:10741
vti6_exit_batch_net+0x3a0/0x670 net/ipv6/ip6_vti.c:1190
ops_exit_list+0x125/0x170 net/core/net_namespace.c:167
cleanup_net+0x4ea/0xb00 net/core/net_namespace.c:594
process_one_work+0x996/0x1610 kernel/workqueue.c:2289
worker_thread+0x665/0x1080 kernel/workqueue.c:2436
kthread+0x2e9/0x3a0 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
</TASK>
----------------
Code disassembly (best guess):
0: 89 44 24 04 mov %eax,0x4(%rsp)
4: e8 33 8d bb ff callq 0xffbb8d3c
9: 0f 0b ud2
b: 8b 44 24 04 mov 0x4(%rsp),%eax
f: eb bd jmp 0xffffffce
11: 66 66 2e 0f 1f 84 00 data16 nopw %cs:0x0(%rax,%rax,1)
18: 00 00 00 00
1c: 41 57 push %r15
1e: 41 56 push %r14
20: 41 55 push %r13
22: 41 bd ff ff ff ff mov $0xffffffff,%r13d
28: 41 54 push %r12
* 2a: 55 push %rbp <-- trapping instruction
2b: 53 push %rbx
2c: 48 83 ec 08 sub $0x8,%rsp
30: 8b 0d 12 7a 56 04 mov 0x4567a12(%rip),%ecx # 0x4567a48
36: 85 c9 test %ecx,%ecx
38: 0f 84 d9 00 00 00 je 0x117
3e: 65 gs
3f: 8b .byte 0x8b


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[Ming Lei]

On Wed, Mar 30, 2022 at 5:23 AM syzbot
<syzbot+934ebb...@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: cb7cbaae7fd9 Merge tag 'drm-next-2022-03-25' of git://anon..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=17ef8b43700000
> kernel config: https://syzkaller.appspot.com/x/.config?x=7094767cefc58fb9
> dashboard link: https://syzkaller.appspot.com/bug?extid=934ebb67352c8a490bf3
> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+934ebb...@syzkaller.appspotmail.com

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git
for-5.18/block

It should be fixed by:

https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?h=for-5.18/block&id=d578c770c85233af592e54537f93f3831bde7e9a

Thanks,

[syzbot]

> On Wed, Mar 30, 2022 at 5:23 AM syzbot
> <syzbot+934ebb...@syzkaller.appspotmail.com> wrote:
>>
>> Hello,
>>
>> syzbot found the following issue on:
>>
>> HEAD commit: cb7cbaae7fd9 Merge tag 'drm-next-2022-03-25' of git://anon..
>> git tree: upstream
>> console output: https://syzkaller.appspot.com/x/log.txt?x=17ef8b43700000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=7094767cefc58fb9
>> dashboard link: https://syzkaller.appspot.com/bug?extid=934ebb67352c8a490bf3
>> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
>>
>> Unfortunately, I don't have any reproducer for this issue yet.
>>
>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>> Reported-by: syzbot+934ebb...@syzkaller.appspotmail.com
>
> #syz test: git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git

This crash does not have a reproducer. I cannot test it.

[Ming Lei]

On Tue, Mar 29, 2022 at 07:52:57PM -0700, syzbot wrote:
> > On Wed, Mar 30, 2022 at 5:23 AM syzbot
> > <syzbot+934ebb...@syzkaller.appspotmail.com> wrote:
> >>
> >> Hello,
> >>
> >> syzbot found the following issue on:
> >>
> >> HEAD commit: cb7cbaae7fd9 Merge tag 'drm-next-2022-03-25' of git://anon..
> >> git tree: upstream
> >> console output: https://syzkaller.appspot.com/x/log.txt?x=17ef8b43700000
> >> kernel config: https://syzkaller.appspot.com/x/.config?x=7094767cefc58fb9
> >> dashboard link: https://syzkaller.appspot.com/bug?extid=934ebb67352c8a490bf3
> >> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> >>
> >> Unfortunately, I don't have any reproducer for this issue yet.
> >>
> >> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> >> Reported-by: syzbot+934ebb...@syzkaller.appspotmail.com
> >
> > #syz test: git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git
>
> This crash does not have a reproducer. I cannot test it.

This one is triggered in case of memory allocation failure, especially blkg_free()
is called from blkg_alloc() directly. That is why it is hard to reproduce, and it
is one old issue.


Thanks,
Ming

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: cb71b93c2dc3 Add linux-next specific files for 20220628
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=154438f2080000
kernel config: https://syzkaller.appspot.com/x/.config?x=badbc1adb2d582eb

dashboard link: https://syzkaller.appspot.com/bug?extid=934ebb67352c8a490bf3
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=17713dee080000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=15d24952080000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+934ebb...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected

5.19.0-rc4-next-20220628-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor358/3597 is trying to acquire lock:
ffff888146f83aa0 (&q->queue_lock){..-.}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:385 [inline]
ffff888146f83aa0 (&q->queue_lock){..-.}-{2:2}, at: throtl_pending_timer_fn+0xf7/0x1690 block/blk-throttle.c:1152

but task is already holding lock:

ffffc900001e0d70 ((&sq->pending_timer)){+.-.}-{0:0}, at: lockdep_copy_map include/linux/lockdep.h:31 [inline]
ffffc900001e0d70 ((&sq->pending_timer)){+.-.}-{0:0}, at: call_timer_fn+0xd5/0x6b0 kernel/time/timer.c:1464

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((&sq->pending_timer)){+.-.}-{0:0}:
del_timer_sync+0x5b/0x1b0 kernel/time/timer.c:1417
throtl_pd_free+0x15/0x40 block/blk-throttle.c:500
blkcg_deactivate_policy block/blk-cgroup.c:1520 [inline]
blkcg_deactivate_policy+0x2d2/0x4e0 block/blk-cgroup.c:1498
blk_throtl_exit+0x8a/0x1a0 block/blk-throttle.c:2336
blkcg_init_queue+0x19b/0x620 block/blk-cgroup.c:1301
__alloc_disk_node+0x260/0x610 block/genhd.c:1365
__blk_mq_alloc_disk+0x133/0x1c0 block/blk-mq.c:3927
loop_add+0x3e2/0xaf0 drivers/block/loop.c:1978
loop_control_ioctl+0x133/0x540 drivers/block/loop.c:2151
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:870 [inline]
__se_sys_ioctl fs/ioctl.c:856 [inline]
__x64_sys_ioctl+0x193/0x200 fs/ioctl.c:856

do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80

entry_SYSCALL_64_after_hwframe+0x46/0xb0

-> #1 (&blkcg->lock){....}-{2:2}:
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:360 [inline]
blkg_create+0x45e/0x1000 block/blk-cgroup.c:344
blkcg_init_queue+0xb2/0x620 block/blk-cgroup.c:1282
__alloc_disk_node+0x260/0x610 block/genhd.c:1365
__blk_alloc_disk+0x35/0x70 block/genhd.c:1405
brd_alloc.part.0+0x27f/0x7a0 drivers/block/brd.c:391
brd_alloc drivers/block/brd.c:374 [inline]
brd_init+0x1b8/0x24b drivers/block/brd.c:477
do_one_initcall+0xfe/0x650 init/main.c:1300
do_initcall_level init/main.c:1375 [inline]
do_initcalls init/main.c:1391 [inline]
do_basic_setup init/main.c:1410 [inline]
kernel_init_freeable+0x6b1/0x73a init/main.c:1617
kernel_init+0x1a/0x1d0 init/main.c:1506
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:302

-> #0 (&q->queue_lock){..-.}-{2:2}:

check_prev_add kernel/locking/lockdep.c:3095 [inline]
check_prevs_add kernel/locking/lockdep.c:3214 [inline]
validate_chain kernel/locking/lockdep.c:3829 [inline]
__lock_acquire+0x2abe/0x5660 kernel/locking/lockdep.c:5053
lock_acquire kernel/locking/lockdep.c:5665 [inline]
lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5630

__raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
_raw_spin_lock_irq+0x32/0x50 kernel/locking/spinlock.c:170

spin_lock_irq include/linux/spinlock.h:385 [inline]
throtl_pending_timer_fn+0xf7/0x1690 block/blk-throttle.c:1152
call_timer_fn+0x1a0/0x6b0 kernel/time/timer.c:1474
expire_timers kernel/time/timer.c:1519 [inline]
__run_timers.part.0+0x674/0xa80 kernel/time/timer.c:1790
__run_timers kernel/time/timer.c:1768 [inline]
run_timer_softirq+0xb3/0x1d0 kernel/time/timer.c:1803
__do_softirq+0x29b/0x9c2 kernel/softirq.c:571
invoke_softirq kernel/softirq.c:445 [inline]
__irq_exit_rcu+0x123/0x180 kernel/softirq.c:650
irq_exit_rcu+0x5/0x20 kernel/softirq.c:662
sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1106
asm_sysvec_apic_timer_interrupt+0x1b/0x20 arch/x86/include/asm/idtentry.h:649
lock_acquire+0x1ef/0x570 kernel/locking/lockdep.c:5633
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:360 [inline]
new_inode_pseudo fs/inode.c:1022 [inline]
new_inode+0x48/0x270 fs/inode.c:1047
debugfs_get_inode+0x1a/0x130 fs/debugfs/inode.c:72
debugfs_create_dir+0xdc/0x4d0 fs/debugfs/inode.c:578
bdi_debug_register mm/backing-dev.c:112 [inline]
bdi_register_va.part.0+0x22d/0x800 mm/backing-dev.c:880
bdi_register_va mm/backing-dev.c:908 [inline]
bdi_register+0x12a/0x140 mm/backing-dev.c:905
device_add_disk+0x83c/0xe20 block/genhd.c:513
add_disk include/linux/blkdev.h:768 [inline]
loop_add+0x884/0xaf0 drivers/block/loop.c:2031
loop_control_ioctl+0x133/0x540 drivers/block/loop.c:2151
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:870 [inline]
__se_sys_ioctl fs/ioctl.c:856 [inline]
__x64_sys_ioctl+0x193/0x200 fs/ioctl.c:856

do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80

entry_SYSCALL_64_after_hwframe+0x46/0xb0

other info that might help us debug this:

Chain exists of:
&q->queue_lock --> &blkcg->lock --> (&sq->pending_timer)

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock((&sq->pending_timer));

lock(&blkcg->lock);

lock((&sq->pending_timer));
lock(&q->queue_lock);

*** DEADLOCK ***

3 locks held by syz-executor358/3597:
#0: ffff888140c48b70 (&sb->s_type->i_mutex_key#3){+.+.}-{3:3}, at: inode_lock include/linux/fs.h:761 [inline]
#0: ffff888140c48b70 (&sb->s_type->i_mutex_key#3){+.+.}-{3:3}, at: start_creating.part.0+0xb2/0x280 fs/debugfs/inode.c:348
#1: ffff8880761cf340 (&sb->s_type->i_lock_key#7){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:360 [inline]
#1: ffff8880761cf340 (&sb->s_type->i_lock_key#7){+.+.}-{2:2}, at: new_inode_pseudo fs/inode.c:1022 [inline]
#1: ffff8880761cf340 (&sb->s_type->i_lock_key#7){+.+.}-{2:2}, at: new_inode+0x48/0x270 fs/inode.c:1047
#2: ffffc900001e0d70 ((&sq->pending_timer)){+.-.}-{0:0}, at: lockdep_copy_map include/linux/lockdep.h:31 [inline]
#2: ffffc900001e0d70 ((&sq->pending_timer)){+.-.}-{0:0}, at: call_timer_fn+0xd5/0x6b0 kernel/time/timer.c:1464

stack backtrace:
CPU: 1 PID: 3597 Comm: syz-executor358 Not tainted 5.19.0-rc4-next-20220628-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/22/2022

Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106

check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2175
check_prev_add kernel/locking/lockdep.c:3095 [inline]
check_prevs_add kernel/locking/lockdep.c:3214 [inline]
validate_chain kernel/locking/lockdep.c:3829 [inline]
__lock_acquire+0x2abe/0x5660 kernel/locking/lockdep.c:5053
lock_acquire kernel/locking/lockdep.c:5665 [inline]
lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5630

__raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
_raw_spin_lock_irq+0x32/0x50 kernel/locking/spinlock.c:170

spin_lock_irq include/linux/spinlock.h:385 [inline]
throtl_pending_timer_fn+0xf7/0x1690 block/blk-throttle.c:1152
call_timer_fn+0x1a0/0x6b0 kernel/time/timer.c:1474
expire_timers kernel/time/timer.c:1519 [inline]
__run_timers.part.0+0x674/0xa80 kernel/time/timer.c:1790
__run_timers kernel/time/timer.c:1768 [inline]
run_timer_softirq+0xb3/0x1d0 kernel/time/timer.c:1803
__do_softirq+0x29b/0x9c2 kernel/softirq.c:571
invoke_softirq kernel/softirq.c:445 [inline]
__irq_exit_rcu+0x123/0x180 kernel/softirq.c:650
irq_exit_rcu+0x5/0x20 kernel/softirq.c:662
sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1106
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1b/0x20 arch/x86/include/asm/idtentry.h:649
RIP: 0010:lock_acquire+0x1ef/0x570 kernel/locking/lockdep.c:5633
Code: e7 a3 7e 83 f8 01 0f 85 e8 02 00 00 9c 58 f6 c4 02 0f 85 fb 02 00 00 48 83 7c 24 08 00 74 01 fb 48 b8 00 00 00 00 00 fc ff df <48> 01 c3 48 c7 03 00 00 00 00 48 c7 43 08 00 00 00 00 48 8b 84 24
RSP: 0018:ffffc90002fffaa0 EFLAGS: 00000206
RAX: dffffc0000000000 RBX: 1ffff920005fff56 RCX: ffffffff815e3c0e
RDX: 1ffff11004088899 RSI: 0000000000000001 RDI: 0000000000000000
RBP: 0000000000000001 R08: 0000000000000000 R09: ffffffff906a5967
R10: fffffbfff20d4b2c R11: 0000000000000001 R12: 0000000000000000
R13: 0000000000000000 R14: ffff8880761cf340 R15: 0000000000000000
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:360 [inline]
new_inode_pseudo fs/inode.c:1022 [inline]
new_inode+0x48/0x270 fs/inode.c:1047
debugfs_get_inode+0x1a/0x130 fs/debugfs/inode.c:72
debugfs_create_dir+0xdc/0x4d0 fs/debugfs/inode.c:578
bdi_debug_register mm/backing-dev.c:112 [inline]
bdi_register_va.part.0+0x22d/0x800 mm/backing-dev.c:880
bdi_register_va mm/backing-dev.c:908 [inline]
bdi_register+0x12a/0x140 mm/backing-dev.c:905
device_add_disk+0x83c/0xe20 block/genhd.c:513
add_disk include/linux/blkdev.h:768 [inline]
loop_add+0x884/0xaf0 drivers/block/loop.c:2031
loop_control_ioctl+0x133/0x540 drivers/block/loop.c:2151
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:870 [inline]
__se_sys_ioctl fs/ioctl.c:856 [inline]
__x64_sys_ioctl+0x193/0x200 fs/ioctl.c:856

do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80

entry_SYSCALL_64_after_hwframe+0x46/0xb0
RIP: 0033:0x7f92d80b4079
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 b1 14 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 c0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffcc0861258 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f92d80b4079
RDX: 0000000000000000 RSI: 0000000000004c80 RDI: 0000000000000003
RBP: 00007ffcc0861270 R08: 0000000000000002 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
R13: 431bde82d7b634db R14: 0000000000000000 R15: 0000000000000000

</TASK>
----------------
Code disassembly (best guess):

0: e7 a3 out %eax,$0xa3
2: 7e 83 jle 0xffffff87
4: f8 clc
5: 01 0f add %ecx,(%rdi)
7: 85 e8 test %ebp,%eax
9: 02 00 add (%rax),%al
b: 00 9c 58 f6 c4 02 0f add %bl,0xf02c4f6(%rax,%rbx,2)
12: 85 fb test %edi,%ebx
14: 02 00 add (%rax),%al
16: 00 48 83 add %cl,-0x7d(%rax)
19: 7c 24 jl 0x3f
1b: 08 00 or %al,(%rax)
1d: 74 01 je 0x20
1f: fb sti
20: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
27: fc ff df
* 2a: 48 01 c3 add %rax,%rbx <-- trapping instruction
2d: 48 c7 03 00 00 00 00 movq $0x0,(%rbx)
34: 48 c7 43 08 00 00 00 movq $0x0,0x8(%rbx)
3b: 00
3c: 48 rex.W
3d: 8b .byte 0x8b
3e: 84 .byte 0x84
3f: 24 .byte 0x24

[Hillf Danton]

On Fri, 29 Jul 2022 16:32:23 -0700

> syzbot has found a reproducer for the following issue on:
>
> HEAD commit: cb71b93c2dc3 Add linux-next specific files for 20220628
> git tree: linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=154438f2080000
> kernel config: https://syzkaller.appspot.com/x/.config?x=badbc1adb2d582eb
> dashboard link: https://syzkaller.appspot.com/bug?extid=934ebb67352c8a490bf3
> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=17713dee080000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=15d24952080000

Free tg in workqueue to avoid holding any lock.

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git cb71b93c2dc3

--- y/block/blk-throttle.h
+++ b/block/blk-throttle.h
@@ -141,6 +141,8 @@ struct throtl_grp {

struct blkg_rwstat stat_bytes;
struct blkg_rwstat stat_ios;
+
+ struct work_struct free_work;
};

extern struct blkcg_policy blkcg_policy_throtl;
--- y/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -493,9 +493,9 @@ static void throtl_pd_offline(struct blk
throtl_upgrade_state(tg->td);
}

-static void throtl_pd_free(struct blkg_policy_data *pd)
+static void tg_free_workfn(struct work_struct *w)
{
- struct throtl_grp *tg = pd_to_tg(pd);
+ struct throtl_grp *tg = container_of(w, struct throtl_grp, free_work);

del_timer_sync(&tg->service_queue.pending_timer);
blkg_rwstat_exit(&tg->stat_bytes);
@@ -503,6 +503,14 @@ static void throtl_pd_free(struct blkg_p
kfree(tg);
}

+static void throtl_pd_free(struct blkg_policy_data *pd)
+{
+ struct throtl_grp *tg = pd_to_tg(pd);
+
+ INIT_WORK(&tg->free_work, tg_free_workfn);
+ queue_work(system_unbound_wq, &tg->free_work);
+}
+
static struct throtl_grp *
throtl_rb_first(struct throtl_service_queue *parent_sq)
{
--

[syzbot]

syzbot has bisected this issue to:

commit 0a9a25ca78437b39e691bcc3dc8240455b803d8d
Author: Ming Lei <ming...@redhat.com>
Date: Fri Mar 18 13:01:43 2022 +0000

block: let blkcg_gq grab request queue's refcnt

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=16c3cfc2080000
start commit: cb71b93c2dc3 Add linux-next specific files for 20220628
git tree: linux-next
final oops: https://syzkaller.appspot.com/x/report.txt?x=15c3cfc2080000
console output: https://syzkaller.appspot.com/x/log.txt?x=11c3cfc2080000

kernel config: https://syzkaller.appspot.com/x/.config?x=badbc1adb2d582eb
dashboard link: https://syzkaller.appspot.com/bug?extid=934ebb67352c8a490bf3

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=17713dee080000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=15d24952080000

Reported-by: syzbot+934ebb...@syzkaller.appspotmail.com
Fixes: 0a9a25ca7843 ("block: let blkcg_gq grab request queue's refcnt")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+934ebb...@syzkaller.appspotmail.com

Tested on:

commit: cb71b93c Add linux-next specific files for 20220628
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
console output: https://syzkaller.appspot.com/x/log.txt?x=128a6b41080000

kernel config: https://syzkaller.appspot.com/x/.config?x=badbc1adb2d582eb
dashboard link: https://syzkaller.appspot.com/bug?extid=934ebb67352c8a490bf3
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

patch: https://syzkaller.appspot.com/x/patch.diff?x=147e67aa080000

Note: testing is done by a robot and is best-effort only.

[Ming Lei]

No, this lockdep warning isn't related with the above commit, which
caused another regression, but fixed by commit d578c770c852
("block: avoid calling blkg_free() in atomic context"). Looks syzbot
can't recognize difference between the two different issues.

This specific issue of '[syzbot] possible deadlock in throtl_pending_timer_fn',
is actually introduced by commit ("27029b4b18aa blkcg: fix memleak for iolatency").

blk_throtl_exit() isn't safe to be called before blkg_destroy_all().

The following change should avoid the issue:


diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 869af9d72bcf..1606acb917fd 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -1268,6 +1268,7 @@ static int blkcg_css_online(struct cgroup_subsys_state *css)
int blkcg_init_queue(struct request_queue *q)
{
struct blkcg_gq *new_blkg, *blkg;
+ bool need_exit_throtl = false;
bool preloaded;
int ret;

@@ -1301,7 +1302,7 @@ int blkcg_init_queue(struct request_queue *q)

ret = blk_iolatency_init(q);
if (ret) {
- blk_throtl_exit(q);
+ need_exit_throtl = true;
blk_ioprio_exit(q);
goto err_destroy_all;
}
@@ -1310,6 +1311,8 @@ int blkcg_init_queue(struct request_queue *q)

err_destroy_all:
blkg_destroy_all(q);
+ if (need_exit_throtl)
+ blk_throtl_exit(q);
return ret;
err_unlock:
spin_unlock_irq(&q->queue_lock);



Thanks,
Ming

[syzbot]

Auto-closing this bug as obsolete.
No recent activity, existing reproducers are no longer triggering the issue.