possible deadlock in __queue_work

15 views
Skip to first unread message

syzbot

unread,
Feb 29, 2020, 10:20:13 PM2/29/20
to syzkaller...@googlegroups.com
Hello,

syzbot found the following crash on:

HEAD commit: 78d697fc Linux 4.14.172
git tree: linux-4.14.y
console output: https://syzkaller.appspot.com/x/log.txt?x=108f7a81e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=31ad682bcda9b93f
dashboard link: https://syzkaller.appspot.com/bug?extid=961bc330f3ee15812d2c
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+961bc3...@syzkaller.appspotmail.com

RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000009
RBP: 000000000076bf20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000000a
R13: 000000000000009b R14: 00000000004c2f07 R15: 0000000000000000
======================================================
WARNING: possible circular locking dependency detected
4.14.172-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor.0/11748 is trying to acquire lock:
(console_owner){-.-.}, at: [<ffffffff8148e310>] console_trylock_spinning kernel/printk/printk.c:1658 [inline]
(console_owner){-.-.}, at: [<ffffffff8148e310>] vprintk_emit kernel/printk/printk.c:1922 [inline]
(console_owner){-.-.}, at: [<ffffffff8148e310>] vprintk_emit+0x2f0/0x600 kernel/printk/printk.c:1888

but task is already holding lock:
(&(&pool->lock)->rlock){-.-.}, at: [<ffffffff813ae284>] spin_lock include/linux/spinlock.h:317 [inline]
(&(&pool->lock)->rlock){-.-.}, at: [<ffffffff813ae284>] start_flush_work kernel/workqueue.c:2831 [inline]
(&(&pool->lock)->rlock){-.-.}, at: [<ffffffff813ae284>] flush_work+0x134/0x780 kernel/workqueue.c:2890

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #4 (&(&pool->lock)->rlock){-.-.}:
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:152
spin_lock include/linux/spinlock.h:317 [inline]
__queue_work+0x1ba/0xf10 kernel/workqueue.c:1419
queue_work_on+0x159/0x1d0 kernel/workqueue.c:1488
queue_work include/linux/workqueue.h:491 [inline]
schedule_work include/linux/workqueue.h:549 [inline]
put_pwq kernel/workqueue.c:1092 [inline]
put_pwq+0x129/0x180 kernel/workqueue.c:1077
put_pwq_unlocked.part.0+0x30/0x70 kernel/workqueue.c:1109
put_pwq_unlocked kernel/workqueue.c:1103 [inline]
destroy_workqueue+0x570/0x660 kernel/workqueue.c:4171
do_floppy_init drivers/block/floppy.c:4732 [inline]
floppy_async_init+0x1abb/0x1c06 drivers/block/floppy.c:4749
async_run_entry_fn+0x102/0x4c0 kernel/async.c:127
process_one_work+0x813/0x1540 kernel/workqueue.c:2114
worker_thread+0x5d1/0x1070 kernel/workqueue.c:2248
kthread+0x30d/0x420 kernel/kthread.c:232
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404

-> #3 (&pool->lock/1){..-.}:
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:152
spin_lock include/linux/spinlock.h:317 [inline]
__queue_work+0x1ba/0xf10 kernel/workqueue.c:1419
queue_work_on+0x159/0x1d0 kernel/workqueue.c:1488
pty_write+0x173/0x1c0 drivers/tty/pty.c:125
n_tty_write+0x8dd/0xe30 drivers/tty/n_tty.c:2356
do_tty_write drivers/tty/tty_io.c:959 [inline]
tty_write+0x3f0/0x700 drivers/tty/tty_io.c:1043
__vfs_write+0xe4/0x630 fs/read_write.c:480
vfs_write+0x192/0x4e0 fs/read_write.c:544
SYSC_write fs/read_write.c:590 [inline]
SyS_write+0xf2/0x210 fs/read_write.c:582
do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #2 (&(&port->lock)->rlock){-.-.}:
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x8c/0xbf kernel/locking/spinlock.c:160
tty_port_tty_get+0x1d/0x80 drivers/tty/tty_port.c:288
tty_port_default_wakeup+0x11/0x40 drivers/tty/tty_port.c:46
serial8250_tx_chars+0x400/0x9e0 drivers/tty/serial/8250/8250_port.c:1810
serial8250_handle_irq.part.0+0x1f8/0x240 drivers/tty/serial/8250/8250_port.c:1883
serial8250_handle_irq drivers/tty/serial/8250/8250_port.c:1869 [inline]
serial8250_default_handle_irq+0x96/0x110 drivers/tty/serial/8250/8250_port.c:1899
serial8250_interrupt+0xe4/0x1a0 drivers/tty/serial/8250/8250_core.c:129
__handle_irq_event_percpu+0x125/0x7e0 kernel/irq/handle.c:147
handle_irq_event_percpu+0x66/0x120 kernel/irq/handle.c:187
handle_irq_event+0xa2/0x12d kernel/irq/handle.c:204
handle_edge_irq+0x215/0x810 kernel/irq/chip.c:770
generic_handle_irq_desc include/linux/irqdesc.h:159 [inline]
handle_irq+0x35/0x50 arch/x86/kernel/irq_64.c:87
do_IRQ+0x93/0x1d0 arch/x86/kernel/irq.c:230
ret_from_intr+0x0/0x1e
arch_local_irq_restore arch/x86/include/asm/paravirt.h:779 [inline]
__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:160 [inline]
_raw_spin_unlock_irqrestore+0x91/0xe0 kernel/locking/spinlock.c:192
spin_unlock_irqrestore include/linux/spinlock.h:372 [inline]
uart_write+0x28c/0x4e0 drivers/tty/serial/serial_core.c:625
process_output_block drivers/tty/n_tty.c:595 [inline]
n_tty_write+0x375/0xe30 drivers/tty/n_tty.c:2333
do_tty_write drivers/tty/tty_io.c:959 [inline]
tty_write+0x3f0/0x700 drivers/tty/tty_io.c:1043
redirected_tty_write+0x9c/0xb0 drivers/tty/tty_io.c:1064
__vfs_write+0xe4/0x630 fs/read_write.c:480
vfs_write+0x192/0x4e0 fs/read_write.c:544
SYSC_write fs/read_write.c:590 [inline]
SyS_write+0xf2/0x210 fs/read_write.c:582
do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #1 (&port_lock_key){-.-.}:
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x8c/0xbf kernel/locking/spinlock.c:160
serial8250_console_write+0x6c1/0x8e0 drivers/tty/serial/8250/8250_port.c:3236
call_console_drivers kernel/printk/printk.c:1725 [inline]
console_unlock+0x9b2/0xec0 kernel/printk/printk.c:2397
vprintk_emit kernel/printk/printk.c:1923 [inline]
vprintk_emit+0x1f8/0x600 kernel/printk/printk.c:1888
vprintk_func+0x58/0x152 kernel/printk/printk_safe.c:401
printk+0x9e/0xbc kernel/printk/printk.c:1996
register_console+0x5ec/0x9a0 kernel/printk/printk.c:2716
univ8250_console_init+0x2f/0x3a drivers/tty/serial/8250/8250_core.c:691
console_init+0x46/0x53 kernel/printk/printk.c:2797
start_kernel+0x437/0x676 init/main.c:634
secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:240

-> #0 (console_owner){-.-.}:
lock_acquire+0x170/0x3f0 kernel/locking/lockdep.c:3994
console_trylock_spinning kernel/printk/printk.c:1679 [inline]
vprintk_emit kernel/printk/printk.c:1922 [inline]
vprintk_emit+0x32d/0x600 kernel/printk/printk.c:1888
vprintk_func+0x58/0x152 kernel/printk/printk_safe.c:401
printk+0x9e/0xbc kernel/printk/printk.c:1996
fail_dump lib/fault-inject.c:44 [inline]
should_fail.cold+0xdf/0x14b lib/fault-inject.c:149
should_failslab+0xd6/0x130 mm/failslab.c:32
slab_pre_alloc_hook mm/slab.h:421 [inline]
slab_alloc mm/slab.c:3376 [inline]
kmem_cache_alloc+0x44/0x770 mm/slab.c:3550
kmem_cache_zalloc include/linux/slab.h:651 [inline]
fill_pool lib/debugobjects.c:110 [inline]
__debug_object_init+0x5b7/0x830 lib/debugobjects.c:341
__init_work kernel/workqueue.c:504 [inline]
insert_wq_barrier kernel/workqueue.c:2498 [inline]
start_flush_work kernel/workqueue.c:2846 [inline]
flush_work+0x1f7/0x780 kernel/workqueue.c:2890
__cancel_work_timer+0x2d0/0x460 kernel/workqueue.c:2962
uhid_dev_destroy+0x6e/0x100 drivers/hid/uhid.c:568
uhid_char_release+0x4c/0xa0 drivers/hid/uhid.c:645
__fput+0x25f/0x790 fs/file_table.c:210
task_work_run+0x113/0x190 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:191 [inline]
exit_to_usermode_loop+0x1d6/0x220 arch/x86/entry/common.c:164
prepare_exit_to_usermode arch/x86/entry/common.c:199 [inline]
syscall_return_slowpath arch/x86/entry/common.c:270 [inline]
do_syscall_64+0x4a3/0x640 arch/x86/entry/common.c:297
entry_SYSCALL_64_after_hwframe+0x42/0xb7

other info that might help us debug this:

Chain exists of:
console_owner --> &pool->lock/1 --> &(&pool->lock)->rlock

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&(&pool->lock)->rlock);
lock(&pool->lock/1);
lock(&(&pool->lock)->rlock);
lock(console_owner);

*** DEADLOCK ***

1 lock held by syz-executor.0/11748:
#0: (&(&pool->lock)->rlock){-.-.}, at: [<ffffffff813ae284>] spin_lock include/linux/spinlock.h:317 [inline]
#0: (&(&pool->lock)->rlock){-.-.}, at: [<ffffffff813ae284>] start_flush_work kernel/workqueue.c:2831 [inline]
#0: (&(&pool->lock)->rlock){-.-.}, at: [<ffffffff813ae284>] flush_work+0x134/0x780 kernel/workqueue.c:2890

stack backtrace:
CPU: 1 PID: 11748 Comm: syz-executor.0 Not tainted 4.14.172-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0x13e/0x194 lib/dump_stack.c:58
print_circular_bug.isra.0.cold+0x1c4/0x282 kernel/locking/lockdep.c:1258
check_prev_add kernel/locking/lockdep.c:1901 [inline]
check_prevs_add kernel/locking/lockdep.c:2018 [inline]
validate_chain kernel/locking/lockdep.c:2460 [inline]
__lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487
lock_acquire+0x170/0x3f0 kernel/locking/lockdep.c:3994
console_trylock_spinning kernel/printk/printk.c:1679 [inline]
vprintk_emit kernel/printk/printk.c:1922 [inline]
vprintk_emit+0x32d/0x600 kernel/printk/printk.c:1888
vprintk_func+0x58/0x152 kernel/printk/printk_safe.c:401
printk+0x9e/0xbc kernel/printk/printk.c:1996
fail_dump lib/fault-inject.c:44 [inline]
should_fail.cold+0xdf/0x14b lib/fault-inject.c:149
should_failslab+0xd6/0x130 mm/failslab.c:32
slab_pre_alloc_hook mm/slab.h:421 [inline]
slab_alloc mm/slab.c:3376 [inline]
kmem_cache_alloc+0x44/0x770 mm/slab.c:3550
kmem_cache_zalloc include/linux/slab.h:651 [inline]
fill_pool lib/debugobjects.c:110 [inline]
__debug_object_init+0x5b7/0x830 lib/debugobjects.c:341
__init_work kernel/workqueue.c:504 [inline]
insert_wq_barrier kernel/workqueue.c:2498 [inline]
start_flush_work kernel/workqueue.c:2846 [inline]
flush_work+0x1f7/0x780 kernel/workqueue.c:2890
__cancel_work_timer+0x2d0/0x460 kernel/workqueue.c:2962
uhid_dev_destroy+0x6e/0x100 drivers/hid/uhid.c:568
uhid_char_release+0x4c/0xa0 drivers/hid/uhid.c:645
__fput+0x25f/0x790 fs/file_table.c:210
task_work_run+0x113/0x190 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:191 [inline]
exit_to_usermode_loop+0x1d6/0x220 arch/x86/entry/common.c:164
prepare_exit_to_usermode arch/x86/entry/common.c:199 [inline]
syscall_return_slowpath arch/x86/entry/common.c:270 [inline]
do_syscall_64+0x4a3/0x640 arch/x86/entry/common.c:297
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x45c479
RSP: 002b:00007fe423e3ec78 EFLAGS: 00000246 ORIG_RAX: 0000000000000021
RAX: 0000000000000008 RBX: 00007fe423e3f6d4 RCX: 000000000045c479
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000009
RBP: 000000000076bf20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000000a
R13: 000000000000009b R14: 00000000004c2f07 R15: 0000000000000000
hid-generic: probe of 0000:0000:0000.016C failed with error -22
hid-generic 0000:0000:0000.016D: unknown main item tag 0x0
ceph: No mds server is up or the cluster is laggy
hid-generic 0000:0000:0000.016D: hidraw0: <UNKNOWN> HID v0.00 Device [syz0] on syz0
libceph: mon1 [::6]:6789 socket closed (con state CONNECTING)
hid-generic 0000:0000:0000.016E: unknown main item tag 0x0
hid-generic 0000:0000:0000.016E: unknown main item tag 0x0
hid-generic 0000:0000:0000.016E: unknown main item tag 0x0
hid-generic 0000:0000:0000.016E: hidraw0: <UNKNOWN> HID v0.00 Device [syz0] on syz0
ceph: No mds server is up or the cluster is laggy
hid-generic 0000:0000:0000.016F: unknown main item tag 0x0
hid-generic 0000:0000:0000.016F: unknown main item tag 0x0
hid-generic 0000:0000:0000.0170: unknown main item tag 0x0
hid-generic 0000:0000:0000.0170: unknown main item tag 0x0
hid-generic 0000:0000:0000.016F: unknown main item tag 0x0
hid-generic 0000:0000:0000.0170: unknown main item tag 0x0
hid-generic 0000:0000:0000.0170: hidraw0: <UNKNOWN> HID v0.00 Device [syz0] on syz0
hid-generic 0000:0000:0000.016F: hidraw1: <UNKNOWN> HID v0.00 Device [syz0] on syz0
libceph: resolve '0�' (ret=-3): failed
hid-generic 0000:0000:0000.0171: unknown main item tag 0x0
libceph: parse_ips bad ip '[d::],0�:6'
hid-generic 0000:0000:0000.0171: unknown main item tag 0x0
hid-generic 0000:0000:0000.0172: unknown main item tag 0x0
libceph: resolve '0�' (ret=-3): failed
hid-generic 0000:0000:0000.0172: unknown main item tag 0x0
libceph: parse_ips bad ip '[d::],0�:6'
hid-generic 0000:0000:0000.0171: unknown main item tag 0x0
hid-generic 0000:0000:0000.0172: unknown main item tag 0x0
hid-generic 0000:0000:0000.0171: hidraw1: <UNKNOWN> HID v0.00 Device [syz0] on syz0
libceph: connect [d::]:6789 error -101
libceph: mon0 [d::]:6789 connect error
hid-generic 0000:0000:0000.0172: hidraw2: <UNKNOWN> HID v0.00 Device [syz0] on syz0
ceph: No mds server is up or the cluster is laggy
libceph: connect [d::]:6789 error -101
libceph: mon0 [d::]:6789 connect error
ceph: No mds server is up or the cluster is laggy
libceph: connect [d::]:6789 error -101
libceph: mon0 [d::]:6789 connect error
ceph: No mds server is up or the cluster is laggy
libceph: mon1 [::6]:6789 socket closed (con state CONNECTING)
libceph: connect [d::]:6789 error -101
libceph: mon0 [d::]:6789 connect error
libceph: connect [d::]:6789 error -101
libceph: mon0 [d::]:6789 connect error
kauditd_printk_skb: 12663 callbacks suppressed
audit: type=1326 audit(1583032743.071:2178439): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 pid=11814 comm="syz-executor.4" exe="/root/syz-executor.4" sig=0 arch=c000003e syscall=1 compat=0 ip=0x415f51 code=0x50000
audit: type=1326 audit(1583032743.051:2178337): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 pid=11814 comm="syz-executor.4" exe="/root/syz-executor.4" sig=0 arch=c000003e syscall=202 compat=0 ip=0x45c479 code=0x50000
audit: type=1326 audit(1583032743.081:2178440): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 pid=11814 comm="syz-executor.4" exe="/root/syz-executor.4" sig=0 arch=c000003e syscall=202 compat=0 ip=0x45c479 code=0x50000
audit: type=1326 audit(1583032743.081:2178441): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 pid=11814 comm="syz-executor.4" exe="/root/syz-executor.4" sig=0 arch=c000003e syscall=202 compat=0 ip=0x45c479 code=0x50000
audit: type=1326 audit(1583032743.081:2178442): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 pid=11814 comm="syz-executor.4" exe="/root/syz-executor.4" sig=0 arch=c000003e syscall=202 compat=0 ip=0x45c479 code=0x50000
audit: type=1326 audit(1583032743.081:2178443): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 pid=11814 comm="syz-executor.4" exe="/root/syz-executor.4" sig=0 arch=c000003e syscall=202 compat=0 ip=0x45c479 code=0x50000
audit: type=1326 audit(1583032743.081:2178444): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 pid=11814 comm="syz-executor.4" exe="/root/syz-executor.4" sig=0 arch=c000003e syscall=202 compat=0 ip=0x45c479 code=0x50000
audit: type=1326 audit(1583032743.081:2178445): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 pid=11814 comm="syz-executor.4" exe="/root/syz-executor.4" sig=0 arch=c000003e syscall=202 compat=0 ip=0x45c479 code=0x50000
audit: type=1326 audit(1583032743.081:2178446): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 pid=11814 comm="syz-executor.4" exe="/root/syz-executor.4" sig=0 arch=c000003e syscall=202 compat=0 ip=0x45c479 code=0x50000
audit: type=1326 audit(1583032743.081:2178447): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 pid=11814 comm="syz-executor.4" exe="/root/syz-executor.4" sig=0 arch=c000003e syscall=202 compat=0 ip=0x45c479 code=0x50000
libceph: connect [d::]:6789 error -101
libceph: mon0 [d::]:6789 connect error
libceph: mon1 [::6]:6789 socket closed (con state CONNECTING)
libceph: connect [d::]:6789 error -101
libceph: mon0 [d::]:6789 connect error


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

syzbot

unread,
Oct 15, 2020, 6:06:18 AM10/15/20
to syzkaller...@googlegroups.com
Auto-closing this bug as obsolete.
Crashes did not happen for a while, no reproducer and no activity.
Reply all
Reply to author
Forward
0 new messages