possible deadlock in __queue_work

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: 033c4ea4 Linux 4.19.122
git tree: linux-4.19.y
console output: https://syzkaller.appspot.com/x/log.txt?x=17008dec100000
kernel config: https://syzkaller.appspot.com/x/.config?x=7f9e43a505369285
dashboard link: https://syzkaller.appspot.com/bug?extid=03b6a2a3d04bd0b09104
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+03b6a2...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
4.19.122-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor.1/14489 is trying to acquire lock:
0000000047e52878 (&pool->lock/1){..-.}, at: spin_lock include/linux/spinlock.h:329 [inline]
0000000047e52878 (&pool->lock/1){..-.}, at: __queue_work+0x1f1/0x1100 kernel/workqueue.c:1419

but task is already holding lock:
000000001796fd64 (&(&port->lock)->rlock){-.-.}, at: pty_write+0xf6/0x1f0 drivers/tty/pty.c:120

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #4 (&(&port->lock)->rlock){-.-.}:
tty_port_tty_get+0x1d/0x80 drivers/tty/tty_port.c:289
tty_port_default_wakeup+0x11/0x40 drivers/tty/tty_port.c:47
serial8250_tx_chars+0x48f/0xae0 drivers/tty/serial/8250/8250_port.c:1806
serial8250_handle_irq.part.0+0x24b/0x290 drivers/tty/serial/8250/8250_port.c:1879
serial8250_handle_irq drivers/tty/serial/8250/8250_port.c:1865 [inline]
serial8250_default_handle_irq+0xb5/0x140 drivers/tty/serial/8250/8250_port.c:1895
serial8250_interrupt+0xf2/0x1d0 drivers/tty/serial/8250/8250_core.c:125
__handle_irq_event_percpu+0x144/0x8e0 kernel/irq/handle.c:149
handle_irq_event_percpu+0x76/0x160 kernel/irq/handle.c:189
handle_irq_event+0xa2/0x12d kernel/irq/handle.c:206
handle_edge_irq+0x24b/0x8c0 kernel/irq/chip.c:797
generic_handle_irq_desc include/linux/irqdesc.h:155 [inline]
handle_irq+0x35/0x50 arch/x86/kernel/irq_64.c:87
do_IRQ+0x93/0x1c0 arch/x86/kernel/irq.c:246
ret_from_intr+0x0/0x1e
arch_local_irq_restore arch/x86/include/asm/paravirt.h:789 [inline]
__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:160 [inline]
_raw_spin_unlock_irqrestore+0x91/0xe0 kernel/locking/spinlock.c:184
spin_unlock_irqrestore include/linux/spinlock.h:384 [inline]
uart_write+0x365/0x680 drivers/tty/serial/serial_core.c:612
do_output_char+0x5dd/0x850 drivers/tty/n_tty.c:445
process_output drivers/tty/n_tty.c:512 [inline]
n_tty_write+0x497/0x1080 drivers/tty/n_tty.c:2343
do_tty_write drivers/tty/tty_io.c:960 [inline]
tty_write+0x452/0x790 drivers/tty/tty_io.c:1044
redirected_tty_write+0xaa/0xb0 drivers/tty/tty_io.c:1065
do_loop_readv_writev fs/read_write.c:704 [inline]
do_loop_readv_writev fs/read_write.c:688 [inline]
do_iter_write fs/read_write.c:962 [inline]
do_iter_write+0x460/0x5e0 fs/read_write.c:941
vfs_writev+0x1b3/0x2f0 fs/read_write.c:1005
do_writev+0x136/0x330 fs/read_write.c:1040
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #3 (&port_lock_key){-.-.}:
serial8250_console_write+0x79f/0x9c0 drivers/tty/serial/8250/8250_port.c:3251
call_console_drivers kernel/printk/printk.c:1736 [inline]
console_unlock+0xb26/0xfe0 kernel/printk/printk.c:2429
vprintk_emit+0x282/0x6e0 kernel/printk/printk.c:1937
vprintk_func+0x79/0x17e kernel/printk/printk_safe.c:398
printk+0xba/0xed kernel/printk/printk.c:2012
register_console+0x752/0xb50 kernel/printk/printk.c:2745
univ8250_console_init+0x3a/0x46 drivers/tty/serial/8250/8250_core.c:684
console_init+0x4cb/0x718 kernel/printk/printk.c:2831
start_kernel+0x594/0x81c init/main.c:660
secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:243

-> #2 (console_owner){-...}:
vprintk_emit+0x282/0x6e0 kernel/printk/printk.c:1937
vprintk_func+0x79/0x17e kernel/printk/printk_safe.c:398
printk+0xba/0xed kernel/printk/printk.c:2012
fail_dump lib/fault-inject.c:44 [inline]
should_fail+0x666/0x7bc lib/fault-inject.c:149
__should_failslab+0x115/0x180 mm/failslab.c:32
should_failslab+0x5/0xf mm/slab_common.c:1558
slab_pre_alloc_hook mm/slab.h:424 [inline]
slab_alloc mm/slab.c:3383 [inline]
kmem_cache_alloc+0x44/0x710 mm/slab.c:3557
kmem_cache_zalloc include/linux/slab.h:699 [inline]
fill_pool lib/debugobjects.c:134 [inline]
__debug_object_init+0x742/0xac0 lib/debugobjects.c:379
__init_work kernel/workqueue.c:502 [inline]
insert_wq_barrier kernel/workqueue.c:2546 [inline]
start_flush_work kernel/workqueue.c:2888 [inline]
__flush_work+0x25f/0x870 kernel/workqueue.c:2927
smc_close_active+0x55b/0x940 net/smc/smc_close.c:191
smc_release+0x5cf/0x7d0 net/smc/af_smc.c:143
__sock_release+0xcd/0x2a0 net/socket.c:579
sock_close+0x15/0x20 net/socket.c:1140
__fput+0x2cd/0x890 fs/file_table.c:278
task_work_run+0x13f/0x1b0 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:193 [inline]
exit_to_usermode_loop+0x25a/0x2b0 arch/x86/entry/common.c:167
prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #1 (&(&pool->lock)->rlock){-.-.}:
spin_lock include/linux/spinlock.h:329 [inline]
__queue_work+0x1f1/0x1100 kernel/workqueue.c:1419
queue_work_on+0x17e/0x1f0 kernel/workqueue.c:1488
queue_work include/linux/workqueue.h:512 [inline]
schedule_work include/linux/workqueue.h:570 [inline]
put_pwq kernel/workqueue.c:1090 [inline]
put_pwq+0x15a/0x1b0 kernel/workqueue.c:1075
put_pwq_unlocked.part.0+0x30/0x70 kernel/workqueue.c:1107
put_pwq_unlocked kernel/workqueue.c:1101 [inline]
destroy_workqueue+0x5f4/0x6f0 kernel/workqueue.c:4243
do_floppy_init drivers/block/floppy.c:4737 [inline]
floppy_async_init+0x1f0a/0x2043 drivers/block/floppy.c:4754
async_run_entry_fn+0x121/0x530 kernel/async.c:127
process_one_work+0x91f/0x1640 kernel/workqueue.c:2155
worker_thread+0x96/0xe20 kernel/workqueue.c:2298
kthread+0x34a/0x420 kernel/kthread.c:246
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

-> #0 (&pool->lock/1){..-.}:
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
spin_lock include/linux/spinlock.h:329 [inline]
__queue_work+0x1f1/0x1100 kernel/workqueue.c:1419
queue_work_on+0x17e/0x1f0 kernel/workqueue.c:1488
pty_write+0x198/0x1f0 drivers/tty/pty.c:125
process_output_block drivers/tty/n_tty.c:593 [inline]
n_tty_write+0x3ee/0x1080 drivers/tty/n_tty.c:2331
do_tty_write drivers/tty/tty_io.c:960 [inline]
tty_write+0x452/0x790 drivers/tty/tty_io.c:1044
__vfs_write+0xf7/0x760 fs/read_write.c:485
vfs_write+0x206/0x550 fs/read_write.c:549
ksys_write+0x12b/0x2a0 fs/read_write.c:599
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

Chain exists of:
&pool->lock/1 --> &port_lock_key --> &(&port->lock)->rlock

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&(&port->lock)->rlock);
lock(&port_lock_key);
lock(&(&port->lock)->rlock);
lock(&pool->lock/1);

*** DEADLOCK ***

5 locks held by syz-executor.1/14489:
#0: 000000004be58daf (&tty->ldisc_sem){++++}, at: tty_ldisc_ref_wait+0x22/0x80 drivers/tty/tty_ldisc.c:272
#1: 00000000056f5429 (&tty->atomic_write_lock){+.+.}, at: tty_write_lock+0x1e/0x80 drivers/tty/tty_io.c:886
#2: 00000000f0fe9e31 (&o_tty->termios_rwsem/1){++++}, at: n_tty_write+0x1b2/0x1080 drivers/tty/n_tty.c:2314
#3: 00000000d8f6f041 (&ldata->output_lock){+.+.}, at: process_output_block drivers/tty/n_tty.c:548 [inline]
#3: 00000000d8f6f041 (&ldata->output_lock){+.+.}, at: n_tty_write+0x505/0x1080 drivers/tty/n_tty.c:2331
#4: 000000001796fd64 (&(&port->lock)->rlock){-.-.}, at: pty_write+0xf6/0x1f0 drivers/tty/pty.c:120

stack backtrace:
CPU: 0 PID: 14489 Comm: syz-executor.1 Not tainted 4.19.122-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x188/0x20d lib/dump_stack.c:118
print_circular_bug.isra.0.cold+0x1c4/0x282 kernel/locking/lockdep.c:1221
check_prev_add kernel/locking/lockdep.c:1865 [inline]
check_prevs_add kernel/locking/lockdep.c:1978 [inline]
validate_chain kernel/locking/lockdep.c:2419 [inline]
__lock_acquire+0x2e19/0x49c0 kernel/locking/lockdep.c:3415
lock_acquire+0x170/0x400 kernel/locking/lockdep.c:3907
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
spin_lock include/linux/spinlock.h:329 [inline]
__queue_work+0x1f1/0x1100 kernel/workqueue.c:1419
queue_work_on+0x17e/0x1f0 kernel/workqueue.c:1488
pty_write+0x198/0x1f0 drivers/tty/pty.c:125
process_output_block drivers/tty/n_tty.c:593 [inline]
n_tty_write+0x3ee/0x1080 drivers/tty/n_tty.c:2331
do_tty_write drivers/tty/tty_io.c:960 [inline]
tty_write+0x452/0x790 drivers/tty/tty_io.c:1044
__vfs_write+0xf7/0x760 fs/read_write.c:485
vfs_write+0x206/0x550 fs/read_write.c:549
ksys_write+0x12b/0x2a0 fs/read_write.c:599
do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x45c829
Code: 0d b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 db b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007fb6b2d80c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 000000000050a260 RCX: 000000000045c829
RDX: 00000000fffffedf RSI: 0000000020000000 RDI: 0000000000000004
RBP: 000000000078bf00 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 0000000000000c5a R14: 00000000004ca0fd R15: 00007fb6b2d816d4
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.0'.
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.0'.
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.1'.
netlink: 24 bytes leftover after parsing attributes in process `syz-executor.1'.
sock: process `syz-executor.3' is using obsolete setsockopt SO_BSDCOMPAT
netlink: 'syz-executor.3': attribute type 5 has an invalid length.
netlink: 'syz-executor.3': attribute type 5 has an invalid length.


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[syzbot]

Auto-closing this bug as obsolete.
Crashes did not happen for a while, no reproducer and no activity.