BUG: workqueue lockup (4)

26 views
Skip to first unread message

syzbot

unread,
Aug 4, 2018, 5:59:03 PM8/4/18
to syzkaller-upst...@googlegroups.com
Hello,

syzbot found the following crash on:

HEAD commit: d1e0b8e0cb7a Add linux-next specific files for 20180725
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=13263558400000
kernel config: https://syzkaller.appspot.com/x/.config?x=eef3552c897e4d33
dashboard link: https://syzkaller.appspot.com/bug?extid=4175b351ead922df4043
compiler: gcc (GCC) 8.0.1 20180413 (experimental)
CC: [christia...@ubuntu.com da...@davemloft.net
dsa...@gmail.com f...@strlen.de jakub.k...@netronome.com
jb...@redhat.com ktk...@virtuozzo.com linux-...@vger.kernel.org
lucie...@gmail.com net...@vger.kernel.org]

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+4175b3...@syzkaller.appspotmail.com

QAT: Invalid ioctl
QAT: Invalid ioctl
IPVS: ftp: loaded support on port[0] = 21
BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 174s!
BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=-20 stuck for
173s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x0
pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=6/256
pending: defense_work_handler, perf_sched_delayed,
defense_work_handler, defense_work_handler, defense_work_handler, cache_reap
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=6/256
in-flight: 5:linkwatch_event, 13:switchdev_deferred_process_work
switchdev_deferred_process_work
pending: defense_work_handler, defense_work_handler,
defense_work_handler
workqueue events_highpri: flags=0x10
pwq 3: cpus=1 node=0 flags=0x0 nice=-20 active=1/256
pending: flush_backlog BAR(3086)
workqueue events_power_efficient: flags=0x80
pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
pending: gc_worker
workqueue mm_percpu_wq: flags=0x8
pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
pending: vmstat_update
workqueue kblockd: flags=0x18
pwq 3: cpus=1 node=0 flags=0x0 nice=-20 active=1/256
pending: blk_mq_timeout_work
workqueue ipv6_addrconf: flags=0x40008
pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1
pending: addrconf_dad_work
delayed: addrconf_dad_work, addrconf_dad_work, addrconf_dad_work
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1
in-flight: 1880:addrconf_dad_work
delayed: addrconf_dad_work, addrconf_dad_work, addrconf_dad_work,
addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work,
addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work,
addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work,
addrconf_dad_work, addrconf_dad_work, addrconf_verify_work
pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=0s workers=6 idle: 8134 26530
4757
INFO: task kworker/0:0:5 blocked for more than 140 seconds.
Not tainted 4.18.0-rc6-next-20180725+ #18
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/0:0 D18608 5 2 0x80000000
Workqueue: events linkwatch_event
Call Trace:
context_switch kernel/sched/core.c:2820 [inline]
__schedule+0x87c/0x1ed0 kernel/sched/core.c:3468
schedule+0xfb/0x450 kernel/sched/core.c:3512
schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:3570
__mutex_lock_common kernel/locking/mutex.c:1003 [inline]
__mutex_lock+0xbf5/0x1680 kernel/locking/mutex.c:1073
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1088
rtnl_lock+0x17/0x20 net/core/rtnetlink.c:77
linkwatch_event+0x5f/0xe0 net/core/link_watch.c:236
process_one_work+0xc73/0x1ba0 kernel/workqueue.c:2153
worker_thread+0x189/0x13c0 kernel/workqueue.c:2296
kthread+0x345/0x410 kernel/kthread.c:246
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:415
INFO: task kworker/0:1:13 blocked for more than 140 seconds.
Not tainted 4.18.0-rc6-next-20180725+ #18
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/0:1 D20656 13 2 0x80000000
Workqueue: events switchdev_deferred_process_work
Call Trace:
context_switch kernel/sched/core.c:2820 [inline]
__schedule+0x87c/0x1ed0 kernel/sched/core.c:3468
schedule+0xfb/0x450 kernel/sched/core.c:3512
schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:3570
__mutex_lock_common kernel/locking/mutex.c:1003 [inline]
__mutex_lock+0xbf5/0x1680 kernel/locking/mutex.c:1073
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1088
rtnl_lock+0x17/0x20 net/core/rtnetlink.c:77
switchdev_deferred_process_work+0xe/0x20 net/switchdev/switchdev.c:150
process_one_work+0xc73/0x1ba0 kernel/workqueue.c:2153
worker_thread+0x189/0x13c0 kernel/workqueue.c:2296
kthread+0x345/0x410 kernel/kthread.c:246
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:415
INFO: task kworker/0:2:1880 blocked for more than 140 seconds.
Not tainted 4.18.0-rc6-next-20180725+ #18
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/0:2 D20672 1880 2 0x80000000
Workqueue: ipv6_addrconf addrconf_dad_work
Call Trace:
context_switch kernel/sched/core.c:2820 [inline]
__schedule+0x87c/0x1ed0 kernel/sched/core.c:3468
schedule+0xfb/0x450 kernel/sched/core.c:3512
schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:3570
__mutex_lock_common kernel/locking/mutex.c:1003 [inline]
__mutex_lock+0xbf5/0x1680 kernel/locking/mutex.c:1073
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1088
rtnl_lock+0x17/0x20 net/core/rtnetlink.c:77
addrconf_dad_work+0xb6/0x1310 net/ipv6/addrconf.c:3973
process_one_work+0xc73/0x1ba0 kernel/workqueue.c:2153
worker_thread+0x189/0x13c0 kernel/workqueue.c:2296
kthread+0x345/0x410 kernel/kthread.c:246
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:415
INFO: task syz-executor5:3086 blocked for more than 140 seconds.
Not tainted 4.18.0-rc6-next-20180725+ #18
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor5 D19904 3086 2726 0x80000002
Call Trace:
context_switch kernel/sched/core.c:2820 [inline]
__schedule+0x87c/0x1ed0 kernel/sched/core.c:3468
schedule+0xfb/0x450 kernel/sched/core.c:3512
schedule_timeout+0x1cc/0x260 kernel/time/timer.c:1777
do_wait_for_common kernel/sched/completion.c:83 [inline]
__wait_for_common kernel/sched/completion.c:104 [inline]
wait_for_common kernel/sched/completion.c:115 [inline]
wait_for_completion+0x430/0x8d0 kernel/sched/completion.c:136
flush_work+0x531/0x900 kernel/workqueue.c:2918
flush_all_backlogs net/core/dev.c:5234 [inline]
rollback_registered_many+0x6ee/0x1250 net/core/dev.c:7909
rollback_registered+0x1e9/0x420 net/core/dev.c:7966
unregister_netdevice_queue+0x32f/0x660 net/core/dev.c:9010
unregister_netdevice include/linux/netdevice.h:2589 [inline]
__tun_detach+0x11d1/0x15e0 drivers/net/tun.c:728
tun_detach drivers/net/tun.c:745 [inline]
tun_chr_close+0xe3/0x180 drivers/net/tun.c:3271
__fput+0x376/0x8a0 fs/file_table.c:279
____fput+0x15/0x20 fs/file_table.c:312
task_work_run+0x1ec/0x2a0 kernel/task_work.c:113
exit_task_work include/linux/task_work.h:22 [inline]
do_exit+0x1b08/0x2750 kernel/exit.c:869
do_group_exit+0x177/0x440 kernel/exit.c:972
get_signal+0x88e/0x1970 kernel/signal.c:2467
do_signal+0x9c/0x21c0 arch/x86/kernel/signal.c:816
exit_to_usermode_loop+0x2e5/0x380 arch/x86/entry/common.c:162
prepare_exit_to_usermode arch/x86/entry/common.c:197 [inline]
syscall_return_slowpath arch/x86/entry/common.c:268 [inline]
do_syscall_64+0x6be/0x820 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x456a09
Code: 24 00 00 00 e8 e8 4e fd ff 0f 0b e8 c1 34 00 00 e9 3c ff ff ff cc cc
cc cc cc cc cc cc cc cc cc cc 64 48 8b 0c 25 f8 ff ff ff <48> 3b 61 10 76
40 48 83 ec 28 48 89 6c 24 20 48 8d 6c 24 20 48 8b
RSP: 002b:00007f0ddb1a3cf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: 0000000000000001 RBX: 0000000000930148 RCX: 0000000000456a09
RDX: 0000000000000003 RSI: 0000000000000001 RDI: 000000000093014c
RBP: 0000000000930140 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000093014c
R13: 00007fff5e6f660f R14: 00007f0ddb1a49c0 R15: 0000000000000001
INFO: lockdep is turned off.
NMI backtrace for cpu 0
CPU: 0 PID: 775 Comm: khungtaskd Not tainted 4.18.0-rc6-next-20180725+ #18
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x1c9/0x2b4 lib/dump_stack.c:113
nmi_cpu_backtrace.cold.3+0x48/0x88 lib/nmi_backtrace.c:101
nmi_trigger_cpumask_backtrace+0x151/0x192 lib/nmi_backtrace.c:62
arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
trigger_all_cpu_backtrace include/linux/nmi.h:144 [inline]
check_hung_uninterruptible_tasks kernel/hung_task.c:204 [inline]
watchdog+0xb39/0x10b0 kernel/hung_task.c:265
kthread+0x345/0x410 kernel/kthread.c:246
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:415
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 2826 Comm: syz-executor2 Not tainted 4.18.0-rc6-next-20180725+
#18
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
RIP: 0010:__lock_acquire+0xce/0x5020 kernel/locking/lockdep.c:3294
Code: c7 40 1c f2 f2 f2 f2 c7 40 20 04 f2 f2 f2 c7 40 24 f2 f2 f2 f2 c7 40
28 04 f2 f2 f2 c7 40 2c f2 f2 f2 f2 c7 40 30 00 f2 f2 f2 <c7> 40 34 f2 f2
f2 f2 c7 40 38 00 f2 f2 f2 c7 40 3c f2 f2 f2 f2 c7
RSP: 0018:ffff880191b4efd0 EFLAGS: 00000082
RAX: ffffed0032369e0e RBX: 1ffff10032369e76 RCX: 0000000000000000
RDX: dffffc0000000000 RSI: 0000000000000000 RDI: ffffffff87e12708
RBP: ffff880191b4f358 R08: 0000000000000001 R09: 0000000000000000
R10: fffffbfff0fc24e5 R11: ffffffff87e1272b R12: 0000000000000000
R13: ffffffff87e12708 R14: ffff8801956b6080 R15: 0000000000000000
FS: 00007f46ae05a700(0000) GS:ffff8801db100000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f46adfb4db8 CR3: 00000001c8b0c000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
lock_acquire+0x1e4/0x540 kernel/locking/lockdep.c:3924
write_seqcount_begin_nested include/linux/seqlock.h:377 [inline]
write_seqcount_begin include/linux/seqlock.h:382 [inline]
write_seqlock include/linux/seqlock.h:449 [inline]
lock_mount_hash fs/mount.h:122 [inline]
__detach_mounts+0xc3/0x430 fs/namespace.c:1533
detach_mounts fs/mount.h:110 [inline]
d_invalidate+0x168/0x2a0 fs/dcache.c:1594
lookup_fast+0xd61/0x12a0 fs/namei.c:1624
walk_component+0x13d/0x2630 fs/namei.c:1806
lookup_last fs/namei.c:2273 [inline]
path_lookupat.isra.45+0x212/0xc00 fs/namei.c:2318
filename_lookup+0x264/0x510 fs/namei.c:2348
user_path_at_empty+0x40/0x50 fs/namei.c:2608
user_path_at include/linux/namei.h:57 [inline]
vfs_statx+0x129/0x210 fs/stat.c:185
vfs_lstat include/linux/fs.h:3132 [inline]
__do_sys_newlstat+0x8f/0x110 fs/stat.c:350
__se_sys_newlstat fs/stat.c:344 [inline]
__x64_sys_newlstat+0x54/0x80 fs/stat.c:344
do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x456a09
Code: fd b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff
ff 0f 83 cb b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f46ae059c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000006
RAX: ffffffffffffffda RBX: 00007f46ae05a6d4 RCX: 0000000000456a09
RDX: 0000000000000000 RSI: 00000000200004c0 RDI: 0000000020000280
RBP: 0000000000930140 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000004d14f8 R14: 00000000004c6d78 R15: 0000000000000001


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
syzbot.

syzbot

unread,
Oct 7, 2018, 6:38:03 AM10/7/18
to syzkaller-upst...@googlegroups.com
syzbot has found a reproducer for the following crash on:

HEAD commit: c1d84a1b42ef Merge git://git.kernel.org/pub/scm/linux/kern..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=14c5f491400000
kernel config: https://syzkaller.appspot.com/x/.config?x=c0af03fe452b65fb
dashboard link: https://syzkaller.appspot.com/bug?extid=4175b351ead922df4043
compiler: gcc (GCC) 8.0.1 20180413 (experimental)
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=14514a6e400000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1025ebb9400000
CC: []

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+4175b3...@syzkaller.appspotmail.com

BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 204s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x0
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256
in-flight: 14:rtc_timer_do_work
pending: vmstat_shepherd, cache_reap, check_corruption
workqueue events_power_efficient: flags=0x80
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
pending: do_cache_clean
workqueue mm_percpu_wq: flags=0x8
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
pending: vmstat_update
workqueue kacpi_notify: flags=0x0
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1
pending: acpi_os_execute_deferred
pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=204s workers=2 idle: 2683

Dmitry Vyukov

unread,
Oct 7, 2018, 8:14:24 AM10/7/18
to syzbot, 'Dmitry Vyukov' via syzkaller-upstream-moderation
#syz upstream
> --
> You received this message because you are subscribed to the Google Groups
> "syzkaller-upstream-moderation" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to syzkaller-upstream-m...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/syzkaller-upstream-moderation/0000000000005f96540577a1195d%40google.com.
>
> For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages