INFO: task hung in fsnotify_mark_destroy_workfn

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: b859aa7d ANDROID: squashfs: resolve merge conflict with 4...
git tree: android-4.14
console output: https://syzkaller.appspot.com/x/log.txt?x=14ede8be400000
kernel config: https://syzkaller.appspot.com/x/.config?x=c0bdd1b757a6ba0b
dashboard link: https://syzkaller.appspot.com/bug?extid=89456abd66e669de2add
compiler: gcc (GCC) 8.0.1 20180413 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+89456a...@syzkaller.appspotmail.com

hid-generic 0000:0000:0000.0002: unknown main item tag 0x0
hid-generic 0000:0000:0000.0002: unknown main item tag 0x0
hid-generic 0000:0000:0000.0002: unknown main item tag 0x0
hid-generic 0000:0000:0000.0002: unknown main item tag 0x0
hid-generic 0000:0000:0000.0002: unknown main item tag 0x0
INFO: task kworker/u4:0:5 blocked for more than 140 seconds.
Not tainted 4.14.68+ #4
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/u4:0 D26384 5 2 0x80000000
Workqueue: events_unbound fsnotify_mark_destroy_workfn
Call Trace:
schedule+0x7f/0x1b0 kernel/sched/core.c:3490
schedule_timeout+0x710/0xe60 kernel/time/timer.c:1718
do_wait_for_common kernel/sched/completion.c:91 [inline]
__wait_for_common kernel/sched/completion.c:112 [inline]
wait_for_common+0x3bc/0x4e0 kernel/sched/completion.c:123
__synchronize_srcu+0x121/0x200 kernel/rcu/srcutree.c:898
fsnotify_mark_destroy_workfn+0xfc/0x310 fs/notify/mark.c:757
process_one_work+0x86e/0x15c0 kernel/workqueue.c:2114
worker_thread+0xdc/0x1000 kernel/workqueue.c:2248
kthread+0x348/0x420 kernel/kthread.c:232
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:402
INFO: task kworker/u4:4:2459 blocked for more than 140 seconds.
Not tainted 4.14.68+ #4
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/u4:4 D28888 2459 2 0x80000000
Workqueue: events_unbound fsnotify_connector_destroy_workfn
Call Trace:
schedule+0x7f/0x1b0 kernel/sched/core.c:3490
schedule_timeout+0x710/0xe60 kernel/time/timer.c:1718
do_wait_for_common kernel/sched/completion.c:91 [inline]
__wait_for_common kernel/sched/completion.c:112 [inline]
wait_for_common+0x3bc/0x4e0 kernel/sched/completion.c:123
__synchronize_srcu+0x121/0x200 kernel/rcu/srcutree.c:898
fsnotify_connector_destroy_workfn+0x3f/0xa0 fs/notify/mark.c:156
process_one_work+0x86e/0x15c0 kernel/workqueue.c:2114
worker_thread+0xdc/0x1000 kernel/workqueue.c:2248
kthread+0x348/0x420 kernel/kthread.c:232
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:402
INFO: task kworker/u4:5:2500 blocked for more than 140 seconds.
Not tainted 4.14.68+ #4
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/u4:5 D27616 2500 2 0x80000000
Workqueue: netns cleanup_net
Call Trace:
schedule+0x7f/0x1b0 kernel/sched/core.c:3490
_synchronize_rcu_expedited+0x5af/0x850 kernel/rcu/tree_exp.h:615
synchronize_net+0x2b/0x40 net/core/dev.c:8186
rollback_registered_many+0x543/0xac0 net/core/dev.c:7170
unregister_netdevice_many+0x43/0x210 net/core/dev.c:8231
ip6_tnl_destroy_tunnels net/ipv6/ip6_tunnel.c:2208 [inline]
ip6_tnl_exit_net+0x3c8/0x580 net/ipv6/ip6_tunnel.c:2255
ops_exit_list.isra.3+0xa8/0x150 net/core/net_namespace.c:142
cleanup_net+0x3e9/0x880 net/core/net_namespace.c:483
process_one_work+0x86e/0x15c0 kernel/workqueue.c:2114
worker_thread+0xdc/0x1000 kernel/workqueue.c:2248
kthread+0x348/0x420 kernel/kthread.c:232
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:402
INFO: task syz-executor1:6798 blocked for more than 140 seconds.
Not tainted 4.14.68+ #4
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor1 D28880 6798 2080 0x00000004
Call Trace:
schedule+0x7f/0x1b0 kernel/sched/core.c:3490
schedule_timeout+0x710/0xe60 kernel/time/timer.c:1718
do_wait_for_common kernel/sched/completion.c:91 [inline]
__wait_for_common kernel/sched/completion.c:112 [inline]
wait_for_common+0x3bc/0x4e0 kernel/sched/completion.c:123
__synchronize_srcu+0x121/0x200 kernel/rcu/srcutree.c:898
debugfs_remove+0xca/0x110 fs/debugfs/inode.c:664
hid_debug_unregister+0x7c/0xf0 drivers/hid/hid-debug.c:1245
hid_remove_device drivers/hid/hid-core.c:3032 [inline]
hid_destroy_device+0xe0/0x150 drivers/hid/hid-core.c:3050
uhid_dev_destroy+0x96/0xf0 drivers/hid/uhid.c:568
uhid_char_release+0x4c/0xa0 drivers/hid/uhid.c:643
__fput+0x25e/0x6f0 fs/file_table.c:210
task_work_run+0x116/0x190 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:191 [inline]
exit_to_usermode_loop+0x12e/0x150 arch/x86/entry/common.c:163
prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
syscall_return_slowpath arch/x86/entry/common.c:267 [inline]
do_syscall_64+0x35d/0x4b0 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x410c51
RSP: 002b:00007ffd51a69680 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000410c51
RDX: 0000000000000001 RSI: 0000000000730b40 RDI: 0000000000000003
RBP: 0000000000000000 R08: ffffffffffffffff R09: ffffffffffffffff
R10: 00007ffd51a695b0 R11: 0000000000000293 R12: 0000000000000005
R13: 000000000001e7c5 R14: 0000000000000013 R15: badc0ffeebadface
INFO: task syz-executor7:6809 blocked for more than 140 seconds.
Not tainted 4.14.68+ #4
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor7 D29656 6809 2075 0x00000004
Call Trace:
schedule+0x7f/0x1b0 kernel/sched/core.c:3490
schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:3548
__mutex_lock_common kernel/locking/mutex.c:833 [inline]
__mutex_lock+0x521/0x1480 kernel/locking/mutex.c:893
ipv6_route_ioctl+0x1bb/0x260 net/ipv6/route.c:2675
inet6_ioctl+0xe7/0x1d0 net/ipv6/af_inet6.c:539
sock_do_ioctl+0x65/0xb0 net/socket.c:973
sock_ioctl+0x263/0x430 net/socket.c:1070
vfs_ioctl fs/ioctl.c:46 [inline]
file_ioctl fs/ioctl.c:500 [inline]
do_vfs_ioctl+0x1a0/0x1030 fs/ioctl.c:684
SYSC_ioctl fs/ioctl.c:701 [inline]
SyS_ioctl+0x7e/0xb0 fs/ioctl.c:692
do_syscall_64+0x19b/0x4b0 arch/x86/entry/common.c:289
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x457099
RSP: 002b:00007f50d4c8bc78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f50d4c8c6d4 RCX: 0000000000457099
RDX: 0000000020000bc0 RSI: 000000000000890b RDI: 0000000000000003
RBP: 00000000009300a0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000004d2610 R14: 00000000004c79fa R15: 0000000000000000
INFO: task syz-executor7:6834 blocked for more than 140 seconds.
Not tainted 4.14.68+ #4
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor7 D28552 6834 6809 0x00000004
Call Trace:
schedule+0x7f/0x1b0 kernel/sched/core.c:3490
schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:3548
__mutex_lock_common kernel/locking/mutex.c:833 [inline]
__mutex_lock+0x521/0x1480 kernel/locking/mutex.c:893
ipv6_route_ioctl+0x1bb/0x260 net/ipv6/route.c:2675
inet6_ioctl+0xe7/0x1d0 net/ipv6/af_inet6.c:539
sock_do_ioctl+0x65/0xb0 net/socket.c:973
sock_ioctl+0x263/0x430 net/socket.c:1070
vfs_ioctl fs/ioctl.c:46 [inline]
file_ioctl fs/ioctl.c:500 [inline]
do_vfs_ioctl+0x1a0/0x1030 fs/ioctl.c:684
SYSC_ioctl fs/ioctl.c:701 [inline]
SyS_ioctl+0x7e/0xb0 fs/ioctl.c:692
do_syscall_64+0x19b/0x4b0 arch/x86/entry/common.c:289
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x457099
RSP: 002b:00007f50d4c8bc78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f50d4c8c6d4 RCX: 0000000000457099
RDX: 0000000020000bc0 RSI: 000000000000890b RDI: 0000000000000007
RBP: 00000000009300a0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000004d2610 R14: 00000000004c79fa R15: 0000000000000000
INFO: task syz-executor4:6858 blocked for more than 140 seconds.
Not tainted 4.14.68+ #4
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor4 D29992 6858 2066 0x00000004
Call Trace:
schedule+0x7f/0x1b0 kernel/sched/core.c:3490
schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:3548
__mutex_lock_common kernel/locking/mutex.c:833 [inline]
__mutex_lock+0x521/0x1480 kernel/locking/mutex.c:893
dev_ioctl+0x281/0xce0 net/core/dev_ioctl.c:421
sock_do_ioctl+0x92/0xb0 net/socket.c:980
sock_ioctl+0x263/0x430 net/socket.c:1070
vfs_ioctl fs/ioctl.c:46 [inline]
file_ioctl fs/ioctl.c:500 [inline]
do_vfs_ioctl+0x1a0/0x1030 fs/ioctl.c:684
SYSC_ioctl fs/ioctl.c:701 [inline]
SyS_ioctl+0x7e/0xb0 fs/ioctl.c:692
do_syscall_64+0x19b/0x4b0 arch/x86/entry/common.c:289
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x457099
RSP: 002b:00007fbca49c0c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007fbca49c16d4 RCX: 0000000000457099
RDX: 0000000000400200 RSI: 0000000000008912 RDI: 0000000000000004
RBP: 0000000000930140 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000004cfd78 R14: 00000000004c5f16 R15: 0000000000000001
INFO: task syz-executor4:6869 blocked for more than 140 seconds.
Not tainted 4.14.68+ #4
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor4 D29768 6869 2066 0x00000004
Call Trace:
schedule+0x7f/0x1b0 kernel/sched/core.c:3490
schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:3548
__mutex_lock_common kernel/locking/mutex.c:833 [inline]
__mutex_lock+0x521/0x1480 kernel/locking/mutex.c:893
dev_ioctl+0x281/0xce0 net/core/dev_ioctl.c:421
sock_do_ioctl+0x92/0xb0 net/socket.c:980
sock_ioctl+0x263/0x430 net/socket.c:1070
vfs_ioctl fs/ioctl.c:46 [inline]
file_ioctl fs/ioctl.c:500 [inline]
do_vfs_ioctl+0x1a0/0x1030 fs/ioctl.c:684
SYSC_ioctl fs/ioctl.c:701 [inline]
SyS_ioctl+0x7e/0xb0 fs/ioctl.c:692
do_syscall_64+0x19b/0x4b0 arch/x86/entry/common.c:289
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x457099
RSP: 002b:00007fbca499fc78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007fbca49a06d4 RCX: 0000000000457099
RDX: 0000000000400200 RSI: 0000000000008912 RDI: 0000000000000009
RBP: 00000000009301e0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000004cfd78 R14: 00000000004c5f16 R15: 0000000000000002
INFO: task syz-executor5:6854 blocked for more than 140 seconds.
Not tainted 4.14.68+ #4
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor5 D29160 6854 2074 0x00000004
Call Trace:
schedule+0x7f/0x1b0 kernel/sched/core.c:3490
schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:3548
__mutex_lock_common kernel/locking/mutex.c:833 [inline]
__mutex_lock+0x521/0x1480 kernel/locking/mutex.c:893
dev_ioctl+0x281/0xce0 net/core/dev_ioctl.c:421
sock_do_ioctl+0x92/0xb0 net/socket.c:980
sock_ioctl+0x263/0x430 net/socket.c:1070
vfs_ioctl fs/ioctl.c:46 [inline]
file_ioctl fs/ioctl.c:500 [inline]
do_vfs_ioctl+0x1a0/0x1030 fs/ioctl.c:684
SYSC_ioctl fs/ioctl.c:701 [inline]
SyS_ioctl+0x7e/0xb0 fs/ioctl.c:692
do_syscall_64+0x19b/0x4b0 arch/x86/entry/common.c:289
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x457099
RSP: 002b:00007fa96a225c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007fa96a2266d4 RCX: 0000000000457099
RDX: 0000000000400200 RSI: 0000000000008912 RDI: 0000000000000005
RBP: 00000000009300a0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000004cfd78 R14: 00000000004c5f16 R15: 0000000000000000
INFO: task syz-executor5:6859 blocked for more than 140 seconds.
Not tainted 4.14.68+ #4
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor5 D28584 6859 2074 0x00000004
Call Trace:
schedule+0x7f/0x1b0 kernel/sched/core.c:3490
schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:3548
__mutex_lock_common kernel/locking/mutex.c:833 [inline]
__mutex_lock+0x521/0x1480 kernel/locking/mutex.c:893
dev_ioctl+0x281/0xce0 net/core/dev_ioctl.c:421
sock_do_ioctl+0x92/0xb0 net/socket.c:980
sock_ioctl+0x263/0x430 net/socket.c:1070
vfs_ioctl fs/ioctl.c:46 [inline]
file_ioctl fs/ioctl.c:500 [inline]
do_vfs_ioctl+0x1a0/0x1030 fs/ioctl.c:684
SYSC_ioctl fs/ioctl.c:701 [inline]
SyS_ioctl+0x7e/0xb0 fs/ioctl.c:692
do_syscall_64+0x19b/0x4b0 arch/x86/entry/common.c:289
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x457099
RSP: 002b:00007fa96a204c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007fa96a2056d4 RCX: 0000000000457099
RDX: 0000000000400200 RSI: 0000000000008912 RDI: 0000000000000008
RBP: 0000000000930140 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000004cfd78 R14: 00000000004c5f16 R15: 0000000000000001

Showing all locks held in the system:
2 locks held by kworker/u4:0/5:
#0: ("events_unbound"){+.+.}, at: [<ffffffff8b727237>]
process_one_work+0x787/0x15c0 kernel/workqueue.c:2085
#1: ((reaper_work).work){+.+.}, at: [<ffffffff8b72726f>]
process_one_work+0x7bf/0x15c0 kernel/workqueue.c:2089
1 lock held by khungtaskd/23:
#0: (tasklist_lock){.+.+}, at: [<ffffffff8b8019d7>]
debug_show_all_locks+0x74/0x20f kernel/locking/lockdep.c:4541
1 lock held by rsyslogd/1863:
#0: (&f->f_pos_lock){+.+.}, at: [<ffffffff8bbbcd02>]
__fdget_pos+0xa2/0xc0 fs/file.c:768
2 locks held by getty/1990:
#0: (&tty->ldisc_sem){++++}, at: [<ffffffff8c31c980>]
tty_ldisc_ref_wait+0x20/0x80 drivers/tty/tty_ldisc.c:275
#1: (&ldata->atomic_read_lock){+.+.}, at: [<ffffffff8c317eff>]
n_tty_read+0x1ff/0x15e0 drivers/tty/n_tty.c:2142
2 locks held by kworker/u4:4/2459:
#0: ("events_unbound"){+.+.}, at: [<ffffffff8b727237>]
process_one_work+0x787/0x15c0 kernel/workqueue.c:2085
#1: (connector_reaper_work){+.+.}, at: [<ffffffff8b72726f>]
process_one_work+0x7bf/0x15c0 kernel/workqueue.c:2089
5 locks held by kworker/u4:5/2500:
#0: ("%s""netns"){+.+.}, at: [<ffffffff8b727237>]
process_one_work+0x787/0x15c0 kernel/workqueue.c:2085
#1: (net_cleanup_work){+.+.}, at: [<ffffffff8b72726f>]
process_one_work+0x7bf/0x15c0 kernel/workqueue.c:2089
#2: (net_mutex){+.+.}, at: [<ffffffff8c8d2b1c>] cleanup_net+0x14c/0x880
net/core/net_namespace.c:449
#3: (rtnl_mutex){+.+.}, at: [<ffffffff8cdadec7>]
ip6_tnl_exit_net+0x77/0x580 net/ipv6/ip6_tunnel.c:2254
#4: (rcu_preempt_state.exp_mutex){+.+.}, at: [<ffffffff8b849726>]
exp_funnel_lock kernel/rcu/tree_exp.h:272 [inline]
#4: (rcu_preempt_state.exp_mutex){+.+.}, at: [<ffffffff8b849726>]
_synchronize_rcu_expedited+0x306/0x850 kernel/rcu/tree_exp.h:596
1 lock held by syz-executor7/6809:
#0: (rtnl_mutex){+.+.}, at: [<ffffffff8ccde2eb>]
ipv6_route_ioctl+0x1bb/0x260 net/ipv6/route.c:2675
1 lock held by syz-executor7/6834:
#0: (rtnl_mutex){+.+.}, at: [<ffffffff8ccde2eb>]
ipv6_route_ioctl+0x1bb/0x260 net/ipv6/route.c:2675
1 lock held by syz-executor4/6858:
#0: (rtnl_mutex){+.+.}, at: [<ffffffff8c960e01>] dev_ioctl+0x281/0xce0
net/core/dev_ioctl.c:421
1 lock held by syz-executor4/6869:
#0: (rtnl_mutex){+.+.}, at: [<ffffffff8c960e01>] dev_ioctl+0x281/0xce0
net/core/dev_ioctl.c:421
1 lock held by syz-executor5/6854:
#0: (rtnl_mutex){+.+.}, at: [<ffffffff8c960e01>] dev_ioctl+0x281/0xce0
net/core/dev_ioctl.c:421
1 lock held by syz-executor5/6859:
#0: (rtnl_mutex){+.+.}, at: [<ffffffff8c960e01>] dev_ioctl+0x281/0xce0
net/core/dev_ioctl.c:421

=============================================

NMI backtrace for cpu 0
CPU: 0 PID: 23 Comm: khungtaskd Not tainted 4.14.68+ #4
Call Trace:
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0xb9/0x11b lib/dump_stack.c:53
nmi_cpu_backtrace.cold.0+0x47/0x85 lib/nmi_backtrace.c:101
nmi_trigger_cpumask_backtrace+0x121/0x146 lib/nmi_backtrace.c:62
trigger_all_cpu_backtrace include/linux/nmi.h:138 [inline]
check_hung_uninterruptible_tasks kernel/hung_task.c:196 [inline]
watchdog+0x574/0xa70 kernel/hung_task.c:252
kthread+0x348/0x420 kernel/kthread.c:232
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:402
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 6805 Comm: syz-executor6 Not tainted 4.14.68+ #4
task: ffff88019bccaf00 task.stack: ffff8801bb1d8000
RIP: 0033:0x4023c4
RSP: 002b:00007f1e07e10690 EFLAGS: 00000296
RAX: 00000000a1738bc0 RBX: 0000000000000007 RCX: 0000000000457099
RDX: 0000000000000000 RSI: 00007f1e07e106c0 RDI: 0000000000000007
RBP: 00000000009300a0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000004d7008 R14: 00000000004ca112 R15: 0000000000000000
FS: 00007f1e07e11700(0000) GS:ffff8801dbb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9c81678330 CR3: 00000001c7df2003 CR4: 00000000001606a0


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[syzbot]

Auto-closing this bug as obsolete.
Crashes did not happen for a while, no reproducer and no activity.