INFO: task hung in blk_trace_remove

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: fb33c651 Linux 5.6-rc6
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=140688d3e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=9f894bd92023de02
dashboard link: https://syzkaller.appspot.com/bug?extid=c07afbbb410e9f712273
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+c07afb...@syzkaller.appspotmail.com

INFO: task syz-executor.4:7237 blocked for more than 143 seconds.
Not tainted 5.6.0-rc6-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor.4 D26576 7237 9609 0x00004004
Call Trace:
schedule+0xd0/0x2a0 kernel/sched/core.c:4154
schedule_preempt_disabled+0xf/0x20 kernel/sched/core.c:4213
__mutex_lock_common kernel/locking/mutex.c:1033 [inline]
__mutex_lock+0x7ab/0x13c0 kernel/locking/mutex.c:1103
blk_trace_remove+0x1e/0x40 kernel/trace/blktrace.c:361
sg_ioctl_common+0x221/0x2710 drivers/scsi/sg.c:1125
sg_ioctl+0x8f/0x120 drivers/scsi/sg.c:1159
vfs_ioctl fs/ioctl.c:47 [inline]
ksys_ioctl+0x11a/0x180 fs/ioctl.c:763
__do_sys_ioctl fs/ioctl.c:772 [inline]
__se_sys_ioctl fs/ioctl.c:770 [inline]
__x64_sys_ioctl+0x6f/0xb0 fs/ioctl.c:770
do_syscall_64+0xf6/0x7d0 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x45c849
Code: Bad RIP value.
RSP: 002b:00007f5ba5a3bc78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f5ba5a3c6d4 RCX: 000000000045c849
RDX: 0000000000000000 RSI: 0000000000001276 RDI: 0000000000000003
RBP: 000000000076bf00 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 000000000000022b R14: 00000000004c4526 R15: 000000000076bf0c
INFO: task syz-executor.4:7266 blocked for more than 146 seconds.
Not tainted 5.6.0-rc6-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor.4 D27752 7266 9609 0x00004004
Call Trace:
schedule+0xd0/0x2a0 kernel/sched/core.c:4154
schedule_preempt_disabled+0xf/0x20 kernel/sched/core.c:4213
__mutex_lock_common kernel/locking/mutex.c:1033 [inline]
__mutex_lock+0x7ab/0x13c0 kernel/locking/mutex.c:1103
blk_trace_setup+0x2f/0x60 kernel/trace/blktrace.c:588
sg_ioctl_common+0x2f2/0x2710 drivers/scsi/sg.c:1116
sg_ioctl+0x8f/0x120 drivers/scsi/sg.c:1159
vfs_ioctl fs/ioctl.c:47 [inline]
ksys_ioctl+0x11a/0x180 fs/ioctl.c:763
__do_sys_ioctl fs/ioctl.c:772 [inline]
__se_sys_ioctl fs/ioctl.c:770 [inline]
__x64_sys_ioctl+0x6f/0xb0 fs/ioctl.c:770
do_syscall_64+0xf6/0x7d0 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x45c849
Code: Bad RIP value.
RSP: 002b:00007f5ba5a1ac78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f5ba5a1b6d4 RCX: 000000000045c849
RDX: 0000000020000080 RSI: 00000000c0481273 RDI: 0000000000000008
RBP: 000000000076bfa0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 0000000000000228 R14: 00000000004c44eb R15: 000000000076bfac
INFO: task syz-executor.5:7265 blocked for more than 149 seconds.
Not tainted 5.6.0-rc6-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor.5 D26736 7265 9613 0x00004004
Call Trace:
schedule+0xd0/0x2a0 kernel/sched/core.c:4154
schedule_preempt_disabled+0xf/0x20 kernel/sched/core.c:4213
__mutex_lock_common kernel/locking/mutex.c:1033 [inline]
__mutex_lock+0x7ab/0x13c0 kernel/locking/mutex.c:1103
blk_trace_setup+0x2f/0x60 kernel/trace/blktrace.c:588
sg_ioctl_common+0x2f2/0x2710 drivers/scsi/sg.c:1116
sg_ioctl+0x8f/0x120 drivers/scsi/sg.c:1159
vfs_ioctl fs/ioctl.c:47 [inline]
ksys_ioctl+0x11a/0x180 fs/ioctl.c:763
__do_sys_ioctl fs/ioctl.c:772 [inline]
__se_sys_ioctl fs/ioctl.c:770 [inline]
__x64_sys_ioctl+0x6f/0xb0 fs/ioctl.c:770
do_syscall_64+0xf6/0x7d0 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x45c849
Code: Bad RIP value.
RSP: 002b:00007f576f48ac78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f576f48b6d4 RCX: 000000000045c849
RDX: 0000000020000080 RSI: 00000000c0481273 RDI: 0000000000000006


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[Dmitry Vyukov]

On Sat, Mar 21, 2020 at 8:12 AM syzbot
<syzbot+c07afb...@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following crash on:
>
> HEAD commit: fb33c651 Linux 5.6-rc6
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=140688d3e00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=9f894bd92023de02
> dashboard link: https://syzkaller.appspot.com/bug?extid=c07afbbb410e9f712273
> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
>
> Unfortunately, I don't have any reproducer for this crash yet.
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+c07afb...@syzkaller.appspotmail.com

+ashmem maintainers

From the log, ashmem spews an infinite sequence of some errors which
probably stall the machine:

[ 1094.685541][ T9460] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641097791
[ 1094.701750][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641383141
[ 1094.708003][ T9576] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641233534
[ 1094.723597][ T1876] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641083174
[ 1094.730025][ T9460] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641098111
[ 1094.745592][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641383221
[ 1094.753153][ T9576] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641233534
[ 1094.767903][ T1876] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641084454
[ 1094.819845][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641526579
[ 1094.830116][ T9460] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641098111
[ 1094.845813][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641527219
[ 1094.852188][ T9460] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641098751
[ 1094.871456][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641527219
[ 1094.874256][ T9460] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641098751
[ 1094.890195][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641528499
[ 1094.895999][ T1876] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641087014
[ 1094.912931][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641528499
[ 1094.918797][ T9460] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641100031
[ 1094.936636][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641531059
[ 1094.941391][ T1876] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641081897
[ 1094.957463][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641531059
[ 1094.963568][ T9460] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641100031
[ 1094.980233][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641536179
[ 1094.985311][ T1876] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641081899
[ 1095.001393][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641536179
[ 1095.008146][ T9460] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641102591
[ 1095.024425][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641525942
[ 1095.029644][ T1876] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641081904
[ 1095.046399][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641525942
[ 1095.052377][ T9460] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641102591
[ 1095.068000][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641525944
[ 1095.073934][ T1876] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641081914
[ 1095.091076][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641525944
[ 1095.096925][ T9460] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641107711
[ 1095.112305][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641525949
[ 1095.118570][ T1876] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641081934
[ 1095.135413][ T7239] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641525949
[ 1095.141206][ T9460] vmscan: shrink_slab:
ashmem_shrink_scan+0x0/0x500 negative objects to delete
nr=-6917529027641107711

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/0000000000000bec9805a1581f05%40google.com.

[Hillf Danton]

On Sat, 21 Mar 2020 00:12:13 -0700

Defer destroying block trace in task context if it makes no sense
to replace synchronize_rcu() with call_rcu().

--- a/include/linux/blktrace_api.h
+++ b/include/linux/blktrace_api.h
@@ -11,6 +11,7 @@
#if defined(CONFIG_BLK_DEV_IO_TRACE)

#include <linux/sysfs.h>
+#include <linux/workqueue.h>

struct blk_trace {
int trace_state;
@@ -27,6 +28,7 @@ struct blk_trace {
struct dentry *msg_file;
struct list_head running_list;
atomic_t dropped;
+ struct work_struct destroy;
};

struct blkcg;
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -317,6 +317,14 @@ static void blk_trace_free(struct blk_tr
kfree(bt);
}

+static void blk_trace_destroy_workfn(struct work_struct *work)
+{
+ struct blk_trace *bt = container_of(work, struct blk_trace, destroy);
+
+ synchronize_rcu();
+ blk_trace_free(bt);
+}
+
static void get_probe_ref(void)
{
mutex_lock(&blk_probe_mutex);
@@ -335,9 +343,9 @@ static void put_probe_ref(void)

static void blk_trace_cleanup(struct blk_trace *bt)
{
- synchronize_rcu();
- blk_trace_free(bt);
put_probe_ref();
+ INIT_WORK(&bt->destroy, blk_trace_destroy_workfn);
+ schedule_work(&bt->destroy);
}

static int __blk_trace_remove(struct request_queue *q)
@@ -1641,9 +1649,7 @@ static int blk_trace_remove_queue(struct
if (bt == NULL)
return -EINVAL;

- put_probe_ref();
- synchronize_rcu();
- blk_trace_free(bt);
+ blk_trace_cleanup(bt);
return 0;
}

[Todd Kjos]

On Sat, Mar 21, 2020 at 12:15 AM Dmitry Vyukov <dvy...@google.com> wrote:
>
> On Sat, Mar 21, 2020 at 8:12 AM syzbot
> <syzbot+c07afb...@syzkaller.appspotmail.com> wrote:
> >
> > Hello,
> >
> > syzbot found the following crash on:
> >
> > HEAD commit: fb33c651 Linux 5.6-rc6
> > git tree: upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=140688d3e00000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=9f894bd92023de02
> > dashboard link: https://syzkaller.appspot.com/bug?extid=c07afbbb410e9f712273
> > compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> >
> > Unfortunately, I don't have any reproducer for this crash yet.
> >
> > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > Reported-by: syzbot+c07afb...@syzkaller.appspotmail.com
>
> +ashmem maintainers

We'll have a look.

[syzbot]

Auto-closing this bug as obsolete.
Crashes did not happen for a while, no reproducer and no activity.