BUG: corrupted list in cpu_stop_queue_work

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: 526674536360 Add linux-next specific files for 20180706
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=142e462c400000
kernel config: https://syzkaller.appspot.com/x/.config?x=c8d1cfc0cb798e48
dashboard link: https://syzkaller.appspot.com/bug?extid=d8a8e42dfba0454286ff
compiler: gcc (GCC) 8.0.1 20180413 (experimental)
syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=1149badc400000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+d8a8e4...@syzkaller.appspotmail.com

R10: 0000000020000000 R11: 0000000000000246 R12: 0000000000000004
R13: 00000000004bbd2c R14: 00000000004c9100 R15: 0000000000000006
IPv6: ADDRCONF(NETDEV_CHANGE): team0: link becomes ready
list_add corruption. prev->next should be next (ffff8801dae26708), but was
0000000000000000. (prev=ffff8801dae26740).
------------[ cut here ]------------
kernel BUG at lib/list_debug.c:28!
invalid opcode: 0000 [#1] SMP KASAN
CPU: 0 PID: 4685 Comm: syz-executor0 Not tainted 4.18.0-rc3-next-20180706+
#1
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
RIP: 0010:__list_add_valid.cold.0+0x23/0x25 lib/list_debug.c:26
Code: e8 9f 72 57 fe eb 97 48 89 d9 48 c7 c7 60 80 1a 88 e8 72 b8 ff fd 0f
0b 48 89 f1 48 c7 c7 20 81 1a 88 48 89 de e8 5e b8 ff fd <0f> 0b 4c 89 e2
48 89 de 48 c7 c7 60 82 1a 88 e8 4a b8 ff fd 0f 0b
RSP: 0018:ffff8801dae079c0 EFLAGS: 00010086
RAX: 0000000000000075 RBX: ffff8801dae26708 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff81634381 RDI: 0000000000000001
RBP: ffff8801dae079d8 R08: ffff8801d70ca380 R09: ffffed003b5c4fc0
R10: ffffed003b5c4fc0 R11: ffff8801dae27e07 R12: ffff8801dae26740
R13: ffff8801dae07ab8 R14: ffff8801dae26740 R15: 0000000000000001
FS: 000000000157c940(0000) GS:ffff8801dae00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002156000 CR3: 00000001b3a95000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<IRQ>
__list_add include/linux/list.h:60 [inline]
list_add_tail include/linux/list.h:93 [inline]
__cpu_stop_queue_work kernel/stop_machine.c:72 [inline]
cpu_stop_queue_work+0x248/0x460 kernel/stop_machine.c:87
stop_one_cpu_nowait+0xd3/0x100 kernel/stop_machine.c:341
watchdog_timer_fn+0x93/0x2e0 kernel/watchdog.c:367
__run_hrtimer kernel/time/hrtimer.c:1398 [inline]
__hrtimer_run_queues+0x3eb/0x10c0 kernel/time/hrtimer.c:1460
hrtimer_interrupt+0x2f3/0x750 kernel/time/hrtimer.c:1518
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1025 [inline]
smp_apic_timer_interrupt+0x165/0x730 arch/x86/kernel/apic/apic.c:1050
apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:867
</IRQ>
RIP: 0010:__sanitizer_cov_trace_pc+0x1/0x50 kernel/kcov.c:94
Code: dd 28 00 00 00 4d 39 d0 72 1b 49 83 c1 01 4a 89 7c 10 e0 4a 89 74 10
e8 4a 89 54 10 f0 4a 89 4c d8 20 4c 89 08 5d c3 66 90 55 <65> 48 8b 04 25
40 ee 01 00 65 8b 15 5f 73 85 7e 48 89 e5 81 e2 00
RSP: 0018:ffff8801d8d0e200 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
RAX: 1ffff1003b1a1c5f RBX: 0000000000000000 RCX: ffff8801d8d0e2e0
RDX: 0000000000000000 RSI: ffffffff819b082c RDI: ffff8801d8d0e2f8
RBP: ffff8801d8d0e5a8 R08: ffff8801d70ca380 R09: fffff94000ddc926
R10: fffff94000ddc926 R11: ffffea0006ee4937 R12: ffff8801d8d0e580
R13: 1ffff1003b1a1c68 R14: dffffc0000000000 R15: 0000000000000000
shmem_undo_range+0x38e/0x29a0 mm/shmem.c:795
shmem_truncate_range+0x27/0xa0 mm/shmem.c:971
shmem_evict_inode+0x3b2/0xcb0 mm/shmem.c:1071
evict+0x4ae/0x990 fs/inode.c:558
iput_final fs/inode.c:1508 [inline]
iput+0x635/0xaa0 fs/inode.c:1534
dentry_unlink_inode+0x4ae/0x640 fs/dcache.c:377
__dentry_kill+0x44c/0x7a0 fs/dcache.c:569
dentry_kill+0xc9/0x5a0 fs/dcache.c:688
dput.part.26+0x66b/0x7a0 fs/dcache.c:849
dput+0x15/0x20 fs/dcache.c:831
__fput+0x558/0x930 fs/file_table.c:235
____fput+0x15/0x20 fs/file_table.c:251
task_work_run+0x1ec/0x2a0 kernel/task_work.c:113
exit_task_work include/linux/task_work.h:22 [inline]
do_exit+0x1b08/0x2750 kernel/exit.c:869
do_group_exit+0x177/0x440 kernel/exit.c:972
__do_sys_exit_group kernel/exit.c:983 [inline]
__se_sys_exit_group kernel/exit.c:981 [inline]
__x64_sys_exit_group+0x3e/0x50 kernel/exit.c:981
do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x455ba9
Code: Bad RIP value.
RSP: 002b:00007ffc13494cf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000455ba9
RDX: 00000000000000a4 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000000006e R11: 0000000000000246 R12: 00000000000003e0
R13: 0000000000000000 R14: 00007ffc134953c0 R15: 0000000000000000
Modules linked in:
Dumping ftrace buffer:
(ftrace buffer empty)
---[ end trace 59c8d15dab110d89 ]---
RIP: 0010:__list_add_valid.cold.0+0x23/0x25 lib/list_debug.c:26
Code: e8 9f 72 57 fe eb 97 48 89 d9 48 c7 c7 60 80 1a 88 e8 72 b8 ff fd 0f
0b 48 89 f1 48 c7 c7 20 81 1a 88 48 89 de e8 5e b8 ff fd <0f> 0b 4c 89 e2
48 89 de 48 c7 c7 60 82 1a 88 e8 4a b8 ff fd 0f 0b
RSP: 0018:ffff8801dae079c0 EFLAGS: 00010086
RAX: 0000000000000075 RBX: ffff8801dae26708 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff81634381 RDI: 0000000000000001
RBP: ffff8801dae079d8 R08: ffff8801d70ca380 R09: ffffed003b5c4fc0
R10: ffffed003b5c4fc0 R11: ffff8801dae27e07 R12: ffff8801dae26740
R13: ffff8801dae07ab8 R14: ffff8801dae26740 R15: 0000000000000001
FS: 000000000157c940(0000) GS:ffff8801dae00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000455b7f CR3: 00000001b3a95000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
syzbot.
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches

[syzbot]

syzbot has found a reproducer for the following crash on:

HEAD commit: 526674536360 Add linux-next specific files for 20180706
git tree: linux-next

console output: https://syzkaller.appspot.com/x/log.txt?x=11ac81b2400000

kernel config: https://syzkaller.appspot.com/x/.config?x=c8d1cfc0cb798e48
dashboard link: https://syzkaller.appspot.com/bug?extid=d8a8e42dfba0454286ff
compiler: gcc (GCC) 8.0.1 20180413 (experimental)

syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=1507ca2c400000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=10431a2c400000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+d8a8e4...@syzkaller.appspotmail.com

RBP: 00000000006cb018 R08: 0000000000000002 R09: 00007fff6f890032
R10: 0000000020377000 R11: 0000000000000246 R12: 0000000000000003
R13: ffffffffffffffff R14: 0000000000000000 R15: 0000000000000000
list_add corruption. prev->next should be next (ffff8801daf26708), but was
0000000000000000. (prev=ffff8801daf26740).

------------[ cut here ]------------
kernel BUG at lib/list_debug.c:28!
invalid opcode: 0000 [#1] SMP KASAN

CPU: 1 PID: 4449 Comm: syz-executor507 Not tainted

4.18.0-rc3-next-20180706+ #1
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
RIP: 0010:__list_add_valid.cold.0+0x23/0x25 lib/list_debug.c:26
Code: e8 9f 72 57 fe eb 97 48 89 d9 48 c7 c7 60 80 1a 88 e8 72 b8 ff fd 0f
0b 48 89 f1 48 c7 c7 20 81 1a 88 48 89 de e8 5e b8 ff fd <0f> 0b 4c 89 e2
48 89 de 48 c7 c7 60 82 1a 88 e8 4a b8 ff fd 0f 0b

RSP: 0018:ffff8801daf079c0 EFLAGS: 00010086
RAX: 0000000000000075 RBX: ffff8801daf26708 RCX: 0000000000000000

RDX: 0000000000000000 RSI: ffffffff81634381 RDI: 0000000000000001

RBP: ffff8801daf079d8 R08: ffff8801ad2464c0 R09: ffffed003b5e4fc0
R10: ffffed003b5e4fc0 R11: ffff8801daf27e07 R12: ffff8801daf26740
R13: ffff8801daf07ab8 R14: ffff8801daf26740 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff8801daf00000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00000000203e8008 CR3: 0000000008e6a000 CR4: 00000000001406e0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<IRQ>
__list_add include/linux/list.h:60 [inline]
list_add_tail include/linux/list.h:93 [inline]
__cpu_stop_queue_work kernel/stop_machine.c:72 [inline]
cpu_stop_queue_work+0x248/0x460 kernel/stop_machine.c:87
stop_one_cpu_nowait+0xd3/0x100 kernel/stop_machine.c:341
watchdog_timer_fn+0x93/0x2e0 kernel/watchdog.c:367
__run_hrtimer kernel/time/hrtimer.c:1398 [inline]
__hrtimer_run_queues+0x3eb/0x10c0 kernel/time/hrtimer.c:1460
hrtimer_interrupt+0x2f3/0x750 kernel/time/hrtimer.c:1518
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1025 [inline]
smp_apic_timer_interrupt+0x165/0x730 arch/x86/kernel/apic/apic.c:1050
apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:867
</IRQ>

RIP: 0010:__sanitizer_cov_trace_pc+0x0/0x50 kernel/kcov.c:146
Code: 14 dd 28 00 00 00 4d 39 d0 72 1b 49 83 c1 01 4a 89 7c 10 e0 4a 89 74

10 e8 4a 89 54 10 f0 4a 89 4c d8 20 4c 89 08 5d c3 66 90 <55> 65 48 8b 04
25 40 ee 01 00 65 8b 15 5f 73 85 7e 48 89 e5 81 e2

RSP: 0018:ffff8801ad79e208 EFLAGS: 00000292 ORIG_RAX: ffffffffffffff13
RAX: ffff8801ad2464c0 RBX: ffffea000701e000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff819b034f RDI: 0000000000000007
RBP: ffff8801ad79e5a8 R08: ffff8801ad2464c0 R09: fffff94000e03c06
R10: fffff94000e03c06 R11: ffffea000701e037 R12: ffff8801ad79e580
R13: 1ffff10035af3c68 R14: dffffc0000000000 R15: 0000000000000000

shmem_undo_range+0x38e/0x29a0 mm/shmem.c:795
shmem_truncate_range+0x27/0xa0 mm/shmem.c:971
shmem_evict_inode+0x3b2/0xcb0 mm/shmem.c:1071
evict+0x4ae/0x990 fs/inode.c:558
iput_final fs/inode.c:1508 [inline]
iput+0x635/0xaa0 fs/inode.c:1534
dentry_unlink_inode+0x4ae/0x640 fs/dcache.c:377
__dentry_kill+0x44c/0x7a0 fs/dcache.c:569
dentry_kill+0xc9/0x5a0 fs/dcache.c:688
dput.part.26+0x66b/0x7a0 fs/dcache.c:849
dput+0x15/0x20 fs/dcache.c:831
__fput+0x558/0x930 fs/file_table.c:235
____fput+0x15/0x20 fs/file_table.c:251
task_work_run+0x1ec/0x2a0 kernel/task_work.c:113
exit_task_work include/linux/task_work.h:22 [inline]
do_exit+0x1b08/0x2750 kernel/exit.c:869
do_group_exit+0x177/0x440 kernel/exit.c:972
__do_sys_exit_group kernel/exit.c:983 [inline]
__se_sys_exit_group kernel/exit.c:981 [inline]
__x64_sys_exit_group+0x3e/0x50 kernel/exit.c:981
do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
entry_SYSCALL_64_after_hwframe+0x49/0xbe

RIP: 0033:0x43f0f8
Code: Bad RIP value.
RSP: 002b:00007fff6f8995b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 000000000043f0f8
RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000
RBP: 00000000004bf1e8 R08: 00000000000000e7 R09: ffffffffffffffd0
R10: 0000000020377000 R11: 0000000000000246 R12: 0000000000000001
R13: 00000000006d1180 R14: 0000000000000000 R15: 0000000000000000

Modules linked in:
Dumping ftrace buffer:
(ftrace buffer empty)

---[ end trace 599d79740552877c ]---

RIP: 0010:__list_add_valid.cold.0+0x23/0x25 lib/list_debug.c:26
Code: e8 9f 72 57 fe eb 97 48 89 d9 48 c7 c7 60 80 1a 88 e8 72 b8 ff fd 0f
0b 48 89 f1 48 c7 c7 20 81 1a 88 48 89 de e8 5e b8 ff fd <0f> 0b 4c 89 e2
48 89 de 48 c7 c7 60 82 1a 88 e8 4a b8 ff fd 0f 0b

RSP: 0018:ffff8801daf079c0 EFLAGS: 00010086
RAX: 0000000000000075 RBX: ffff8801daf26708 RCX: 0000000000000000

RDX: 0000000000000000 RSI: ffffffff81634381 RDI: 0000000000000001

RBP: ffff8801daf079d8 R08: ffff8801ad2464c0 R09: ffffed003b5e4fc0
R10: ffffed003b5e4fc0 R11: ffff8801daf27e07 R12: ffff8801daf26740
R13: ffff8801daf07ab8 R14: ffff8801daf26740 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff8801daf00000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 000000000043f0ce CR3: 0000000008e6a000 CR4: 00000000001406e0

[Tetsuo Handa]

Hello Matthew,

It seems to me that there are other locations which do not check xas_store()
failure. Is that really OK? If they are OK, I think we want a comment like
/* This never fails. */ or /* Failure is OK because ... */
for each call without failure check.



From d6f24d6eecd79836502527624f8086f4e3e4c331 Mon Sep 17 00:00:00 2001
From: Tetsuo Handa <penguin...@I-love.SAKURA.ne.jp>
Date: Mon, 9 Jul 2018 15:58:44 +0900
Subject: [PATCH] shmem: Fix crash upon xas_store() failure.

syzbot is reporting list corruption [1]. This is because xas_store() from
shmem_add_to_page_cache() is not handling memory allocation failure. Fix
this by checking xas_error() after xas_store().

[1] https://syzkaller.appspot.com/bug?id=ccf34b892706ff6f3321da526329476eab23d5a0

Signed-off-by: Tetsuo Handa <penguin...@I-love.SAKURA.ne.jp>
Reported-by: syzbot+d8a8e4...@syzkaller.appspotmail.com
Fixes: "shmem: Convert shmem_add_to_page_cache to XArray"
---
mm/shmem.c | 2 ++
1 file changed, 2 insertions(+)

diff --git a/mm/shmem.c b/mm/shmem.c
index 0f151c8..382aeb3 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -606,6 +606,8 @@ static int shmem_add_to_page_cache(struct page *page,
}
next:
xas_store(&xas, page + i);
+ if (xas_error(&xas))
+ goto unlock;
if (++i < nr) {
xas_next(&xas);
goto next;
--
1.8.3.1

[Matthew Wilcox]

On Mon, Jul 09, 2018 at 09:55:17PM +0900, Tetsuo Handa wrote:
> Hello Matthew,
>
> It seems to me that there are other locations which do not check xas_store()
> failure. Is that really OK? If they are OK, I think we want a comment like
> /* This never fails. */ or /* Failure is OK because ... */
> for each call without failure check.

Good grief, no, I'm not adding a comment to all 50 calls to
xas_store(). Here are some rules:

- xas_store(NULL) cannot fail.
- xas_store(p) cannot fail if we know something was already in
that slot beforehand (ie a replace operation).
- xas_store(p) cannot fail if xas_create_range() was previously
successful.
- xas_store(p) can fail, but it's OK if the only things after that are
other xas_*() calls. Because every xas_*() call checks xas_error().
So this is fine:

do {
xas_store(&xas, p);
xas_set_tag(&xas, XA_TAG_0);
} while (xas_nomem(&xas, GFP_KERNEL));

> >From d6f24d6eecd79836502527624f8086f4e3e4c331 Mon Sep 17 00:00:00 2001
> From: Tetsuo Handa <penguin...@I-love.SAKURA.ne.jp>
> Date: Mon, 9 Jul 2018 15:58:44 +0900
> Subject: [PATCH] shmem: Fix crash upon xas_store() failure.
>
> syzbot is reporting list corruption [1]. This is because xas_store() from
> shmem_add_to_page_cache() is not handling memory allocation failure. Fix
> this by checking xas_error() after xas_store().

I have no idea why you wrote this patch on Monday when I already said
I knew what the problem was on Friday, fixed the problem and pushed it
out to my git tree on Saturday.

[Tetsuo Handa]

On 2018/07/09 22:32, Matthew Wilcox wrote:
>> >From d6f24d6eecd79836502527624f8086f4e3e4c331 Mon Sep 17 00:00:00 2001
>> From: Tetsuo Handa <penguin...@I-love.SAKURA.ne.jp>
>> Date: Mon, 9 Jul 2018 15:58:44 +0900
>> Subject: [PATCH] shmem: Fix crash upon xas_store() failure.
>>
>> syzbot is reporting list corruption [1]. This is because xas_store() from
>> shmem_add_to_page_cache() is not handling memory allocation failure. Fix
>> this by checking xas_error() after xas_store().
>
> I have no idea why you wrote this patch on Monday when I already said
> I knew what the problem was on Friday, fixed the problem and pushed it
> out to my git tree on Saturday.
>

Because syzbot found a C reproducer on 2018/07/09 02:29 UTC, and your fix was
not in time for a kernel version syzbot was testing, and you were not listed
as a recipient of this bug, and I didn't know you already fixed this bug.

Anyway, linux-next-20180709 still does not have this fix.
What is the title of your fix you pushed on Saturday?

[Matthew Wilcox]

I folded it into shmem: Convert shmem_add_to_page_cache to XArray.
I can see it's fixed in today's linux-next. I fixed it differently
from the way you fixed it, so if you're looking for an xas_error check
after xas_store, you won't find it.

[Tetsuo Handa]

On 2018/07/09 23:24, Matthew Wilcox wrote:
>> Anyway, linux-next-20180709 still does not have this fix.
>> What is the title of your fix you pushed on Saturday?
>
> I folded it into shmem: Convert shmem_add_to_page_cache to XArray.
> I can see it's fixed in today's linux-next. I fixed it differently
> from the way you fixed it, so if you're looking for an xas_error check
> after xas_store, you won't find it.
>

OK. linux-next-20180709 should no longer hit this bug. Closing with

#syz fix: shmem: Convert shmem_add_to_page_cache to XArray

Thanks.