kernel BUG at mm/vmalloc.c:LINE! (2)

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: 7cc2a8ea Merge tag 'block-5.8-2020-07-01' of git://git.ker..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=129af755100000
kernel config: https://syzkaller.appspot.com/x/.config?x=183dd243398ba7ec
dashboard link: https://syzkaller.appspot.com/bug?extid=5f326d255ca648131f87
compiler: clang version 10.0.0 (https://github.com/llvm/llvm-project/ c2443155a0fb245c8f17f2c1c72b6ea391e86e81)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+5f326d...@syzkaller.appspotmail.com

------------[ cut here ]------------
kernel BUG at mm/vmalloc.c:553!
invalid opcode: 0000 [#1] PREEMPT SMP KASAN
CPU: 0 PID: 3491 Comm: syz-executor.2 Not tainted 5.8.0-rc3-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:find_va_links mm/vmalloc.c:549 [inline]
RIP: 0010:merge_or_add_vmap_area mm/vmalloc.c:778 [inline]
RIP: 0010:__purge_vmap_area_lazy+0x18af/0x18c0 mm/vmalloc.c:1381
Code: e1 07 80 c1 03 38 c1 0f 8c f9 e8 ff ff 48 c7 c7 c8 2b 6d 89 e8 22 81 09 00 e9 e8 e8 ff ff e8 38 82 ca ff 0f 0b e8 31 82 ca ff <0f> 0b 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 e8 1b 82 ca ff
RSP: 0018:ffffc900187f7800 EFLAGS: 00010283
RAX: ffffffff81a9f9df RBX: ffffc90007ea8000 RCX: 0000000000040000
RDX: ffffc9000d74c000 RSI: 0000000000026b24 RDI: 0000000000026b25
RBP: ffffc90007ea7000 R08: ffffffff81a9e3fd R09: fffffbfff12631ef
R10: fffffbfff12631ef R11: 0000000000000000 R12: ffffc90008703000
R13: dffffc0000000000 R14: ffff88808e51ea90 R15: ffffc90000000000
FS: 00007f40b9e29700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000000b770 CR3: 000000019dad6000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
_vm_unmap_aliases+0x442/0x4d0 mm/vmalloc.c:1800
change_page_attr_set_clr+0x24b/0x5c0 arch/x86/mm/pat/set_memory.c:1732
change_page_attr_clear arch/x86/mm/pat/set_memory.c:1789 [inline]
set_memory_ro+0x5d/0x80 arch/x86/mm/pat/set_memory.c:1935
bpf_jit_binary_lock_ro include/linux/filter.h:815 [inline]
bpf_int_jit_compile+0x84a1/0x8910 arch/x86/net/bpf_jit_comp.c:1929
bpf_prog_select_runtime+0x76d/0xa60 kernel/bpf/core.c:1807
bpf_prog_load kernel/bpf/syscall.c:2198 [inline]
__do_sys_bpf+0xfabc/0x10c80 kernel/bpf/syscall.c:4114
do_syscall_64+0x73/0xe0 arch/x86/entry/common.c:359
entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x45cb29
Code: Bad RIP value.
RSP: 002b:00007f40b9e28c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00000000004db2a0 RCX: 000000000045cb29
RDX: 0000000000000048 RSI: 0000000020000080 RDI: 0000000000000005
RBP: 000000000078bf00 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 0000000000000070 R14: 00000000004c3450 R15: 00007f40b9e296d4
Modules linked in:
---[ end trace 0c5c57c9d5f27037 ]---
RIP: 0010:find_va_links mm/vmalloc.c:549 [inline]
RIP: 0010:merge_or_add_vmap_area mm/vmalloc.c:778 [inline]
RIP: 0010:__purge_vmap_area_lazy+0x18af/0x18c0 mm/vmalloc.c:1381
Code: e1 07 80 c1 03 38 c1 0f 8c f9 e8 ff ff 48 c7 c7 c8 2b 6d 89 e8 22 81 09 00 e9 e8 e8 ff ff e8 38 82 ca ff 0f 0b e8 31 82 ca ff <0f> 0b 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 e8 1b 82 ca ff
RSP: 0018:ffffc900187f7800 EFLAGS: 00010283
RAX: ffffffff81a9f9df RBX: ffffc90007ea8000 RCX: 0000000000040000
RDX: ffffc9000d74c000 RSI: 0000000000026b24 RDI: 0000000000026b25
RBP: ffffc90007ea7000 R08: ffffffff81a9e3fd R09: fffffbfff12631ef
R10: fffffbfff12631ef R11: 0000000000000000 R12: ffffc90008703000
R13: dffffc0000000000 R14: ffff88808e51ea90 R15: ffffc90000000000
FS: 00007f40b9e29700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000000b770 CR3: 000000019dad6000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit: ab8be66e Add linux-next specific files for 20200720
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=161a0cc8900000
kernel config: https://syzkaller.appspot.com/x/.config?x=c4bf77d63d0cf88c
dashboard link: https://syzkaller.appspot.com/bug?extid=5f326d255ca648131f87
compiler: gcc (GCC) 10.1.0-syz 20200507
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=151192bb100000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=12d7a873100000

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by: syzbot+5f326d...@syzkaller.appspotmail.com

------------[ cut here ]------------

kernel BUG at mm/vmalloc.c:3089!

invalid opcode: 0000 [#1] PREEMPT SMP KASAN

CPU: 0 PID: 12 Comm: kworker/0:1 Not tainted 5.8.0-rc6-next-20200720-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011

Workqueue: events pcpu_balance_workfn
RIP: 0010:free_vm_area mm/vmalloc.c:3089 [inline]
RIP: 0010:free_vm_area mm/vmalloc.c:3085 [inline]
RIP: 0010:pcpu_free_vm_areas+0x96/0xc0 mm/vmalloc.c:3432
Code: 75 48 48 8b 2b 48 8d 7d 08 48 89 f8 48 c1 e8 03 42 80 3c 30 00 75 2c 48 8b 7d 08 e8 c4 c8 ff ff 48 39 c5 74 a5 e8 ea c3 c9 ff <0f> 0b e8 e3 c3 c9 ff 4c 89 ff 5b 5d 41 5c 41 5d 41 5e 41 5f e9 71
RSP: 0018:ffffc90000d2fba8 EFLAGS: 00010293
RAX: 0000000000000000 RBX: ffff8880a801be00 RCX: 0000000000000000
RDX: ffff8880a95fa300 RSI: ffffffff81aa7c76 RDI: 0000000000000001
RBP: ffff8880a2b38180 R08: 0000000000000000 R09: ffffffff89cfecc3
R10: fffffbfff139fd98 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000001 R14: dffffc0000000000 R15: ffff8880a801be00
FS: 0000000000000000(0000) GS:ffff8880ae600000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00000000004c8e48 CR3: 00000000a4c08000 CR4: 00000000001506f0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:

pcpu_destroy_chunk mm/percpu-vm.c:366 [inline]
__pcpu_balance_workfn mm/percpu.c:1982 [inline]
pcpu_balance_workfn+0x8b3/0x1310 mm/percpu.c:2069
process_one_work+0x94c/0x1670 kernel/workqueue.c:2269
worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
kthread+0x3b5/0x4a0 kernel/kthread.c:292
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
Modules linked in:
---[ end trace 6a2e56ec52e1f480 ]---
RIP: 0010:free_vm_area mm/vmalloc.c:3089 [inline]
RIP: 0010:free_vm_area mm/vmalloc.c:3085 [inline]
RIP: 0010:pcpu_free_vm_areas+0x96/0xc0 mm/vmalloc.c:3432
Code: 75 48 48 8b 2b 48 8d 7d 08 48 89 f8 48 c1 e8 03 42 80 3c 30 00 75 2c 48 8b 7d 08 e8 c4 c8 ff ff 48 39 c5 74 a5 e8 ea c3 c9 ff <0f> 0b e8 e3 c3 c9 ff 4c 89 ff 5b 5d 41 5c 41 5d 41 5e 41 5f e9 71
RSP: 0018:ffffc90000d2fba8 EFLAGS: 00010293
RAX: 0000000000000000 RBX: ffff8880a801be00 RCX: 0000000000000000
RDX: ffff8880a95fa300 RSI: ffffffff81aa7c76 RDI: 0000000000000001
RBP: ffff8880a2b38180 R08: 0000000000000000 R09: ffffffff89cfecc3
R10: fffffbfff139fd98 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000001 R14: dffffc0000000000 R15: ffff8880a801be00
FS: 0000000000000000(0000) GS:ffff8880ae600000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00000000004c8e48 CR3: 00000000a4c08000 CR4: 00000000001506f0

[Uladzislau Rezki]

That is because of below revert:

<snip>
commit bdbfb1d52d5e576c1d275fd8ab59b677011229e8
Author: Ingo Molnar <mi...@kernel.org>
Date: Sun Jun 7 21:12:51 2020 +0200

Revert "mm/vmalloc: modify struct vmap_area to reduce its size"

This reverts commit 688fcbfc06e4fdfbb7e1d5a942a1460fe6379d2d.

Signed-off-by: Ingo Molnar <mi...@kernel.org>

Conflicts:
mm/vmalloc.c
<snip>

I can check further, but it can be it was not correctly reverted,
because everything should work just fine even with the revert,
though i i do not understand a reason of reverting.

--
Vlad Rezki

[syzbot]

syzbot has bisected this issue to:

commit bdbfb1d52d5e576c1d275fd8ab59b677011229e8
Author: Ingo Molnar <mi...@kernel.org>

Date: Sun Jun 7 19:12:51 2020 +0000

Revert "mm/vmalloc: modify struct vmap_area to reduce its size"

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=166e6b7f100000
start commit: ab8be66e Add linux-next specific files for 20200720
git tree: linux-next
final oops: https://syzkaller.appspot.com/x/report.txt?x=156e6b7f100000
console output: https://syzkaller.appspot.com/x/log.txt?x=116e6b7f100000

kernel config: https://syzkaller.appspot.com/x/.config?x=c4bf77d63d0cf88c
dashboard link: https://syzkaller.appspot.com/bug?extid=5f326d255ca648131f87

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=151192bb100000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=12d7a873100000

Reported-by: syzbot+5f326d...@syzkaller.appspotmail.com
Fixes: bdbfb1d52d5e ("Revert "mm/vmalloc: modify struct vmap_area to reduce its size"")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[Qian Cai]

Vlad, how sure are you about this? We also start to trigger this now on
linux-next, but the reverting patch surely looks like doggy without any useful
information in the commit description.

[Uladzislau Rezki]

Hello, Andrew, Qian.

I am not aware of reason of the revert, though i tried to get through Ingo.
I can send out a patch that fixes the revert. Another option to drop the
revert, but it is up to Andrew and Ingo.

Andrew, could you please comment on?

Thank you in advance!

--
Vlad Rezki

[Andrew Morton]

On Wed, 22 Jul 2020 16:46:50 +0200 Uladzislau Rezki <ure...@gmail.com> wrote:

> > > I can check further, but it can be it was not correctly reverted,
> > > because everything should work just fine even with the revert,
> > > though i i do not understand a reason of reverting.
> >
> > Vlad, how sure are you about this? We also start to trigger this now on
> > linux-next, but the reverting patch surely looks like doggy without any useful
> > information in the commit description.
> >
> Hello, Andrew, Qian.
>
> I am not aware of reason of the revert, though i tried to get through Ingo.
> I can send out a patch that fixes the revert. Another option to drop the
> revert, but it is up to Andrew and Ingo.
>
> Andrew, could you please comment on?

All a bit mysterious. I think it's best that we revert this from
linux-next until we hear from Ingo. I queued a patch - I expect
Stephen will see and grab it, thanks.

[Stephen Rothwell]

Hi Andrew,

On Thu, 23 Jul 2020 19:50:29 -0700 Andrew Morton <ak...@linux-foundation.org> wrote:
>
> All a bit mysterious. I think it's best that we revert this from
> linux-next until we hear from Ingo. I queued a patch - I expect
> Stephen will see and grab it, thanks.

Wiil do.

--
Cheers,
Stephen Rothwell

[Stephen Rothwell]

Hi Andrew,

On Thu, 23 Jul 2020 19:50:29 -0700 Andrew Morton <ak...@linux-foundation.org> wrote:
>

In the end I actually did the revert (of the revert) in the merge of
the tip tree (so that -next will bisect better if necessary). So you
will not need the revert in your quilt series after today.

--
Cheers,
Stephen Rothwell

[Thomas Gleixner]

Stephen Rothwell <s...@canb.auug.org.au> writes:
>> All a bit mysterious. I think it's best that we revert this from
>> linux-next until we hear from Ingo. I queued a patch - I expect
>> Stephen will see and grab it, thanks.
>
> In the end I actually did the revert (of the revert) in the merge of
> the tip tree (so that -next will bisect better if necessary). So you
> will not need the revert in your quilt series after today.

Sigh. I have no idea why this was in tip auto-latest. I just
reintegrated that branch and the annoyance should be gone now.

Sorry for not paying attention.

Thanks,

tglx

[syzbot]

syzbot suspects this issue was fixed by commit:

commit 537cf4e3cc2f6cc9088dcd6162de573f603adc29
Author: Magnus Karlsson <magnus....@intel.com>
Date: Fri Nov 20 11:53:39 2020 +0000

xsk: Fix umem cleanup bug at socket destruct

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=139f3dfb500000
start commit: e87d24fc Merge branch 'net-iucv-fixes-2020-11-09'
git tree: net
kernel config: https://syzkaller.appspot.com/x/.config?x=61033507391c77ff
dashboard link: https://syzkaller.appspot.com/bug?extid=5f326d255ca648131f87
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10d10006500000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=126c9eaa500000

If the result looks correct, please mark the issue as fixed by replying with:

#syz fix: xsk: Fix umem cleanup bug at socket destruct

[Dmitry Vyukov]

FTR, the bisection log looks clean, but this does not look like the
fix for this. The reproducer does not destroy sockets.

[Vegard Nossum]

I think it's the correct fix.

The crash report also has this, which shows the reproducer does
actually destroy sockets:

xdp_umem_addr_unmap net/xdp/xdp_umem.c:44 [inline]
xdp_umem_release net/xdp/xdp_umem.c:62 [inline]
xdp_put_umem+0x113/0x330 net/xdp/xdp_umem.c:80
xsk_destruct net/xdp/xsk.c:1150 [inline]
xsk_destruct+0xc0/0xf0 net/xdp/xsk.c:1142
__sk_destruct+0x4b/0x8f0 net/core/sock.c:1759
rcu_do_batch kernel/rcu/tree.c:2476 [inline]

I've tested the reproducer on both 537cf4e3cc2f and 537cf4e3cc2f^ and
it only reproduces on 537cf4e3cc2f^ here (with the same stack trace as
the syzbot report).

The repro I used was
https://syzkaller.appspot.com/text?tag=ReproSyz&x=10d10006500000 which
is just:

r0 = socket$xdp(0x2c, 0x3, 0x0)
setsockopt$XDP_UMEM_REG(r0, 0x11b, 0x4,
&(0x7f0000000040)={&(0x7f0000000000)=""/2, 0x1000000, 0x1000}, 0x20)

so the socket definitely gets created/destroyed.

Feel free to undo if you disagree:

#syz fix: xsk: Fix umem cleanup bug at socket destruct

Vegard