BUG: unable to handle kernel paging request in pcpu_alloc

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: 1ab75b2e Add linux-next specific files for 20191203
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=10edf2eae00000
kernel config: https://syzkaller.appspot.com/x/.config?x=de1505c727f0ec20
dashboard link: https://syzkaller.appspot.com/bug?extid=82e323920b78d54aaed5
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=156ef061e00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11641edae00000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+82e323...@syzkaller.appspotmail.com

RDX: 000000000000003c RSI: 0000000020000080 RDI: 0c00000000000000
RBP: 0000000000000000 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000018
R13: 0000000000000004 R14: 0000000000000005 R15: 0000000000000000
BUG: unable to handle page fault for address: fffff91ffff00000
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 21ffe6067 P4D 21ffe6067 PUD aa56c067 PMD aa56d067 PTE 0
Oops: 0000 [#1] PREEMPT SMP KASAN
CPU: 1 PID: 8999 Comm: syz-executor865 Not tainted
5.4.0-next-20191203-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
RIP: 0010:memory_is_nonzero mm/kasan/generic.c:121 [inline]
RIP: 0010:memory_is_poisoned_n mm/kasan/generic.c:135 [inline]
RIP: 0010:memory_is_poisoned mm/kasan/generic.c:166 [inline]
RIP: 0010:check_memory_region_inline mm/kasan/generic.c:182 [inline]
RIP: 0010:check_memory_region+0x9c/0x1a0 mm/kasan/generic.c:192
Code: c9 4d 0f 49 c1 49 c1 f8 03 45 85 c0 0f 84 10 01 00 00 41 83 e8 01 4e
8d 44 c0 08 eb 0d 48 83 c0 08 4c 39 c0 0f 84 a7 00 00 00 <48> 83 38 00 74
ed 4c 8d 40 08 eb 09 48 83 c0 01 49 39 c0 74 53 80
RSP: 0018:ffffc90001f67a80 EFLAGS: 00010216
RAX: fffff91ffff00000 RBX: fffff91ffff01000 RCX: ffffffff819e1589
RDX: 0000000000000001 RSI: 0000000000008000 RDI: ffffe8ffff800000
RBP: ffffc90001f67a98 R08: fffff91ffff01000 R09: 0000000000001000
R10: fffff91ffff00fff R11: ffffe8ffff807fff R12: fffff91ffff00000
R13: 0000000000008000 R14: 0000000000000000 R15: ffff88821fffd100
FS: 00000000011a7880(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: fffff91ffff00000 CR3: 00000000a94ad000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
memset+0x24/0x40 mm/kasan/common.c:107
memset include/linux/string.h:410 [inline]
pcpu_alloc+0x589/0x1380 mm/percpu.c:1734
__alloc_percpu_gfp+0x28/0x30 mm/percpu.c:1783
bpf_array_alloc_percpu kernel/bpf/arraymap.c:35 [inline]
array_map_alloc+0x698/0x7d0 kernel/bpf/arraymap.c:159
find_and_alloc_map kernel/bpf/syscall.c:123 [inline]
map_create kernel/bpf/syscall.c:654 [inline]
__do_sys_bpf+0x478/0x3810 kernel/bpf/syscall.c:3012
__se_sys_bpf kernel/bpf/syscall.c:2989 [inline]
__x64_sys_bpf+0x73/0xb0 kernel/bpf/syscall.c:2989
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x442f99
Code: e8 ec 09 03 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff
ff 0f 83 cb 08 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ffc8aa156d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000442f99
RDX: 000000000000003c RSI: 0000000020000080 RDI: 0c00000000000000
RBP: 0000000000000000 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000018
R13: 0000000000000004 R14: 0000000000000005 R15: 0000000000000000
Modules linked in:
CR2: fffff91ffff00000
---[ end trace 449f8b43dad6ffb8 ]---
RIP: 0010:memory_is_nonzero mm/kasan/generic.c:121 [inline]
RIP: 0010:memory_is_poisoned_n mm/kasan/generic.c:135 [inline]
RIP: 0010:memory_is_poisoned mm/kasan/generic.c:166 [inline]
RIP: 0010:check_memory_region_inline mm/kasan/generic.c:182 [inline]
RIP: 0010:check_memory_region+0x9c/0x1a0 mm/kasan/generic.c:192
Code: c9 4d 0f 49 c1 49 c1 f8 03 45 85 c0 0f 84 10 01 00 00 41 83 e8 01 4e
8d 44 c0 08 eb 0d 48 83 c0 08 4c 39 c0 0f 84 a7 00 00 00 <48> 83 38 00 74
ed 4c 8d 40 08 eb 09 48 83 c0 01 49 39 c0 74 53 80
RSP: 0018:ffffc90001f67a80 EFLAGS: 00010216
RAX: fffff91ffff00000 RBX: fffff91ffff01000 RCX: ffffffff819e1589
RDX: 0000000000000001 RSI: 0000000000008000 RDI: ffffe8ffff800000
RBP: ffffc90001f67a98 R08: fffff91ffff01000 R09: 0000000000001000
R10: fffff91ffff00fff R11: ffffe8ffff807fff R12: fffff91ffff00000
R13: 0000000000008000 R14: 0000000000000000 R15: ffff88821fffd100
FS: 00000000011a7880(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: fffff91ffff00000 CR3: 00000000a94ad000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches

[Dmitry Vyukov]

On Wed, Dec 4, 2019 at 9:15 AM syzbot
<syzbot+82e323...@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following crash on:
>
> HEAD commit: 1ab75b2e Add linux-next specific files for 20191203
> git tree: linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=10edf2eae00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=de1505c727f0ec20
> dashboard link: https://syzkaller.appspot.com/bug?extid=82e323920b78d54aaed5
> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=156ef061e00000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11641edae00000
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+82e323...@syzkaller.appspotmail.com

+Daniel, is it the same as:
https://syzkaller.appspot.com/bug?id=f6450554481c55c131cc23d581fbd8ea42e63e18
If so, is it possible to make KASAN detect this consistently with the
same crash type so that syzbot does not report duplicates?

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/000000000000314c120598dc69bd%40google.com.

[syzbot]

syzbot has bisected this bug to:

commit 0609ae011deb41c9629b7f5fd626dfa1ac9d16b0
Author: Daniel Axtens <d...@axtens.net>
Date: Sun Dec 1 01:55:00 2019 +0000

x86/kasan: support KASAN_VMALLOC

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=166cf97ee00000
start commit: 1ab75b2e Add linux-next specific files for 20191203
git tree: linux-next
final crash: https://syzkaller.appspot.com/x/report.txt?x=156cf97ee00000
console output: https://syzkaller.appspot.com/x/log.txt?x=116cf97ee00000

kernel config: https://syzkaller.appspot.com/x/.config?x=de1505c727f0ec20
dashboard link: https://syzkaller.appspot.com/bug?extid=82e323920b78d54aaed5

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=156ef061e00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11641edae00000

Reported-by: syzbot+82e323...@syzkaller.appspotmail.com
Fixes: 0609ae011deb ("x86/kasan: support KASAN_VMALLOC")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[Daniel Axtens]

>> HEAD commit: 1ab75b2e Add linux-next specific files for 20191203
>> git tree: linux-next
>> console output: https://syzkaller.appspot.com/x/log.txt?x=10edf2eae00000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=de1505c727f0ec20
>> dashboard link: https://syzkaller.appspot.com/bug?extid=82e323920b78d54aaed5
>> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=156ef061e00000
>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11641edae00000
>>
>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> Reported-by: syzbot+82e323...@syzkaller.appspotmail.com
>
> +Daniel, is it the same as:
> https://syzkaller.appspot.com/bug?id=f6450554481c55c131cc23d581fbd8ea42e63e18
> If so, is it possible to make KASAN detect this consistently with the
> same crash type so that syzbot does not report duplicates?

It looks like both of these occur immediately after failure injection. I
think my assumption that I could ignore the chance of failures in the
per-cpu allocation path will have to be revisited. That's annoying.

I'll try to spin something today but Andrey feel free to pip me at the
post again :)

I'm not 100% confident to call them dups just yet, but I'm about 80%
confident that they are.

Regards,
Daniel

[Daniel Borkmann]

On Thu, Dec 05, 2019 at 03:35:21PM +1100, Daniel Axtens wrote:
> >> HEAD commit: 1ab75b2e Add linux-next specific files for 20191203
> >> git tree: linux-next
> >> console output: https://syzkaller.appspot.com/x/log.txt?x=10edf2eae00000
> >> kernel config: https://syzkaller.appspot.com/x/.config?x=de1505c727f0ec20
> >> dashboard link: https://syzkaller.appspot.com/bug?extid=82e323920b78d54aaed5
> >> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> >> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=156ef061e00000
> >> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11641edae00000
> >>
> >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> >> Reported-by: syzbot+82e323...@syzkaller.appspotmail.com
> >
> > +Daniel, is it the same as:
> > https://syzkaller.appspot.com/bug?id=f6450554481c55c131cc23d581fbd8ea42e63e18
> > If so, is it possible to make KASAN detect this consistently with the
> > same crash type so that syzbot does not report duplicates?
>
> It looks like both of these occur immediately after failure injection. I
> think my assumption that I could ignore the chance of failures in the
> per-cpu allocation path will have to be revisited. That's annoying.
>
> I'll try to spin something today but Andrey feel free to pip me at the
> post again :)
>
> I'm not 100% confident to call them dups just yet, but I'm about 80%
> confident that they are.

Ok. Double checked BPF side yesterday night, but looks sane to me and the
fault also hints into pcpu_alloc() rather than BPF code. Daniel, from your
above reply, I read that you are aware of how the bisected commit would
have caused the fault?

Thanks,
Daniel

[Daniel Axtens]

Yes, this one is on me - I did not take into account the brutal
efficiency of the fault injector when implementing my KASAN support for
vmalloc areas. I have a fix, I'm just doing final tests now.

Regards,
Daniel

>
> Thanks,
> Daniel
>
> --
> You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/20191205125900.GB29780%40localhost.localdomain.

[Daniel Borkmann]

Perfect, thanks a lot!

[syzbot]

Hello,

syzbot tried to test the proposed patch but build/boot failed:

failed to apply patch:
checking file mm/vmalloc.c
Hunk #1 succeeded at 3297 (offset 8 lines).
Hunk #2 succeeded at 3427 (offset 8 lines).
Hunk #3 FAILED at 3436.
Hunk #4 succeeded at 3461 (offset 7 lines).
Hunk #5 succeeded at 3501 (offset 7 lines).
1 out of 5 hunks FAILED



Tested on:

commit: 282ffdf3 Add linux-next specific files for 20191205
git tree: linux-next

dashboard link: https://syzkaller.appspot.com/bug?extid=82e323920b78d54aaed5
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

patch: https://syzkaller.appspot.com/x/patch.diff?x=1093af36e00000