[syzbot] linux-next boot error: BUG: unable to handle kernel paging request in kernel_execve

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: bc6c6584ffb2 Add linux-next specific files for 20220810
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=115034c3080000
kernel config: https://syzkaller.appspot.com/x/.config?x=5784be4315a4403b
dashboard link: https://syzkaller.appspot.com/bug?extid=3250d9c8925ef29e975f
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+3250d9...@syzkaller.appspotmail.com

BUG: unable to handle page fault for address: ffffdc0000000000
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 11826067 P4D 11826067 PUD 0
Oops: 0000 [#1] PREEMPT SMP KASAN
CPU: 0 PID: 1100 Comm: kworker/u4:5 Not tainted 5.19.0-next-20220810-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/22/2022
RIP: 0010:strnlen+0x3b/0x70 lib/string.c:504
Code: 74 3c 48 bb 00 00 00 00 00 fc ff df 49 89 fc 48 89 f8 eb 09 48 83 c0 01 48 39 e8 74 1e 48 89 c2 48 89 c1 48 c1 ea 03 83 e1 07 <0f> b6 14 1a 38 ca 7f 04 84 d2 75 11 80 38 00 75 d9 4c 29 e0 48 83
RSP: 0000:ffffc90005c5fe10 EFLAGS: 00010246
RAX: ffff000000000000 RBX: dffffc0000000000 RCX: 0000000000000000
RDX: 1fffe00000000000 RSI: 0000000000020000 RDI: ffff000000000000
RBP: ffff000000020000 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000006 R11: 0000000000000000 R12: ffff000000000000
R13: ffff88814764cc00 R14: ffff000000000000 R15: ffff88814764cc00
FS: 0000000000000000(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffdc0000000000 CR3: 000000000bc8e000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
strnlen include/linux/fortify-string.h:119 [inline]
copy_string_kernel+0x26/0x250 fs/exec.c:616
copy_strings_kernel+0xb3/0x190 fs/exec.c:655
kernel_execve+0x377/0x500 fs/exec.c:1998
call_usermodehelper_exec_async+0x2e3/0x580 kernel/umh.c:112
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
</TASK>
Modules linked in:
CR2: ffffdc0000000000
---[ end trace 0000000000000000 ]---
RIP: 0010:strnlen+0x3b/0x70 lib/string.c:504
Code: 74 3c 48 bb 00 00 00 00 00 fc ff df 49 89 fc 48 89 f8 eb 09 48 83 c0 01 48 39 e8 74 1e 48 89 c2 48 89 c1 48 c1 ea 03 83 e1 07 <0f> b6 14 1a 38 ca 7f 04 84 d2 75 11 80 38 00 75 d9 4c 29 e0 48 83
RSP: 0000:ffffc90005c5fe10 EFLAGS: 00010246
RAX: ffff000000000000 RBX: dffffc0000000000 RCX: 0000000000000000
RDX: 1fffe00000000000 RSI: 0000000000020000 RDI: ffff000000000000
RBP: ffff000000020000 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000006 R11: 0000000000000000 R12: ffff000000000000
R13: ffff88814764cc00 R14: ffff000000000000 R15: ffff88814764cc00
FS: 0000000000000000(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffdc0000000000 CR3: 000000000bc8e000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
----------------
Code disassembly (best guess):
0: 74 3c je 0x3e
2: 48 bb 00 00 00 00 00 movabs $0xdffffc0000000000,%rbx
9: fc ff df
c: 49 89 fc mov %rdi,%r12
f: 48 89 f8 mov %rdi,%rax
12: eb 09 jmp 0x1d
14: 48 83 c0 01 add $0x1,%rax
18: 48 39 e8 cmp %rbp,%rax
1b: 74 1e je 0x3b
1d: 48 89 c2 mov %rax,%rdx
20: 48 89 c1 mov %rax,%rcx
23: 48 c1 ea 03 shr $0x3,%rdx
27: 83 e1 07 and $0x7,%ecx
* 2a: 0f b6 14 1a movzbl (%rdx,%rbx,1),%edx <-- trapping instruction
2e: 38 ca cmp %cl,%dl
30: 7f 04 jg 0x36
32: 84 d2 test %dl,%dl
34: 75 11 jne 0x47
36: 80 38 00 cmpb $0x0,(%rax)
39: 75 d9 jne 0x14
3b: 4c 29 e0 sub %r12,%rax
3e: 48 rex.W
3f: 83 .byte 0x83


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[Kees Cook]

Hi Fabio,

It seems likely that the kmap change[1] might be causing this crash. Is
there a boot-time setup race between kmap being available and early umh
usage?

-Kees

[1] https://git.kernel.org/linus/c6e8e36c6ae4b11bed5643317afb66b6c3cadba8

--
Kees Cook

[Ira Weiny]

On Thu, Aug 11, 2022 at 08:33:16AM -0700, Kees Cook wrote:
> Hi Fabio,
>
> It seems likely that the kmap change[1] might be causing this crash. Is
> there a boot-time setup race between kmap being available and early umh
> usage?

I don't see how this is a setup problem with the config reported here.

CONFIG_64BIT=y

...and HIGHMEM is not set.
...and PREEMPT_RT is not set.

So the kmap_local_page() call in that stack should be a page_address() only.

I think the issue must be some sort of race which was being prevented because
of the preemption and/or pagefault disable built into kmap_atomic().

Is this reproducable?

The hunk below will surely fix it but I think the pagefault_disable() is
the only thing that is required. It would be nice to test it.

Ira


diff --git a/fs/exec.c b/fs/exec.c
index b51dd14e7388..3da588c858ca 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -640,7 +640,11 @@ int copy_string_kernel(const char *arg, struct linux_binprm *bprm)
if (!page)
return -E2BIG;
flush_arg_page(bprm, pos & PAGE_MASK, page);
+ preempt_disable();
+ pagefault_disable();
memcpy_to_page(page, offset_in_page(pos), arg, bytes_to_copy);
+ pagefault_enable();
+ preempt_enable();
put_arg_page(page);

[Ira Weiny]

On Thu, Aug 11, 2022 at 10:39:29AM -0700, Ira wrote:
> On Thu, Aug 11, 2022 at 08:33:16AM -0700, Kees Cook wrote:
> > Hi Fabio,
> >
> > It seems likely that the kmap change[1] might be causing this crash. Is
> > there a boot-time setup race between kmap being available and early umh
> > usage?
>
> I don't see how this is a setup problem with the config reported here.
>
> CONFIG_64BIT=y
>
> ...and HIGHMEM is not set.
> ...and PREEMPT_RT is not set.
>
> So the kmap_local_page() call in that stack should be a page_address() only.
>
> I think the issue must be some sort of race which was being prevented because
> of the preemption and/or pagefault disable built into kmap_atomic().
>
> Is this reproducable?
>
> The hunk below will surely fix it but I think the pagefault_disable() is
> the only thing that is required. It would be nice to test it.

Fabio and I discussed this. And he also mentioned that pagefault_disable() is
all that is required.

Do we have a way to test this?
Ira

[Kees Cook]

On Thu, Aug 11, 2022 at 11:51:34AM -0700, Ira Weiny wrote:
> On Thu, Aug 11, 2022 at 10:39:29AM -0700, Ira wrote:
> > On Thu, Aug 11, 2022 at 08:33:16AM -0700, Kees Cook wrote:
> > > Hi Fabio,
> > >
> > > It seems likely that the kmap change[1] might be causing this crash. Is
> > > there a boot-time setup race between kmap being available and early umh
> > > usage?
> >
> > I don't see how this is a setup problem with the config reported here.
> >
> > CONFIG_64BIT=y
> >
> > ...and HIGHMEM is not set.
> > ...and PREEMPT_RT is not set.
> >
> > So the kmap_local_page() call in that stack should be a page_address() only.
> >
> > I think the issue must be some sort of race which was being prevented because
> > of the preemption and/or pagefault disable built into kmap_atomic().
> >
> > Is this reproducable?
> >
> > The hunk below will surely fix it but I think the pagefault_disable() is
> > the only thing that is required. It would be nice to test it.
>
> Fabio and I discussed this. And he also mentioned that pagefault_disable() is
> all that is required.

Okay, sounds good.

> Do we have a way to test this?

It doesn't look like syzbot has a reproducer yet, so its patch testing
system[1] will not work. But if you can send me a patch, I could land it
in -next and we could see if the reproduction frequency drops to zero.
(Looking at the dashboard, it's seen 2 crashes, most recently 8 hours
ago.)

-Kees

[1] https://github.com/google/syzkaller/blob/master/docs/syzbot.md#testing-patches

> [...]

[Ira Weiny]

Patch sent.

https://lore.kernel.org/lkml/20220812000919.4...@intel.com/

But I'm more confused after looking at this again.

Ira

[Dmitry Vyukov]

There is splat of random crashes in linux-next happened at the same time:

https://groups.google.com/g/syzkaller-bugs/search?q=%22linux-next%20boot%20error%3A%22

There are 10 different crashes in completely random places.
I would assume they have the same root cause, some silent memory
corruption or something similar.

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/YvWaqhLGsBp9ynIq%40iweiny-desk3.

[Kees Cook]

Thank you!

> >
> > But I'm more confused after looking at this again.
>
> There is splat of random crashes in linux-next happened at the same time:
>
> https://groups.google.com/g/syzkaller-bugs/search?q=%22linux-next%20boot%20error%3A%22
>
> There are 10 different crashes in completely random places.
> I would assume they have the same root cause, some silent memory
> corruption or something similar.

Yeah, I noticed the crashes stopped "on their own", so I think I'll
wait a bit more, and if it start back up, we can try Ira's patch, though
I'd agree with the assessment that it looks like it shouldn't be needed.

-Kees

--
Kees Cook

[syzbot]

Auto-closing this bug as obsolete.
Crashes did not happen for a while, no reproducer and no activity.