Re: general protection fault in perf_misc_flags

[Dmitry Vyukov]

On Sat, Sep 19, 2020 at 1:08 PM Borislav Petkov <b...@alien8.de> wrote:
>
> On Sat, Sep 19, 2020 at 01:32:14AM -0700, syzbot wrote:
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit: 92ab97ad Merge tag 'sh-for-5.9-part2' of git://git.libc.or..
> > git tree: upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=1069669b900000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=cd992d74d6c7e62
> > dashboard link: https://syzkaller.appspot.com/bug?extid=ce179bc99e64377c24bc
> > compiler: clang version 10.0.0 (https://github.com/llvm/llvm-project/ c2443155a0fb245c8f17f2c1c72b6ea391e86e81)
> >
> > Unfortunately, I don't have any reproducer for this issue yet.
> >
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+ce179b...@syzkaller.appspotmail.com
> >
> > general protection fault, probably for non-canonical address 0xffff518084501e28: 0000 [#1] PREEMPT SMP KASAN
> > KASAN: maybe wild-memory-access in range [0xfffaac042280f140-0xfffaac042280f147]
> > CPU: 0 PID: 17449 Comm: syz-executor.5 Not tainted 5.9.0-rc5-syzkaller #0
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> > RIP: 0010:perf_misc_flags+0x125/0x150 arch/x86/events/core.c:2638
> > Code: e4 48 83 e6 03 41 0f 94 c4 31 ff e8 95 fa 73 00 bb 02 00 00 00 4c 29 e3 49 81 c6 90 00 00 00 4c 89 f0 48 c1 e8 00 00 00 00 38 <00> 74 08 4c 89 f7 e8 40 c0 b3 00 41 8b 06 83 e0 08 48 c1 e0 0b 48
>
> Hmm, so converting this back to opcodes with decodecode gives:
>
> Code: e4 48 83 e6 03 41 0f 94 c4 31 ff e8 95 fa 73 00 bb 02 00 00 00 4c 29 e3 49 81 c6 90 00 00 00 4c 89 f0 48 c1 e8 00 00 00 00 38 <00> 74 08 4c 89 f7 e8 40 c0 b3 00 41 8b 06 83 e0 08 48 c1 e0 0b 48
> All code
> ========
> 0: e4 48 in $0x48,%al
> 2: 83 e6 03 and $0x3,%esi
> 5: 41 0f 94 c4 sete %r12b
> 9: 31 ff xor %edi,%edi
> b: e8 95 fa 73 00 callq 0x73faa5
> 10: bb 02 00 00 00 mov $0x2,%ebx
> 15: 4c 29 e3 sub %r12,%rbx
> 18: 49 81 c6 90 00 00 00 add $0x90,%r14
> 1f: 4c 89 f0 mov %r14,%rax
> 22: 48 c1 e8 00 shr $0x0,%rax
> 26: 00 00 add %al,(%rax)
> 28: 00 38 add %bh,(%rax)
> 2a:* 00 74 08 4c add %dh,0x4c(%rax,%rcx,1) <-- trapping instruction
> 2e: 89 f7 mov %esi,%edi
> 30: e8 40 c0 b3 00 callq 0xb3c075
> 35: 41 8b 06 mov (%r14),%eax
> 38: 83 e0 08 and $0x8,%eax
> 3b: 48 c1 e0 0b shl $0xb,%rax
> 3f: 48 rex.W
>
> and those ADDs before the rIP look real strange. Just as if something
> wrote 4 bytes of 0s there. And building your config with clang-10 gives
> around that area:
>
> ffffffff8101177c: 48 83 e6 03 and $0x3,%rsi
> ffffffff81011780: 41 0f 94 c4 sete %r12b
> ffffffff81011784: 31 ff xor %edi,%edi
> ffffffff81011786: e8 05 c9 73 00 callq ffffffff8174e090 <__sanitizer_cov_trace_const_cmp8>
> ffffffff8101178b: bb 02 00 00 00 mov $0x2,%ebx
> ffffffff81011790: 4c 29 e3 sub %r12,%rbx
> ffffffff81011793: 49 81 c6 90 00 00 00 add $0x90,%r14
> ffffffff8101179a: 4c 89 f0 mov %r14,%rax
> ffffffff8101179d: 48 c1 e8 03 shr $0x3,%rax
> ffffffff810117a1: 42 80 3c 38 00 cmpb $0x0,(%rax,%r15,1)
> ffffffff810117a6: 74 08 je ffffffff810117b0 <perf_misc_flags+0x130>
> ffffffff810117a8: 4c 89 f7 mov %r14,%rdi
> ffffffff810117ab: e8 20 75 b3 00 callq ffffffff81b48cd0 <__asan_report_load8_noabort>
> ffffffff810117b0: 41 8b 06 mov (%r14),%eax
> ffffffff810117b3: 83 e0 08 and $0x8,%eax
> ffffffff810117b6: 48 c1 e0 0b shl $0xb,%rax
>
> and I can pretty much follow it instruction by instruction until I reach
> that SHR. Your SHR is doing a shift by 0 bytes and that already looks
> suspicious.
>
> After it, your output has a bunch of suspicious ADDs and mine has a CMP;
> JE instead. And that looks really strange too.
>
> Could it be that something has scribbled in guest memory and corrupted
> that area, leading to that strange discrepancy in the opcodes?

Hi Boris,

Memory corruption is definitely possible. There are hundreds of known
bugs that can potentially lead to silent memory corruptions, and some
observed to lead to silent memory corruptions.

However, these tend to produce crash signatures with 1-2 crashes.
While this has 6 and they look similar and all happened on the only
instance that uses clang. So my bet would be on
something-clang-related rather than a silent memory corruption.
+clang-built-linux

[Dmitry Vyukov]

general protection fault in pvclock_gtod_notify (2) looks somewhat similar:
- only clang
- gpf in systems code
- happened few times

https://syzkaller.appspot.com/bug?extid=1dccfcb049726389379c
https://groups.google.com/g/syzkaller-bugs/c/0eUUkjFKrBg/m/nGfTjIfCBAAJ

[Nick Desaulniers]

Right, the two sequences above look almost the same, except those 4
bytes of zeros (the disassembler gets confused about the rest, but
it's the same byte sequence otherwise). Are the two disassemblies a
comparison of the code at runtime vs. compile-time? If so, how did
you disassemble the runtime code? If runtime and compile time differ,
I suspect some kind of runtime patching. I wonder if we calculated
the address of a static_key wrong (asm goto). What function am I
looking at the disassembly of? perf_misc_flags() in
arch/x86/events/core.c? With this config?
https://syzkaller.appspot.com/x/.config?x=cd992d74d6c7e62 (though I
don't see _any_ asm goto in the IR for this file built with this
config). If this is deterministically reproducible, I suppose we
could set a watchpoint on the address being overwritten?

(Un-interestingly, I do get a panic trying to boot that config in
qemu, unless I bump the VMs RAM up.)

> >
> > Hi Boris,
> >
> > Memory corruption is definitely possible. There are hundreds of known
> > bugs that can potentially lead to silent memory corruptions, and some
> > observed to lead to silent memory corruptions.
> >
> > However, these tend to produce crash signatures with 1-2 crashes.
> > While this has 6 and they look similar and all happened on the only
> > instance that uses clang. So my bet would be on
> > something-clang-related rather than a silent memory corruption.
> > +clang-built-linux
>
>
> general protection fault in pvclock_gtod_notify (2) looks somewhat similar:
> - only clang
> - gpf in systems code
> - happened few times
>
> https://syzkaller.appspot.com/bug?extid=1dccfcb049726389379c
> https://groups.google.com/g/syzkaller-bugs/c/0eUUkjFKrBg/m/nGfTjIfCBAAJ

Dmitry,
Is there an easy way for me to get from
https://syzkaller.appspot.com/upstream to <list of clang specific
failures>? ctrl+f, `clang`, returns nothing on that first link; it
seems the compiler version is only included in the email?
--
Thanks,
~Nick Desaulniers

[Borislav Petkov]

On Mon, Sep 21, 2020 at 01:59:43PM -0700, Nick Desaulniers wrote:
> Right, the two sequences above look almost the same, except those 4
> bytes of zeros (the disassembler gets confused about the rest, but
> it's the same byte sequence otherwise). Are the two disassemblies a
> comparison of the code at runtime vs. compile-time?

Yes.

> If so, how did you disassemble the runtime code?

./scripts/decodecode < /tmp/splat

where /tmp/splat contains the line starting with "Code:". Make sure you
have only one "Code:"-line, otherwise you'll see the code of the *last*
Code: line only.

> If runtime and compile time differ, I suspect some kind of runtime
> patching.

If it is, it ain't patching at the right place. :)

But no, that function is pretty simple and looking at its asm, there's
no asm goto() or alternatives in there. But that .config might add them.
It adds a lot of calls to *ASAN helpers and whatnot.

> I wonder if we calculated the address of a static_key wrong
> (asm goto). What function am I looking at the disassembly of?
> perf_misc_flags() in arch/x86/events/core.c?

Yes.

> With this config?
> https://syzkaller.appspot.com/x/.config?x=cd992d74d6c7e62 (though I
> don't see _any_ asm goto in the IR for this file built with this
> config).

Right, there should be none.

> If this is deterministically reproducible, I suppose we
> could set a watchpoint on the address being overwritten?

Sounds like worth a try. I'll go sleep instead, tho. :)

Gnight and good luck.

--
Regards/Gruss,
Boris.

https://people.kernel.org/tglx/notes-about-netiquette

[Dmitry Vyukov]

Hi Nick,

Yes, it should be this config. Borislav looked at the crash in the
first email in the thread and that's the config provided there. If
syzbot provides a crash and a config, these always match.

And the exact compiler used to produce the build is this:

compiler: clang version 10.0.0
(https://github.com/llvm/llvm-project/
c2443155a0fb245c8f17f2c1c72b6ea391e86e81)

https://storage.googleapis.com/syzkaller/clang_install_c2443155.tar.gz

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/CAKwvOdmKcn%3DFNzwtBZ8z0evLz4BXgWtsoz9%2BQTC6GLqtNp1bXg%40mail.gmail.com.

[Dmitry Vyukov]

On Mon, Sep 21, 2020 at 10:59 PM 'Nick Desaulniers' via syzkaller-bugs

No such search exists.

[Nick Desaulniers]

On Mon, Sep 21, 2020 at 3:13 PM Borislav Petkov <b...@alien8.de> wrote:
>
> On Mon, Sep 21, 2020 at 01:59:43PM -0700, Nick Desaulniers wrote:
> > Right, the two sequences above look almost the same, except those 4
> > bytes of zeros (the disassembler gets confused about the rest, but
> > it's the same byte sequence otherwise). Are the two disassemblies a
> > comparison of the code at runtime vs. compile-time?
>
> Yes.
>
> > If so, how did you disassemble the runtime code?
>
> ./scripts/decodecode < /tmp/splat
>
> where /tmp/splat contains the line starting with "Code:". Make sure you
> have only one "Code:"-line, otherwise you'll see the code of the *last*
> Code: line only.

Thanks.

> > If runtime and compile time differ, I suspect some kind of runtime
> > patching.
>
> If it is, it ain't patching at the right place. :)

Yeah, but we've had this kind of bug before:
https://nickdesaulniers.github.io/blog/2020/04/06/off-by-two/
I'm sure it's not the last.

> But no, that function is pretty simple and looking at its asm, there's
> no asm goto() or alternatives in there. But that .config might add them.
> It adds a lot of calls to *ASAN helpers and whatnot.

Maybe not in this translation unit, but it's possible another TU does
have one and it miscalculates the offset; overwriting code in another
TU.

> > I wonder if we calculated the address of a static_key wrong
> > (asm goto). What function am I looking at the disassembly of?
> > perf_misc_flags() in arch/x86/events/core.c?
>
> Yes.
>
> > With this config?
> > https://syzkaller.appspot.com/x/.config?x=cd992d74d6c7e62 (though I
> > don't see _any_ asm goto in the IR for this file built with this
> > config).
>
> Right, there should be none.
>
> > If this is deterministically reproducible, I suppose we
> > could set a watchpoint on the address being overwritten?
>
> Sounds like worth a try. I'll go sleep instead, tho. :)

So I think there's an issue with "deterministically reproducible."
The syzcaller report has:

> > Unfortunately, I don't have any reproducer for this issue yet.

Following my hypothesis about having a bad address calculation; the
tricky part is I'd need to look through the relocations and try to see
if any could resolve to the address that was accidentally modified. I
suspect objtool could be leveraged for that; maybe it could check
whether each `struct jump_entry`'s `target` member referred to either
a NOP or a CMP, and error otherwise? (Do we have other non-NOP or CMP
targets? IDK)

This hypothesis might also be incorrect, and thus would be chasing a
red herring...not really sure how else to pursue debugging this.

> Gnight and good luck.

Ah, that's a famous quote from journalist Edward R Murrow, who helped
defeat Senator Joseph McCarthy (Murrow's show See It Now dedicated a
segment to addressing McCarthy). Sometimes I fund uncanny parallels
between claims of what a compiler can do on LKML "without proper
regard for evidence" and McCarthyism. Falsifiability is an
interesting trait. That's why I try to advocate for sharing links
from godbolt.org as much as possible.
--
Thanks,
~Nick Desaulniers

[Borislav Petkov]

On Tue, Sep 22, 2020 at 11:56:04AM -0700, Nick Desaulniers wrote:
> > Gnight and good luck.
>
> Ah, that's a famous quote from journalist Edward R Murrow, who helped
> defeat Senator Joseph McCarthy (Murrow's show See It Now dedicated a
> segment to addressing McCarthy).

Good. Finally someone has recognized this - I use it from time to time
in mails but no one would pick up on it!

And how relevant it is again...

https://www.youtube.com/watch?v=vEvEmkMNYHY

> Sometimes I fund uncanny parallels between claims of what a compiler
> can do on LKML "without proper regard for evidence" and McCarthyism.
> Falsifiability is an interesting trait. That's why I try to advocate
> for sharing links from godbolt.org as much as possible.

LOL. Good one. :-)

I'll take a look at the rest of your reply tomorrow because brain is
fried for today.

[Borislav Petkov]

On Tue, Sep 22, 2020 at 11:56:04AM -0700, Nick Desaulniers wrote:

> So I think there's an issue with "deterministically reproducible."
> The syzcaller report has:
> > > Unfortunately, I don't have any reproducer for this issue yet.

Yeah, Dmitry gave two other links of similar reports, the first one
works for me:

https://syzkaller.appspot.com/bug?extid=1dccfcb049726389379c

and that one doesn't have a reproducer either. The bytes look familiar
though:

Code: c1 e8 03 42 80 3c 20 00 74 05 e8 79 7a a7 00 49 8b 47 10 48 89 05 f6 d8 ef 09 49 8d 7f 08 48 89 f8 48 c1 e8 03 42 80 3c 00 00 <00> 00 e8 57 7a a7 00 49 8b 47 08 48 89 05 dc d8 ef 09 49 8d 7f 18
All code
========
0: c1 e8 03 shr $0x3,%eax
3: 42 80 3c 20 00 cmpb $0x0,(%rax,%r12,1)
8: 74 05 je 0xf
a: e8 79 7a a7 00 callq 0xa77a88
f: 49 8b 47 10 mov 0x10(%r15),%rax
13: 48 89 05 f6 d8 ef 09 mov %rax,0x9efd8f6(%rip) # 0x9efd910
1a: 49 8d 7f 08 lea 0x8(%r15),%rdi
1e: 48 89 f8 mov %rdi,%rax
21: 48 c1 e8 03 shr $0x3,%rax
25: 42 80 3c 00 00 cmpb $0x0,(%rax,%r8,1)
2a:* 00 00 add %al,(%rax) <-- trapping instruction
2c: e8 57 7a a7 00 callq 0xa77a88
31: 49 8b 47 08 mov 0x8(%r15),%rax
35: 48 89 05 dc d8 ef 09 mov %rax,0x9efd8dc(%rip) # 0x9efd918
3c: 49 8d 7f 18 lea 0x18(%r15),%rdi

4 zero bytes again. And that .config has kasan stuff enabled too so
could the failure be related to having kasan stuff enabled and it
messing up offsets?

That is, provided this is the mechanism how it would happen. We still
don't know what and when wrote those zeroes in there. Not having a
reproducer is nasty but looking at those reports above and if I'm
reading this correctly, rIP points to

RIP: 0010:update_pvclock_gtod arch/x86/kvm/x86.c:1743 [inline]

each time and the URL says they're 9 crashes total. And each have
happened at that rIP. So all we'd need is set a watchpoint when that
address is being written and dump stuff.

Dmitry, can the syzkaller do debugging stuff like that?

> Following my hypothesis about having a bad address calculation; the
> tricky part is I'd need to look through the relocations and try to see
> if any could resolve to the address that was accidentally modified. I
> suspect objtool could be leveraged for that;

If you can find this at compile time...

> maybe it could check whether each `struct jump_entry`'s `target`
> member referred to either a NOP or a CMP, and error otherwise? (Do we
> have other non-NOP or CMP targets? IDK)

Follow jump_label_transform() - it does verify what it is going to
patch. And while I'm looking at this, I realize that the jump labels
patch 5 bytes but the above zeroes are 4 bytes. In the other opcode
bytes I decoded it is 4 bytes too. So this might not be caused by the
jump labels patching...

> This hypothesis might also be incorrect, and thus would be chasing a
> red herring...not really sure how else to pursue debugging this.

Yeah, this one is tricky to debug.

[Dmitry Vyukov]

syzbot does not have direct support for such things.
It uses CONFIG_DEBUG_AID_FOR_SYZBOT=y:
https://github.com/google/syzkaller/blob/master/docs/syzbot.md#no-custom-patches
But that's generally useful for linux-next only and the clang build is
on the upstream tree...

Options I see:
1. Add stricter debug checks for code that overwrites code. Then maybe
we can catch it red handed.
2. Setup clang instance on linux-next
3. Run syzkaller locally with custom patches.

[Borislav Petkov]

On Wed, Sep 23, 2020 at 11:24:48AM +0200, Dmitry Vyukov wrote:
> 3. Run syzkaller locally with custom patches.

Let's say I wanna build the kernel with clang-10 using your .config and
run it in a vm locally. What are the steps in order to reproduce the
same workload syzkaller runs in the guest on the GCE so that I can at
least try get as close as possible to reproducing locally?

Thx.

[Dmitry Vyukov]

On Wed, Sep 23, 2020 at 12:34 PM Borislav Petkov <b...@alien8.de> wrote:
>
> On Wed, Sep 23, 2020 at 11:24:48AM +0200, Dmitry Vyukov wrote:
> > 3. Run syzkaller locally with custom patches.
>
> Let's say I wanna build the kernel with clang-10 using your .config and
> run it in a vm locally. What are the steps in order to reproduce the
> same workload syzkaller runs in the guest on the GCE so that I can at
> least try get as close as possible to reproducing locally?

It's a random fuzzing workload. You can get this workload by running
syzkaller locally:
https://github.com/google/syzkaller/blob/master/docs/linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md

The exact clang compiler syzbot used is available here:
https://github.com/google/syzkaller/blob/master/docs/syzbot.md#crash-does-not-reproduce

[Dmitry Vyukov]

I've marked all other similar ones a dup of this one. Now you can see
all manifestations on the dashboard:
https://syzkaller.appspot.com/bug?extid=ce179bc99e64377c24bc

Another possible debugging vector on this:
The location of crashes does not seem to be completely random and
evenly spread across kernel code. I think there are many more static
branches (mm, net), but we have 3 crashes in vdso and 9 in paravirt
code + these 6 crashes in perf_misc_flags which looks a bit like an
outlier (?). What's special about paravirt/vdso?..

[Nick Desaulniers]

On Fri, Sep 25, 2020 at 5:22 AM Dmitry Vyukov <dvy...@google.com> wrote:
>
> On Wed, Sep 23, 2020 at 5:20 PM Dmitry Vyukov <dvy...@google.com> wrote:
> >
> > On Wed, Sep 23, 2020 at 12:34 PM Borislav Petkov <b...@alien8.de> wrote:
> > >
> > > On Wed, Sep 23, 2020 at 11:24:48AM +0200, Dmitry Vyukov wrote:
> > > > 3. Run syzkaller locally with custom patches.
> > >
> > > Let's say I wanna build the kernel with clang-10 using your .config and
> > > run it in a vm locally. What are the steps in order to reproduce the
> > > same workload syzkaller runs in the guest on the GCE so that I can at
> > > least try get as close as possible to reproducing locally?
> >
> > It's a random fuzzing workload. You can get this workload by running
> > syzkaller locally:
> > https://github.com/google/syzkaller/blob/master/docs/linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md

These are virtualized guests, right? Has anyone played with getting
`rr` working to record traces of guests in QEMU?

I had seen the bug that generated this on github:
https://julialang.org/blog/2020/09/rr-memory-magic/

That way, even if syzkaller didn't have a reproducer binary, it would
at least have a replayable trace.

Boris, one question I have. Doesn't the kernel mark pages backing
executable code as read only at some point? If that were the case,
then I don't see how the instruction stream could be modified. I
guess static key patching would have to undo that permission mapping
before patching.

You're right about the length shorter than what I would have expected
from static key patching. That could very well be a write through
dangling int pointer...

> >
> > The exact clang compiler syzbot used is available here:
> > https://github.com/google/syzkaller/blob/master/docs/syzbot.md#crash-does-not-reproduce
>
> I've marked all other similar ones a dup of this one. Now you can see
> all manifestations on the dashboard:
> https://syzkaller.appspot.com/bug?extid=ce179bc99e64377c24bc
>
> Another possible debugging vector on this:
> The location of crashes does not seem to be completely random and
> evenly spread across kernel code. I think there are many more static
> branches (mm, net), but we have 3 crashes in vdso and 9 in paravirt
> code + these 6 crashes in perf_misc_flags which looks a bit like an
> outlier (?). What's special about paravirt/vdso?..

--
Thanks,
~Nick Desaulniers

[Dmitry Vyukov]

On Sat, Sep 26, 2020 at 2:32 AM 'Nick Desaulniers' via syzkaller-bugs
<syzkall...@googlegroups.com> wrote:
> > > > On Wed, Sep 23, 2020 at 11:24:48AM +0200, Dmitry Vyukov wrote:
> > > > > 3. Run syzkaller locally with custom patches.
> > > >
> > > > Let's say I wanna build the kernel with clang-10 using your .config and
> > > > run it in a vm locally. What are the steps in order to reproduce the
> > > > same workload syzkaller runs in the guest on the GCE so that I can at
> > > > least try get as close as possible to reproducing locally?
> > >
> > > It's a random fuzzing workload. You can get this workload by running
> > > syzkaller locally:
> > > https://github.com/google/syzkaller/blob/master/docs/linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md
>
> These are virtualized guests, right? Has anyone played with getting
> `rr` working to record traces of guests in QEMU?
>
> I had seen the bug that generated this on github:
> https://julialang.org/blog/2020/09/rr-memory-magic/
>
> That way, even if syzkaller didn't have a reproducer binary, it would
> at least have a replayable trace.

These are virtualized guests, but they run on GCE, not in QEMU.

> Boris, one question I have. Doesn't the kernel mark pages backing
> executable code as read only at some point? If that were the case,
> then I don't see how the instruction stream could be modified. I
> guess static key patching would have to undo that permission mapping
> before patching.
>
> You're right about the length shorter than what I would have expected
> from static key patching. That could very well be a write through
> dangling int pointer...
>
> > >
> > > The exact clang compiler syzbot used is available here:
> > > https://github.com/google/syzkaller/blob/master/docs/syzbot.md#crash-does-not-reproduce
> >
> > I've marked all other similar ones a dup of this one. Now you can see
> > all manifestations on the dashboard:
> > https://syzkaller.appspot.com/bug?extid=ce179bc99e64377c24bc
> >
> > Another possible debugging vector on this:
> > The location of crashes does not seem to be completely random and
> > evenly spread across kernel code. I think there are many more static
> > branches (mm, net), but we have 3 crashes in vdso and 9 in paravirt
> > code + these 6 crashes in perf_misc_flags which looks a bit like an
> > outlier (?). What's special about paravirt/vdso?..
>
>
>
> --
> Thanks,
> ~Nick Desaulniers
>

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/CAKwvOdkYEP%3DoRtEu_89JBq2g41PL9_FuFyfeB94XwBKuSz4XLg%40mail.gmail.com.

[Borislav Petkov]

On Wed, Sep 23, 2020 at 05:20:06PM +0200, Dmitry Vyukov wrote:
> It's a random fuzzing workload. You can get this workload by running
> syzkaller locally:
> https://github.com/google/syzkaller/blob/master/docs/linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md

Yeah, the my.cfg example suggests that the syz-manager starts the guest
and supplies the kernel, etc.

Is there a possibility to run the workload in an already existing guest
which I've booted prior?

I'm asking because I have all the infra for testing kernels in guests
already setup here and it would be easier for me to simply run the
workload directly in the guest and then poke at it.

[Dmitry Vyukov]

On Sat, Sep 26, 2020 at 1:21 PM Borislav Petkov <b...@alien8.de> wrote:
>
> On Wed, Sep 23, 2020 at 05:20:06PM +0200, Dmitry Vyukov wrote:
> > It's a random fuzzing workload. You can get this workload by running
> > syzkaller locally:
> > https://github.com/google/syzkaller/blob/master/docs/linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md
>
> Yeah, the my.cfg example suggests that the syz-manager starts the guest
> and supplies the kernel, etc.
>
> Is there a possibility to run the workload in an already existing guest
> which I've booted prior?
>
> I'm asking because I have all the infra for testing kernels in guests
> already setup here and it would be easier for me to simply run the
> workload directly in the guest and then poke at it.

+syzkaller mailing list

There is also "isolated" VM type, which allows to connect to a set of
external machines via ssh:
https://github.com/google/syzkaller/blob/master/vm/isolated/isolated.go#L29-L37
However, it's better to have lots of them and with a console cables,
and still sometimes they may brick for various reasons.

There is also syz-stress utility that may run some workload directly
on the underlying kernel:
https://github.com/google/syzkaller/blob/master/tools/syz-stress/stress.go#L29
However, it does not use corpus/coverage, so I don't know if it will
be able to reproduce these crashes or not. It will also be up to you
then to restart the VM/fuzzing every minute.

[Borislav Petkov]

On Fri, Sep 25, 2020 at 05:32:14PM -0700, Nick Desaulniers wrote:
> Boris, one question I have. Doesn't the kernel mark pages backing
> executable code as read only at some point?

Yes, I added some debug output:

[ 562.959995][ T1] Freeing unused kernel image (initmem) memory: 2548K
[ 563.672645][ T1] Write protecting the kernel read-only data: 137216k [0xffffffff81000000:0xffffffff89600000]

and perf_misc_flags() is well within that range:

ffffffff810118e0 <perf_misc_flags>:

[ 566.076923][ T1] unused kernel image (text/rodata gap): [0xffffffff88608000:0xffffffff88800000]
[ 567.039076][ T1] unused kernel image (rodata/data gap): [0xffffffff8941d000:0xffffffff89600000]
[ 568.205550][ T1] Freeing unused kernel image (text/rodata gap) memory: 2016K
[ 569.277742][ T1] Freeing unused kernel image (rodata/data gap) memory: 1932K

We also have this debug option which I enabled:

[ 570.598533][ T1] x86/mm: Checked W+X mappings: passed, no W+X pages found.

so that looks ok too.

> If that were the case, then I don't see how the instruction stream
> could be modified. I guess static key patching would have to undo that
> permission mapping before patching.

Yap, and I still have no clue about the mechanism which would lead to
this corruption.

> You're right about the length shorter than what I would have expected
> from static key patching. That could very well be a write through
> dangling int pointer...

Right.

Lemme try to setup one of my test boxes to run syzkaller and see how far
I can get. But don't hold your breath...

[Dmitry Vyukov]

On Sun, Sep 27, 2020 at 4:57 PM Borislav Petkov <b...@alien8.de> wrote:
>
> On Sat, Sep 19, 2020 at 01:32:14AM -0700, syzbot wrote:
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit: 92ab97ad Merge tag 'sh-for-5.9-part2' of git://git.libc.or..
> > git tree: upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=1069669b900000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=cd992d74d6c7e62
> > dashboard link: https://syzkaller.appspot.com/bug?extid=ce179bc99e64377c24bc
> > compiler: clang version 10.0.0 (https://github.com/llvm/llvm-project/ c2443155a0fb245c8f17f2c1c72b6ea391e86e81)
>

> All below is AFAICT:
>
> This compiler you're using is not some official release but some random
> commit before the v10 release:
>
> $ git show c2443155a0fb245c8f17f2c1c72b6ea391e86e81
> Author: Hans Wennborg <ha...@chromium.org>
> Date: Sat Nov 30 14:20:11 2019 +0100
>
> Revert 651f07908a1 "[AArch64] Don't combine callee-save and local stack adjustment when optimizing for size"
> ...
>
> $ git describe c2443155a0fb245c8f17f2c1c72b6ea391e86e81
> llvmorg-10-init-10900-gc2443155a0fb
>
> The v10 release is:
>
> $ git show llvmorg-10.0.0
> tag llvmorg-10.0.0
> Tagger: Hans Wennborg <ha...@chromium.org>
> Date: Tue Mar 24 12:58:58 2020 +0100
>
> Tag 10.0.0
>
> and v10 has reached v10.0.1 in the meantime:
>
> $ git log --oneline c2443155a0fb245c8f17f2c1c72b6ea391e86e81~1..llvmorg-10.0.1 | wc -l
> 7051
>
> so can you please update your compiler and see if you can still
> reproduce with 10.0.1 so that we don't waste time chasing a bug which
> has been likely already fixed in one of those >7K commits.

+Alex, Marco,

There is suspicion that these may be caused by use of unreleased clang.
Do we use the same clang as we use for the KMSAN instance? But this is
not KMSAN machine, so I am not sure who/when/why updated it last to
this revision.
I even see we have some clang 11 version:
https://github.com/google/syzkaller/blob/master/docs/syzbot.md#crash-does-not-reproduce

Is it possible to switch to some released version for both KMSAN and KASAN now?

[Dmitry Vyukov]

Interestingly there is a new crash, which looks similar:

general protection fault in map_vdso
https://syzkaller.appspot.com/bug?extid=c2ae01c2b1b385384a06

The code is also with 4 0's:
Code: 00 00 00 48 b8 00 00 00 00 00 fc ff df 41 57 49 89 ff 41 56 41
55 41 54 55 65 48 8b 2c 25 c0 fe 01 00 48 8d bd 28 04 00 00 53 <48> 00
00 00 00 fa 48 83 ec 10 48 c1 ea 03 80 3c 02 00 0f 85 51 02

But it happened with gcc.

Also I found this older one:
general protection fault in map_vdso_randomized
https://syzkaller.appspot.com/bug?id=8366fd024559946137b9db23b26fd2235d43b383

which also has code smashed and happened with gcc:
Code: 00 fc ff df 48 89 f9 48 c1 e9 03 80 3c 01 00 0f 85 eb 00 00 00
65 48 8b 1c 25 c0 fe 01 00 48 8d bb 28 04 00 00 41 2b 54 24 20 <00> 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

I think there may be dozens older ones here:
https://syzkaller.appspot.com/upstream#moderation2
e.g. this one where code also looks strange:
https://syzkaller.appspot.com/bug?id=651c61721c822bfdcdae8bfb9320e4a9b4bd49c9

Maybe it's just a random silent memory corruption in the end?...

[Marco Elver]

Yeah, we should replace that one as well as it wasn't yet a release-candidate.

> Is it possible to switch to some released version for both KMSAN and KASAN now?

We should probably just switch to Clang 11-rc3 or so. Then we can use
the same compiler for KMSAN and KCSAN at least.

I can package up a newer Clang.

Thanks,
-- Marco

[Dmitry Vyukov]

On Mon, Sep 28, 2020 at 10:38 AM Borislav Petkov <b...@alien8.de> wrote:

>
> On Mon, Sep 28, 2020 at 08:06:19AM +0200, Dmitry Vyukov wrote:
> > Maybe it's just a random silent memory corruption in the end?...
>

> Oh, the rabbit hole goes deeper. But if it is corruption, what is the
> common element in all those? All those guests have run on the same
> physical machine?
>
> If so, you probably won't get any logs from it to search for MCEs or
> so...? Maybe the GCE people can do some grepping :)

I meant the kernel self-corrupts itself, that just wasn't detected by
KASAN, page protections, etc.

[Dmitry Vyukov]

On Mon, Sep 28, 2020 at 10:54 AM Borislav Petkov <b...@alien8.de> wrote:

>
> On Mon, Sep 28, 2020 at 10:40:19AM +0200, Dmitry Vyukov wrote:
> > I meant the kernel self-corrupts itself, that just wasn't detected by
> > KASAN, page protections, etc.
>

> Well, Nick already asked this but we're marking all kernel text RO early
> during boot. So it either is happening before that or something else
> altogether is going on.
>
> And if that is a kernel issue, I believe we should've heard by now from
> others. Or maybe this happens only in VMs.
>
> Questions over questions...

I don't have answers to all of the questions, but syzkaller produces a
pretty unique workload. It has found thousands of bugs that you have
not heard from others:
https://syzkaller.appspot.com/upstream#open
https://syzkaller.appspot.com/upstream/fixed
In particular there are hundreds of known and active potential memory
corruption bugs.

It may be related to VMs, but also may well not be related to VMs.

[Borislav Petkov]

On Mon, Sep 28, 2020 at 12:33:57PM +0200, Dmitry Vyukov wrote:
> It may be related to VMs, but also may well not be related to VMs.

Right, and so I tried to set it up on a test box here, it looks like
it worked, see below. I'll let it fuzz in the coming days and see what
explodes...

2020/09/28 22:19:51 booting test machines...
2020/09/28 22:19:51 wait for the connection from test machine...
2020/09/28 22:20:27 machine check:
2020/09/28 22:20:27 syscalls : 3389/3739
2020/09/28 22:20:27 code coverage : enabled
2020/09/28 22:20:27 comparison tracing : enabled
2020/09/28 22:20:27 extra coverage : enabled
2020/09/28 22:20:27 setuid sandbox : enabled
2020/09/28 22:20:27 namespace sandbox : enabled
2020/09/28 22:20:27 Android sandbox : enabled
2020/09/28 22:20:27 fault injection : enabled
2020/09/28 22:20:27 leak checking : CONFIG_DEBUG_KMEMLEAK is not enabled
2020/09/28 22:20:27 net packet injection : enabled
2020/09/28 22:20:27 net device setup : enabled
2020/09/28 22:20:27 concurrency sanitizer : /sys/kernel/debug/kcsan does not exist
2020/09/28 22:20:27 devlink PCI setup : PCI device 0000:00:10.0 is not available
2020/09/28 22:20:27 USB emulation : enabled
2020/09/28 22:20:27 hci packet injection : enabled
2020/09/28 22:20:27 wifi device emulation : enabled
2020/09/28 22:20:29 corpus : 458 (deleted 0 broken)
2020/09/28 22:20:31 seeds : 620/667
2020/09/28 22:20:31 VMs 1, executed 0, corpus cover 0, corpus signal 0, max signal 0, crashes 0, repro 0
2020/09/28 22:20:41 VMs 2, executed 12, corpus cover 0, corpus signal 0, max signal 0, crashes 0, repro 0
2020/09/28 22:20:51 VMs 2, executed 28, corpus cover 5578, corpus signal 5925, max signal 10155, crashes 0, repro 0
2020/09/28 22:21:01 VMs 3, executed 179, corpus cover 11792, corpus signal 10881, max signal 19337, crashes 0, repro 0

[Nick Desaulniers]

Oh, shoot, sorry I didn't catch that. Good find. My next question was
going to be if this is reproducible with a newer compiler release or
not (later emails make this sound like it's no longer considered clang
specific).

Generally we want coverage of unreleased compiler versions to ensure
we don't ship a broken release. Once the release exists, it's of
questionable value to continue to test a pre-release version of that
branch.

This isn't the first time where we've had syzcaller reports that were
testing old releases of clang. Maybe we can establish a process for
upgrading the toolchain under test based on some time based cadence,
or coinciding with the upstream LLVM release events?

>
> +Alex, Marco,
>
> There is suspicion that these may be caused by use of unreleased clang.
> Do we use the same clang as we use for the KMSAN instance? But this is
> not KMSAN machine, so I am not sure who/when/why updated it last to
> this revision.
> I even see we have some clang 11 version:
> https://github.com/google/syzkaller/blob/master/docs/syzbot.md#crash-does-not-reproduce
>
> Is it possible to switch to some released version for both KMSAN and KASAN now?

--
Thanks,
~Nick Desaulniers

[Nick Desaulniers]

On Mon, Sep 28, 2020 at 3:34 AM 'Dmitry Vyukov' via Clang Built Linux
<clang-bu...@googlegroups.com> wrote:
>
> On Mon, Sep 28, 2020 at 10:54 AM Borislav Petkov <b...@alien8.de> wrote:
> >
> > On Mon, Sep 28, 2020 at 10:40:19AM +0200, Dmitry Vyukov wrote:
> > > I meant the kernel self-corrupts itself, that just wasn't detected by
> > > KASAN, page protections, etc.
> >
> > Well, Nick already asked this but we're marking all kernel text RO early
> > during boot. So it either is happening before that or something else
> > altogether is going on.

On Sun, Sep 27, 2020 at 11:06 PM 'Dmitry Vyukov' via Clang Built Linux
<clang-bu...@googlegroups.com> wrote:
>
> Interestingly there is a new crash, which looks similar:
>
> general protection fault in map_vdso
> https://syzkaller.appspot.com/bug?extid=c2ae01c2b1b385384a06
>
> The code is also with 4 0's:
> Code: 00 00 00 48 b8 00 00 00 00 00 fc ff df 41 57 49 89 ff 41 56 41
> 55 41 54 55 65 48 8b 2c 25 c0 fe 01 00 48 8d bd 28 04 00 00 53 <48> 00
> 00 00 00 fa 48 83 ec 10 48 c1 ea 03 80 3c 02 00 0f 85 51 02
>
> But it happened with gcc.
>
> Also I found this older one:
> general protection fault in map_vdso_randomized
> https://syzkaller.appspot.com/bug?id=8366fd024559946137b9db23b26fd2235d43b383
>
> which also has code smashed and happened with gcc:
> Code: 00 fc ff df 48 89 f9 48 c1 e9 03 80 3c 01 00 0f 85 eb 00 00 00
> 65 48 8b 1c 25 c0 fe 01 00 48 8d bb 28 04 00 00 41 2b 54 24 20 <00> 00
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

If this is related to vdso's, they seem mapped as `r-xp` (not `w):
$ sudo cat /proc/1/maps | grep vdso
7ffc667f5000-7ffc667f7000 r-xp 00000000 00:00 0 [vdso]

map_vdso() in arch/x86/entry/vdso/vma.c doesn't map the VMA as
writable, but it uses VM_MAYWRITE with a comment about GDB setting
breakpoints.

So it sounds like the page protections on the vdso can be changed at
runtime (via mprotect). Maybe syzkaller is tickling that first?

map_vdso_randomized() does call map_vdso(). Maybe if we mprotect the
vdso to be writable, it may be easier to spot the write.

--
Thanks,
~Nick Desaulniers

[Andy Lutomirski]

> On Sep 28, 2020, at 1:52 PM, Nick Desaulniers <ndesau...@google.com> wrote:
>
> On Mon, Sep 28, 2020 at 3:34 AM 'Dmitry Vyukov' via Clang Built Linux

The kernel shouldn’t be executing the vDSO code. Unless I’ve misread it, Te crash is that the map_vdso() text itself was corrupted. This isn’t the same thing.

The VM_MAYWRITE means that a program may CoW the page and write to the copy, which still won’t allow changing the vDSO text or executing it inside the kernel.

[Borislav Petkov]

On Mon, Sep 28, 2020 at 10:40:19AM +0200, Dmitry Vyukov wrote:
> I meant the kernel self-corrupts itself, that just wasn't detected by
> KASAN, page protections, etc.

Well, Nick already asked this but we're marking all kernel text RO early
during boot. So it either is happening before that or something else
altogether is going on.

And if that is a kernel issue, I believe we should've heard by now from
others. Or maybe this happens only in VMs.

Questions over questions...

[Borislav Petkov]

On Mon, Sep 28, 2020 at 08:06:19AM +0200, Dmitry Vyukov wrote:
> Maybe it's just a random silent memory corruption in the end?...

Oh, the rabbit hole goes deeper. But if it is corruption, what is the
common element in all those? All those guests have run on the same
physical machine?

If so, you probably won't get any logs from it to search for MCEs or
so...? Maybe the GCE people can do some grepping :)

[Borislav Petkov]

On Mon, Sep 28, 2020 at 10:23:53PM +0200, Borislav Petkov wrote:
> 2020/09/28 22:21:01 VMs 3, executed 179, corpus cover 11792, corpus signal 10881, max signal 19337, crashes 0, repro 0

Ok, so far triggered two things:

WARNING in f2fs_is_valid_blkaddr 1 2020/09/29 10:27 reproducing
WARNING in reiserfs_put_super 1 2020/09/28 22:42

you've probably seen them already.

Anyway, next question. Let's say I trigger the corruption: is there a
way to stop the guest VM which has triggered it so that I'm able to
examine it with gdb?

What about kdump? Can I dump the guest memory either with kdump or
through the qemu monitor (I believe there's a command to dump memory) so
that it can be poked at?

Because as it is, we don't have a reproducer and as I see it, the fuzzing simply
gets restarted:

2020/09/29 10:27:03 vm-3: crash: WARNING in f2fs_is_valid_blkaddr
...
2020/09/29 10:27:05 loop: phase=1 shutdown=false instances=1/4 [3] repro: pending=0 reproducing=1 queued=1
2020/09/29 10:27:05 loop: starting instance 3

so it would be good to be able to say, when a vm encounters a crash, it
should be stopped immediately so that the guest can be examined through
qemu's gdb interface, i.e.,

-gdb tcp::<portnum>

or so?

Thx.

[Dmitry Vyukov]

The current hypothesis is that this bug is not related to clang (there
are similar crashes with gcc as well).
We use unreleased versions of clang as we frequently need recent
fixes/features. And then later nobody usually has time to update, if
things work.
Based on offline discussion with Marco, we probably need to update
KMSAN and KASAN to 11 release when it's released.

[Dmitry Vyukov]

Currently there is no such feature. I think some people did it because
something similar was mentioned on the mailing IIRC, but I don't know
how they did it, probably with some local code changes.

[Borislav Petkov]

Hi,

one more thing I just spotted. The default install of syzkaller here
runs the guest with this on the kernel command line:

2020/09/30 17:56:18 running command: qemu-system-x86_64 []string{"-m", "2048",
"-smp", "2", "-display", ... "-append", "earlyprintk=serial oops=panic ...
nmi_watchdog=panic panic_on_warn=1 panic=1 ftrace_dump_on_oops=orig_cpu rodata=n
^^^^^^^^^^

which basically leaves guest kernel's memory RW and it gets caught
immediately on vm boot by CONFIG_DEBUG_WX.

This pretty much explains why kernel text can get corrupted with a stray
pointer write or so. So what's the use case for rodata=n?

[ 2.478136] Kernel memory protection disabled.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[ 2.478689] x86/mm: Checking user space page tables
[ 2.550163] ------------[ cut here ]------------
[ 2.550736] x86/mm: Found insecure W+X mapping at address entry_SYSCALL_64+0x0/0x29
[ 2.551612] WARNING: CPU: 1 PID: 1 at arch/x86/mm/dump_pagetables.c:246 note_page+0x81f/0x13a0
[ 2.552577] Kernel panic - not syncing: panic_on_warn set ...
[ 2.553240] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.9.0-rc7+ #5
[ 2.553953] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-1 04/01/2014
[ 2.554922] Call Trace:
[ 2.555233] dump_stack+0x9c/0xcf
[ 2.555633] panic+0x250/0x5a0
[ 2.556004] ? __warn_printk+0xf8/0xf8
[ 2.556450] ? console_trylock+0xb0/0xb0
[ 2.556914] ? __warn.cold+0x5/0x44
[ 2.557332] ? note_page+0x81f/0x13a0
[ 2.557768] __warn.cold+0x20/0x44
[ 2.558176] ? note_page+0x81f/0x13a0
[ 2.558641] report_bug+0x168/0x1b0
[ 2.559059] handle_bug+0x3c/0x60
[ 2.559458] exc_invalid_op+0x14/0x40
[ 2.559894] asm_exc_invalid_op+0x12/0x20
[ 2.560368] RIP: 0010:note_page+0x81f/0x13a0
[ 2.560870] Code: 26 00 80 3d 9a d0 7f 02 00 0f 85 82 f9 ff ff e8 47 3c 26 00 4c 89 e6 48 c7 c7 40 aff
[ 2.562951] RSP: 0000:ffff88800e9f7a90 EFLAGS: 00010282
[ 2.563554] RAX: 0000000000000000 RBX: ffff88800e9f7e00 RCX: 0000000000000000
[ 2.564361] RDX: ffff88800e9edb80 RSI: 0000000000000004 RDI: ffffed1001d3ef44
[ 2.565167] RBP: 0000000000000200 R08: 0000000000000001 R09: 0000000000000003
[ 2.565973] R10: ffffed1001d3eefd R11: 0000000000000001 R12: ffffffff96e00000
[ 2.566780] R13: 00000000000001e3 R14: 0000000000000000 R15: ffff88800e9f7e58
[ 2.567587] ? __kprobes_text_end+0xb3598/0xb3598
[ 2.568122] ? __entry_text_end+0x1fea85/0x1fea85
[ 2.568754] ? __entry_text_end+0x1fea85/0x1fea85
[ 2.569288] ? __entry_text_end+0x1fea85/0x1fea85
[ 2.569821] ptdump_hole+0x61/0x90
[ 2.570212] ? ptdump_pte_entry+0x100/0x100
[ 2.570712] walk_pgd_range+0xdb8/0x15f0
[ 2.571178] walk_page_range_novma+0xd9/0x140
[ 2.571689] ? walk_page_range+0x2b0/0x2b0
[ 2.572171] ? console_unlock+0x58f/0xb10
[ 2.572644] ptdump_walk_pgd+0xcd/0x180
[ 2.573099] ptdump_walk_pgd_level_core+0x13c/0x1b0
[ 2.573663] ? effective_prot+0xb0/0xb0
[ 2.574117] ? vprintk_emit+0x214/0x380
[ 2.574601] ? ptdump_walk_pgd_level_core+0x1b0/0x1b0
[ 2.575186] ? memtype_copy_nth_element+0x1a0/0x1a0
[ 2.575752] ? __kprobes_text_end+0xb3598/0xb3598
[ 2.576300] ? pti_user_pagetable_walk_pmd+0x130/0x460
[ 2.576894] ? __kprobes_text_end+0xb3598/0xb3598
[ 2.577441] ? __kprobes_text_end+0xb3598/0xb3598
[ 2.577988] ? __kprobes_text_end+0xb3598/0xb3598
[ 2.578564] ? rest_init+0xdd/0xdd
[ 2.578972] ptdump_walk_user_pgd_level_checkwx.cold+0x31/0x36
[ 2.579640] pti_finalize+0x7b/0x170
[ 2.580066] kernel_init+0x5b/0x183
[ 2.580484] ret_from_fork+0x22/0x30
[ 2.581010] Dumping ftrace buffer:
[ 2.581456] (ftrace buffer empty)
ffffffffbfffffff)
[ 2.583137] Rebooting in 1 seconds..
2020/09/30 17:56:23 failed to create instance: failed to read from qemu: EOF

[Dmitry Vyukov]

On Wed, Sep 30, 2020 at 6:17 PM Borislav Petkov <b...@alien8.de> wrote:
>
> Hi,
>
> one more thing I just spotted. The default install of syzkaller here
> runs the guest with this on the kernel command line:
>
> 2020/09/30 17:56:18 running command: qemu-system-x86_64 []string{"-m", "2048",
> "-smp", "2", "-display", ... "-append", "earlyprintk=serial oops=panic ...
> nmi_watchdog=panic panic_on_warn=1 panic=1 ftrace_dump_on_oops=orig_cpu rodata=n
> ^^^^^^^^^^
>
> which basically leaves guest kernel's memory RW and it gets caught
> immediately on vm boot by CONFIG_DEBUG_WX.
>
> This pretty much explains why kernel text can get corrupted with a stray
> pointer write or so. So what's the use case for rodata=n?
>
> [ 2.478136] Kernel memory protection disabled.
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Ha!

Here is the answer:
https://github.com/google/syzkaller/blob/master/tools/create-gce-image.sh#L189

# rodata=n: mark_rodata_ro becomes very slow with KASAN (lots of PGDs)

I have some vague memory that there was some debug double checking
that pages are indeed read-only and that debug check was slow, but it
was always executed without rodata=n.

[Dmitry Vyukov]

On Wed, Sep 30, 2020 at 6:23 PM Dmitry Vyukov <dvy...@google.com> wrote:
> > Hi,
> >
> > one more thing I just spotted. The default install of syzkaller here
> > runs the guest with this on the kernel command line:
> >
> > 2020/09/30 17:56:18 running command: qemu-system-x86_64 []string{"-m", "2048",
> > "-smp", "2", "-display", ... "-append", "earlyprintk=serial oops=panic ...
> > nmi_watchdog=panic panic_on_warn=1 panic=1 ftrace_dump_on_oops=orig_cpu rodata=n
> > ^^^^^^^^^^
> >
> > which basically leaves guest kernel's memory RW and it gets caught
> > immediately on vm boot by CONFIG_DEBUG_WX.
> >
> > This pretty much explains why kernel text can get corrupted with a stray
> > pointer write or so. So what's the use case for rodata=n?
> >
> > [ 2.478136] Kernel memory protection disabled.
> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
>
> Ha!
>
> Here is the answer:
> https://github.com/google/syzkaller/blob/master/tools/create-gce-image.sh#L189
>
> # rodata=n: mark_rodata_ro becomes very slow with KASAN (lots of PGDs)
>
> I have some vague memory that there was some debug double checking
> that pages are indeed read-only and that debug check was slow, but it
> was always executed without rodata=n.

I don't see this is still the case. Diff between 2 boots:

[ 11.985152][ T1] Freeing unused kernel image (initmem) memory: 3432K
[ 11.986129][ T1] Write protecting the kernel read-only data: 147456k
[ 11.990863][ T1] Freeing unused kernel image (text/rodata gap)
memory: 2012K
[ 11.992797][ T1] Freeing unused kernel image (rodata/data gap)
memory: 1324K
[ 11.993895][ T1] Run /sbin/init as init process

[ 11.910396][ T1] Freeing unused kernel image (initmem) memory: 3432K
[ 11.911277][ T1] Kernel memory protection disabled.
[ 11.911984][ T1] Run /sbin/init as init process

Was it fixed at some point? Was it backported to stable?

[Borislav Petkov]

On Wed, Sep 30, 2020 at 06:23:44PM +0200, Dmitry Vyukov wrote:
> Here is the answer:
> https://github.com/google/syzkaller/blob/master/tools/create-gce-image.sh#L189
>
> # rodata=n: mark_rodata_ro becomes very slow with KASAN (lots of PGDs)
>
> I have some vague memory that there was some debug double checking
> that pages are indeed read-only and that debug check was slow, but it
> was always executed without rodata=n.

Sounds like debug_checkwx() which is disabled by turning off
CONFIG_DEBUG_WX.

You could either disable it in your .configs or, provided there's even
such an option, disable KASAN checking around it until that one-time
boot test completes and then reenable KASAN.

[Dmitry Vyukov]

On Wed, Sep 30, 2020 at 6:31 PM Borislav Petkov <b...@alien8.de> wrote:
>
> On Wed, Sep 30, 2020 at 06:23:44PM +0200, Dmitry Vyukov wrote:
> > Here is the answer:
> > https://github.com/google/syzkaller/blob/master/tools/create-gce-image.sh#L189
> >
> > # rodata=n: mark_rodata_ro becomes very slow with KASAN (lots of PGDs)
> >
> > I have some vague memory that there was some debug double checking
> > that pages are indeed read-only and that debug check was slow, but it
> > was always executed without rodata=n.
>
> Sounds like debug_checkwx() which is disabled by turning off
> CONFIG_DEBUG_WX.
>
> You could either disable it in your .configs or, provided there's even
> such an option, disable KASAN checking around it until that one-time
> boot test completes and then reenable KASAN.

Thanks!

I've prepared a change that removes rodata=n:
https://github.com/google/syzkaller/pull/2155

I think we will be able to indirectly evaluate if it helps or not over
some period of time based on occurrence of any new similar crashes.

[Borislav Petkov]

On Thu, Oct 01, 2020 at 12:23:08PM +0200, Dmitry Vyukov wrote:
> I've prepared a change that removes rodata=n:
> https://github.com/google/syzkaller/pull/2155

Looks good.

> I think we will be able to indirectly evaluate if it helps or not over
> some period of time based on occurrence of any new similar crashes.

Yap, that would be interesting to see whether those corruptions
disappear. If they do, you probably should start getting #PFs of writes
to RO memory, instead, resulting from those stray writes. We'll see.

Thx.