KCOV and KASAN for external modules

[Bhargava Shastry]

Hi all,

Have any of you gotten around to compiling external kernel modules (in-tree build) with KCOV and KASAN instrumentation. I have enabled CONFIG_KCOV/KASAN and verified that the kernel image has been properly instrumented. It's just that external kernel modules have not been instrumented.

I am guessing once the ext modules are instrumented, syzkaller will work off-the-shelf?

Thanks,

Bhargava

[Dmitry Vyukov]

On Mon, Oct 23, 2017 at 9:28 PM, Bhargava Shastry <bsh...@gmail.com> wrote:
> Hi all,
>
> Have any of you gotten around to compiling external kernel modules (in-tree
> build) with KCOV and KASAN instrumentation. I have enabled CONFIG_KCOV/KASAN
> and verified that the kernel image has been properly instrumented. It's just
> that external kernel modules have not been instrumented.

Hi,

It should work. Makefile uses the same CFLAGS when compiling core
kernel and modules.

I've just enabled CONFIG_TEST_KASAN=m on:

commit 36ef71cae353f88fd6e095e2aaa3e5953af1685d (HEAD, tag:
next-20171018, linux-next/master)
Date: Thu Oct 19 01:15:23 2017 +0100
Add linux-next specific files for 20171018

and objdump -d ./lib/test_kasan.ko shows me:

./lib/test_kasan.ko: file format elf64-x86-64
Disassembly of section .init.text:
0000000000000000 <kasan_stack_oob>:
0: 55 push %rbp
1: 48 89 e5 mov %rsp,%rbp
4: 41 55 push %r13
6: 41 54 push %r12
8: 53 push %rbx
9: 48 8d 9d 48 ff ff ff lea -0xb8(%rbp),%rbx
10: 49 bd 00 00 00 00 00 movabs $0xdffffc0000000000,%r13
17: fc ff df
1a: 48 c1 eb 03 shr $0x3,%rbx
1e: 48 81 ec a0 00 00 00 sub $0xa0,%rsp
25: 48 c7 85 48 ff ff ff movq $0x41b58ab3,-0xb8(%rbp)


That 0xdffffc0000000000 is inline KASAN instrumentation.

> I am guessing once the ext modules are instrumented, syzkaller will work
> off-the-shelf?

Yes. It should not matter if code is in module or not.
One note is syzkaller/KCOV can misbehave if a modules are unloaded and
loaded again at different addresses, because we will see new PCs
covered all the time. But if modules are loaded once it should work
fine.

[mike.s...@gmail.com]

Yes. It should not matter if code is in module or not.
One note is syzkaller/KCOV can misbehave if a modules are unloaded and
loaded again at different addresses, because we will see new PCs
covered all the time. But if modules are loaded once it should work
fine.

Every time a QEMU instance is restarted, the test modules might be loaded at different addresses (depending on the syscall order, etc.). PCs will be different? Or are you just assuming that modules will be loaded at the same addresses for all booted QEMU instances?

[Dmitry Vyukov]

We don't use modules.

[mike.s...@gmail.com]

You're saying "It should not matter if code is in module or not..." The question is asking about testing external modules.

[Dmitry Vyukov]

On Tue, Mar 26, 2019 at 9:37 AM <mike.s...@gmail.com> wrote:
>
> You're saying "It should not matter if code is in module or not..." The question is asking about testing external modules.

If modules are loaded at different addresses, this will make
syzkaller+KCOV misbehave. If they are loaded at the same address, this
should work.

[bruce...@gmail.com]

On Tuesday, March 26, 2019 at 5:04:12 AM UTC-4, Dmitry Vyukov wrote:

If modules are loaded at different addresses, this will make
syzkaller+KCOV misbehave. If they are loaded at the same address, this
should work.

While searching online, I found that disabling KASLR would solve the problem of randomized module address.

To do that, set CONFIG_RANDOMIZE_BASE=n in .config file.

 

[Joey Jiao]

Hi Dmitry,

But I didn't find the kcov instrument even vmlinux has.

linux-source-4.19$ objdump -d ./lib/test_kasan.ko|grep __sanitizer_cov

linux-source-4.19$ nm ./lib/test_kasan.ko|grep __sanitizer_cov

                 U __sanitizer_cov_trace_const_cmp4

                 U __sanitizer_cov_trace_const_cmp8

                 U __sanitizer_cov_trace_pc

Do you know reason why?

[Joey Jiao]

I made a test code on amd64 and arm64, on arm64, it can be decoded in objdump.

==arm64

0000000000000000 <main>:

   0:   a9bd7bfd        stp     x29, x30, [sp,#-48]!

   4:   910003fd        mov     x29, sp

   8:   f9000bf3        str     x19, [sp,#16]

   c:   94000000        bl      0 <__sanitizer_cov_trace_pc>

  10:   b9402fe0        ldr     w0, [sp,#44]

  14:   7100001f        cmp     w0, #0x0

  18:   540000c1        b.ne    30 <main+0x30>

==amd64

  if (i == 0)

   9:   e8 00 00 00 00          callq  e <main+0xe> // I believe this line is __sanitizer_cov_trace_pc

   e:   83 7d ec 00             cmpl   $0x0,-0x14(%rbp)

  12:   75 13                   jne    27 <main+0x27>

    printf("0\n");

It might be difference between objdump of arm64 and amd64, but don't know why amd64 cannot show the callq __sanitizer_cov_trace_pc.

[Dmitry Vyukov]

On Thu, Mar 4, 2021 at 3:21 AM Joey Jiao <joey....@gmail.com> wrote:
>
> Hi Dmitry,
> But I didn't find the kcov instrument even vmlinux has.
>
> linux-source-4.19$ objdump -d ./lib/test_kasan.ko|grep __sanitizer_cov
> linux-source-4.19$ nm ./lib/test_kasan.ko|grep __sanitizer_cov
> U __sanitizer_cov_trace_const_cmp4
> U __sanitizer_cov_trace_const_cmp8
> U __sanitizer_cov_trace_pc
>
> Do you know reason why?

No, I don't. It looks like CONFIG_KCOV is not enabled. if you provide
reproduction instructions, I can take a look at it.

> 在2020年2月27日星期四 UTC+8 上午6:22:38<bruce...@gmail.com> 写道：
>>
>> On Tuesday, March 26, 2019 at 5:04:12 AM UTC-4, Dmitry Vyukov wrote:
>>>
>>> If modules are loaded at different addresses, this will make
>>> syzkaller+KCOV misbehave. If they are loaded at the same address, this
>>> should work.
>>
>>
>> While searching online, I found that disabling KASLR would solve the problem of randomized module address.
>> To do that, set CONFIG_RANDOMIZE_BASE=n in .config file.
>>
>

> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/4f438c8b-39a6-441b-bc7e-15b71d055510n%40googlegroups.com.

[Dmitry Vyukov]

On Thu, Mar 4, 2021 at 3:55 AM Joey Jiao <joey....@gmail.com> wrote:
>
> I made a test code on amd64 and arm64, on arm64, it can be decoded in objdump.
>
> ==arm64
> 0000000000000000 <main>:
> 0: a9bd7bfd stp x29, x30, [sp,#-48]!
> 4: 910003fd mov x29, sp
> 8: f9000bf3 str x19, [sp,#16]
> c: 94000000 bl 0 <__sanitizer_cov_trace_pc>
> 10: b9402fe0 ldr w0, [sp,#44]
> 14: 7100001f cmp w0, #0x0
> 18: 540000c1 b.ne 30 <main+0x30>
>
>
> ==amd64
> if (i == 0)
> 9: e8 00 00 00 00 callq e <main+0xe> // I believe this line is __sanitizer_cov_trace_pc
> e: 83 7d ec 00 cmpl $0x0,-0x14(%rbp)
> 12: 75 13 jne 27 <main+0x27>
> printf("0\n");
>
> It might be difference between objdump of arm64 and amd64, but don't know why amd64 cannot show the callq __sanitizer_cov_trace_pc.

It looks like you are disassembling an object file rather than an ELF
binary, right? If so, it is expected.

> 在2021年3月4日星期四 UTC+8 上午10:21:06<Joey Jiao> 写道：
>>
>> Hi Dmitry,
>> But I didn't find the kcov instrument even vmlinux has.
>>
>> linux-source-4.19$ objdump -d ./lib/test_kasan.ko|grep __sanitizer_cov
>> linux-source-4.19$ nm ./lib/test_kasan.ko|grep __sanitizer_cov
>> U __sanitizer_cov_trace_const_cmp4
>> U __sanitizer_cov_trace_const_cmp8
>> U __sanitizer_cov_trace_pc
>>
>> Do you know reason why?
>>
>> 在2020年2月27日星期四 UTC+8 上午6:22:38<bruce...@gmail.com> 写道：
>>>
>>> On Tuesday, March 26, 2019 at 5:04:12 AM UTC-4, Dmitry Vyukov wrote:
>>>>
>>>> If modules are loaded at different addresses, this will make
>>>> syzkaller+KCOV misbehave. If they are loaded at the same address, this
>>>> should work.
>>>
>>>
>>> While searching online, I found that disabling KASLR would solve the problem of randomized module address.
>>> To do that, set CONFIG_RANDOMIZE_BASE=n in .config file.
>>>
>

> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/6e2edebb-698a-4002-9302-771df911431en%40googlegroups.com.

[joey....@gmail.com]

right, I had known the reason, relocation offset and addend can be used to get that info.