Is it feasible and effective to make syzkaller some awareness of state in kernel ?

[Kaipeng Zeng]

Hi,

   I have written a demonstration which run a syz-execprog under a ebpf monitoring. ebpf can collect the kernel socket state and feedback to the execprog.

(https://github.com/hardenedlinux/harbian-qa/tree/master/syzkaller/kstat_demo)

  Actually, progs hit no new coverage but with new kernel data state can add to corpus. That may make syzkaller kernel-data-state-sensitive? Is it feasible and effective that collect kernel data state by ebpf and feedback to fuzzer. Just as the executor collect coverage and feedback to fuzzer. Will syzkaller support the similar feature?

[Dmitry Vyukov]

Hi Kaipeng,

Thanks for sharing this interesting work.

What exactly state do you propose to capture? Do you have any
indication that it is useful (allows to find more bugs)?
Is it possible to make it work with RLIMIT_STACK/CLONE_NEWPID?
What is the performance impact of this tracing?

The concerns that I have are:
- more complexity
- more complex build/setup/deploy
- performance impact (executions/sec)
- only linux-specific
- dependence on particular kernel version and undocumented internals
(sock does not have to have flags, and flags do not necessary have
these values, e.g. is it true for 3.18?)
- maintenance burden as the result
- unclear testing story (how will we ensure that all of this
continues to work?)

So the benefits must be large enough to outweigh all of this.

[Kaipeng Zeng]

Hi Dmitry,

Thanks for your feedback!

> What exactly state do you propose to capture? Do you have any
> indication that it is useful (allows to find more bugs)?

This simple demonstration is just a example. 

The exactly state should be determined by user but not syzkaller. Syzkaller only support a similar feature like this. 

For example, syzkaller remain a 64-bit memory for any kinds of state. And user put state they want into 64-bit memory( by writing their own ebpf)

and feedback to fuzzer. Fuzzer handle these state like coverage siganl, if there is a new one, add the prog to the corpus.

 

> Is it possible to make it work with RLIMIT_STACK/CLONE_NEWPID?

ebpf can work with CLONE_NEWPID. RLIMIT_STACK seems break ebpf( bcc) loading. I don't know the reason.

> What is the performance impact of this tracing?

>  - performance impact (executions/sec)

The performance impact mainly contain: ebpf load( several second), synchronization (need more test), and state handling( not so much).

ebpf load only one time at every time the vm start.

> The concerns that I have are:
>  - more complexity
>  - more complex build/setup/deploy

If only implement as the mentioned example, it can be optional, no more complex build/setup/deploy.

>  - only linux-specific

if we choose ebpf as kernel state monitor, the answer is "yes".

>  - dependence on particular kernel version and undocumented internals
> (sock does not have to have flags, and flags do not necessary have
> these values, e.g. is it true for 3.18?)

>  - maintenance burden as the result

Yes, all of this work dependence on particular kernel version and undocumented internals.

But, if syzkaller Implement that as a optional feature. Syzkaller can run as usual if there is no state feedback( the same as no new state).

And user can use their ebpf as their will. Of course syzkaller can maintian a specific version as a example.

> - unclear testing story (how will we ensure that all of this

continues to work?)

Need more testing.

[Dmitry Vyukov]

On Wed, Jan 9, 2019 at 5:28 PM Kaipeng Zeng <kaip...@gmail.com> wrote:
>
> Hi Dmitry,
> Thanks for your feedback!
>
> > What exactly state do you propose to capture? Do you have any
> > indication that it is useful (allows to find more bugs)?
> This simple demonstration is just a example.
> The exactly state should be determined by user but not syzkaller. Syzkaller only support a similar feature like this.
> For example, syzkaller remain a 64-bit memory for any kinds of state. And user put state they want into 64-bit memory( by writing their own ebpf)
> and feedback to fuzzer. Fuzzer handle these state like coverage siganl, if there is a new one, add the prog to the corpus.

It's hard to judge and evaluate without seeing actual uses.
Maybe there are no good uses for this at all? I don't know.
Maybe it does not provide any benefit in terms of found bugs? I don't know.
Maybe there are other, better ways to do this? E.g. adding explicit,
annotations to kernel code maintained alongside with kernel code.
And in the end the actual uses must be part of this functionality.
Just like syzkaller provides not just abstract engine to describe
kernel interfaces, but also actual descriptions for thousands of
syscalls. Without this I afraid only you will use it and nobody else
will benefit from this.
The principles that we try to follow:
- work out of-the-box
- easy to use

> > Is it possible to make it work with RLIMIT_STACK/CLONE_NEWPID?
> ebpf can work with CLONE_NEWPID. RLIMIT_STACK seems break ebpf( bcc) loading. I don't know the reason.
>
> > What is the performance impact of this tracing?
> > - performance impact (executions/sec)
> The performance impact mainly contain: ebpf load( several second), synchronization (need more test), and state handling( not so much).
> ebpf load only one time at every time the vm start.

What's the cumulative effect in number of executions per second?

> > The concerns that I have are:
> > - more complexity
> > - more complex build/setup/deploy
> If only implement as the mentioned example, it can be optional, no more complex build/setup/deploy.

Yes, but then it is also not useful for anybody.

[Kaipeng Zeng]

Thanks!

On Thu, Jan 10, 2019 at 7:48 PM Dmitry Vyukov <dvy...@google.com> wrote:
>
> On Wed, Jan 9, 2019 at 5:28 PM Kaipeng Zeng <kaip...@gmail.com> wrote:

> It's hard to judge and evaluate without seeing actual uses.
> Maybe there are no good uses for this at all? I don't know.
> Maybe it does not provide any benefit in terms of found bugs? I don't know.
> Maybe there are other, better ways to do this? E.g. adding explicit,
> annotations to kernel code maintained alongside with kernel code.
> And in the end the actual uses must be part of this functionality.
> Just like syzkaller provides not just abstract engine to describe
> kernel interfaces, but also actual descriptions for thousands of
> syscalls. Without this I afraid only you will use it and nobody else
> will benefit from this.
> The principles that we try to follow:
>  - work out of-the-box
>  - easy to use

Yes, it need more test to evaluate the benefit.

It will be more effective if kernel support interfaces like that. But it seems that there is still no interface can be use in it. Right?

Become more complex of course.

> What's the cumulative effect in number of executions per second?

I only tested some single prog by running exeprog.

[Dmitry Vyukov]

I think KCOV is the closest interface. It can be extended to inject
some custom "coverage" into per-task KCOV buffer with some kind of
annotations, e.g. kernel code contains:

void sock_connect(...)
{
KCOV_INJECT_COVER(KCOV_SOCK_CONNECT_FLAGS, sock->flags);
...
}

and then user-space reads out this pseudo-coverage.

[Kaipeng Zeng]

Sorry for delay.

Thanks for your feedback!

So, i have to rebuild kernel, right?  

One of the reason I choose ebpf is that customizing is more flexible, without any modify of kernel.

I have never try it in this way, thanks for your advise.

[Dmitry Vyukov]

Well, if one is testing kernel and fixing bugs in kernel, presumably
they have source and can rebuild kernel. Special kernel build is very
desirable for syzkaller (or just any testing) anyway (e.g. KCOV,
KASAN, LOCKDEP, etc).
All else being equal I would of course choose an option that does not
require kernel rebuild. But if we are choosing between "lots of
complex changes that are hard to maintain but without kernel rebuild"
vs "fewer, simpler changes that are much easier to maintain but with
kernel rebuild", I would probably choose the second option.

[Kaipeng Zeng]

Hi,

I already finish feeding data colleceted by ebpf to syzkaller. 

For reducing performence impact, I rewrite a new ebpf which is more simple than the old one.

But the new ebpf can't track the socket one on one. 

Without maintaining a large socket list, data collecting can be quicker and syzkaller doesn't need to wait for its data.

So, less performence impact.

It can work well with RLIMIT_STACK. May the reason of old ebpf broken is too much data.

This is the patch for syzkaller:

https://github.com/hardenedlinux/harbian-qa/blob/master/syzkaller/kstat_demo/patch_executor_pipe.patch

The performence impact include: ebpf load and read the rawSignal from pipe.

The new ebpf is very simple:

https://github.com/hardenedlinux/harbian-qa/blob/master/syzkaller/kstat_demo/pipe_monitor.go

And ebpf text can rewrite as your will:

https://github.com/hardenedlinux/harbian-qa/blob/master/syzkaller/kstat_demo/ebpf/ebpftext.go

And the usage is the same as original syzkaller.

[Kaipeng Zeng]

Yeah, kernel rebuild is acceptable in these general case. 

ebpf will make it complex, not easy to use, but it's flexible in defferent special case.

As a supplement, Hope it can help users of syzkaller. :)

[Dmitry Vyukov]

Do you have any data on effectiveness of this technique?

[Kaipeng Zeng]

No, I don't. I just finished it​ not long ago. Any progress will share to community.

[Kaipeng Zeng]

Hi,

As a update, I'd finished some case of this technique.

https://github.com/hardenedlinux/harbian-qa/blob/master/syzkaller/kstat_demo/test.md

There are also other strategies to use runtime state feedback. Need more test.