I'm new to syzkaller. Where do I start?

[Xing Yi Han]

Hello syzkaller devs! 

I'm a university student in Singapore and I'm currently looking into syzkaller for fuzzing for concurrency bugs in the Linux kernel for a school project.

For this, I wrote an eBPF program that I would like to insert into the kernel before the start of any fuzzing campaign. My current workflow looks like this – start the QEMU VM, run my eBPF program via the command line, run a simple multithreaded program. I would love to integrate this with syzkaller and I've looked into `executor/executor.cc` and the `syz-fuzzer` directory, but can't really pinpoint where I should insert code to run my eBPF program before fuzzing commences.

Also, I have learnt that syzkaller tries fuzzing for concurrency bugs by creating multiple threads during execution. May I know if these threads are created right at the start of every execution? Or do they spawn randomly during the execution? I would like to set additional attributes for the thread such as its priority, scheduling policy and CPU affinity before the thread gets created via syscalls `sched_setscheduler()` and `sched_setaffinity()`. I assume this could be added to the start of the syzkaller-generated program, but I'm not too sure if this would work. It would be good to know where in the code I can look at to implement this as well!

I've watched the talks on syzkaller so I'm familiar with the overall architecture and I'm excited to get started. As syzkaller is really quite a big project, I would greatly appreciate it if you could help point me in the right direction! Thank you so much for your time and I hope you have an awesome week ahead :)

Cheers,

Xing Yi

[Aleksandr Nogikh]

Hi Xing Yi,

On Tue, Nov 7, 2023 at 7:25 AM Xing Yi Han <hxing...@gmail.com> wrote:
>
> Hello syzkaller devs!
>
> I'm a university student in Singapore and I'm currently looking into syzkaller for fuzzing for concurrency bugs in the Linux kernel for a school project.
>
> For this, I wrote an eBPF program that I would like to insert into the kernel before the start of any fuzzing campaign. My current workflow looks like this – start the QEMU VM, run my eBPF program via the command line, run a simple multithreaded program. I would love to integrate this with syzkaller and I've looked into `executor/executor.cc` and the `syz-fuzzer` directory, but can't really pinpoint where I should insert code to run my eBPF program before fuzzing commences.

There's unfortunately no single answer. It depends on what privileges you need to run the eBPF program and how often it should be executed.
1) If after the VM is booted (once), then look for the "setup_features" function (and the related functionality).
2) If once for every syz-executor process
  a) After all sandboxings: "setup_loop()"
  b) As part of specific sandboxing stages: look at do_sandbox_* calls.
3) If before every executed program, look at "setup_test()"

>
> Also, I have learnt that syzkaller tries fuzzing for concurrency bugs by creating multiple threads during execution. May I know if these threads are created right at the start of every execution? Or do they spawn randomly during the execution?

They are created lazily. Syz-executor tries to find a free thread for every program call. If all existing ones are busy, it creates one more:
https://github.com/google/syzkaller/blob/4862372a57ee80af8186cd80f6c9c8f741a45e40/executor/executor.cc#L967

> I would like to set additional attributes for the thread such as its priority, scheduling policy and CPU affinity before the thread gets created via syscalls `sched_setscheduler()` and `sched_setaffinity()`.

Actual thread creation occurs here:
https://github.com/google/syzkaller/blob/4862372a57ee80af8186cd80f6c9c8f741a45e40/executor/common.h#L354

> I assume this could be added to the start of the syzkaller-generated program, but I'm not too sure if this would work. It would be good to know where in the code I can look at to implement this as well!
>
> I've watched the talks on syzkaller so I'm familiar with the overall architecture and I'm excited to get started. As syzkaller is really quite a big project, I would greatly appreciate it if you could help point me in the right direction! Thank you so much for your time and I hope you have an awesome week ahead :)

If you want to look into concurrency bugs, you might also want to look into how syzkaller tries to trigger them now. A few pointers:
https://github.com/google/syzkaller/blob/master/docs/program_syntax.md#async
https://github.com/google/syzkaller/blob/master/prog/collide.go
https://github.com/google/syzkaller/blob/4862372a57ee80af8186cd80f6c9c8f741a45e40/syz-fuzzer/proc.go#L296
https://github.com/google/syzkaller/blob/4862372a57ee80af8186cd80f6c9c8f741a45e40/executor/executor.cc#L890

It's all extensible and it should not be too difficult to add extra call properties if needed (to control from syz-fuzzer what syz-executor is doing), but you of course need to understand the corresponding code first.

Good luck!

-- 
Aleksandr
>
> Cheers,
> Xing Yi
>

[Xing Yi Han]

Hey Aleksandr, 

Thank you so much for your response! It really helped me out a lot and I am now able to achieve the goals set out in the previous email :)

However, running syzkaller with `-debug` enabled, I realised that despite `kMaxThreads` being set to 32 in `executor.cc`, only one thread is actually created to handle the sequence of syscalls (see image below). One thread with id=0 is created and it runs all the syscalls. From the exec opts, threaded=1 which should mean that more threads should be created to handle the syscalls? How do I get more threads to be created to run the syscalls? 

Here is the config that I run syzkaller with:

{

"target": "linux/amd64",

"http": "127.0.0.1:56741",

"workdir": "workdir",

"kernel_obj": "externals/sched_ext",

"image": "image/bookworm.img",

"sshkey": "image/bookworm.id_rsa",

"syzkaller": "externals/syzkaller",

"disable_syscalls": ["sched_setattr"],

"suppressions":[ "pids_release"],

"procs": 4,

"type": "qemu",

"vm": {

"count": 4,

"kernel": "externals/sched_ext/arch/x86/boot/bzImage",

"cpu": 2,

"mem": 2048

}

}

Thank you for reading this and for all the help provided thus far! Happy 2024 :)

Cheers,

Xing Yi

[Dmitry Vyukov]

On Thu, 4 Jan 2024 at 08:54, Xing Yi Han <hxing...@gmail.com> wrote:

Hey Aleksandr, 

Thank you so much for your response! It really helped me out a lot and I am now able to achieve the goals set out in the previous email :)

However, running syzkaller with `-debug` enabled, I realised that despite `kMaxThreads` being set to 32 in `executor.cc`, only one thread is actually created to handle the sequence of syscalls (see image below). One thread with id=0 is created and it runs all the syscalls. From the exec opts, threaded=1 which should mean that more threads should be created to handle the syscalls? How do I get more threads to be created to run the syscalls? 

Hi Xing,

Executor will use more threads only if previous syscalls blocks.

If all syscalls don't block and return, then the executor will just reuse the single thread.

If you add a blocking sleep syscall to the program, then the next syscall after it should be started on another thread.

 

--
You received this message because you are subscribed to the Google Groups "syzkaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/3c73f09e-f29d-4d18-a70a-6ab1fcc84cean%40googlegroups.com.

[Xing Yi Han]

Hey Vyukov, 

Thanks for the reply! That makes a lot of sense, since syzkaller does generate sequences of syscalls that may be dependent on the one happening before... However, adding a sleep syscall after every syscall does seem to be quite a bit of unnecessary overhead. 

Just to check with you if my idea is feasible: Since I am fuzzing for concurrency bugs, would it be possible to modify syz-fuzzer to generate 2 separate sequences of syscalls, and then modify executor to execute those on 2 separate threads? Would this require a lot of effort or can this be done easily?

Please feel free to provide any feedback and/or suggestions, I am all ears! Very grateful for all the help received. Thank you so much to the syzkaller team :)

Cheers,

Xing Yi

[Dmitry Vyukov]

On Mon, 8 Jan 2024 at 19:31, Xing Yi Han <hxing...@gmail.com> wrote:
>
> Hey Vyukov,
>
> Thanks for the reply! That makes a lot of sense, since syzkaller does generate sequences of syscalls that may be dependent on the one happening before... However, adding a sleep syscall after every syscall does seem to be quite a bit of unnecessary overhead.
>
> Just to check with you if my idea is feasible: Since I am fuzzing for concurrency bugs, would it be possible to modify syz-fuzzer to generate 2 separate sequences of syscalls, and then modify executor to execute those on 2 separate threads? Would this require a lot of effort or can this be done easily?
>
> Please feel free to provide any feedback and/or suggestions, I am all ears! Very grateful for all the help received. Thank you so much to the syzkaller team :)

Hi Xing,

Thanks!

syzkaller can execute syscalls concurrently to trigger races:
https://github.com/google/syzkaller/blob/4c0fd4bb60ad179a6cf6be0edf416b2fca287b40/executor/executor.cc#L895-L898

What calls are executed in parallel is controlled by the program
generation/mutation logic.

I think this is mostly functionally equivalent to what you described.

> Cheers,
> Xing Yi
>
> On Thursday, January 4, 2024 at 6:01:26 PM UTC+8 dvy...@google.com wrote:
>>
>> On Thu, 4 Jan 2024 at 08:54, Xing Yi Han <hxing...@gmail.com> wrote:
>>>
>>> Hey Aleksandr,
>>>
>>> Thank you so much for your response! It really helped me out a lot and I am now able to achieve the goals set out in the previous email :)
>>>
>>> However, running syzkaller with `-debug` enabled, I realised that despite `kMaxThreads` being set to 32 in `executor.cc`, only one thread is actually created to handle the sequence of syscalls (see image below). One thread with id=0 is created and it runs all the syscalls. From the exec opts, threaded=1 which should mean that more threads should be created to handle the syscalls? How do I get more threads to be created to run the syscalls?
>>
>>
>> Hi Xing,
>>
>> Executor will use more threads only if previous syscalls blocks.
>> If all syscalls don't block and return, then the executor will just reuse the single thread.
>> If you add a blocking sleep syscall to the program, then the next syscall after it should be started on another thread.
>>
>>>
>>>
>>>
>>>

> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/847d84b5-8f58-42d9-a5ef-12408a343a89n%40googlegroups.com.

[Xing Yi Han]

Hey Vyukov, 

Thanks so much for your help! The async thing works well for my use case :) I've been able to add my modifications and it runs well. 

I've been trying to make the syscall sequences much longer, as that would be more likely to incur data races. However, I've noticed that even though prog.recommendedCalls = 30, most syscall progams end up only having 1 or 2 syscalls, with the occasional 12+ number of syscalls. 

I'm currently looking at this: https://github.com/google/syzkaller/blob/2a7bcc7f9f751f2ec13586584e8e25b8f6817e8c/syz-fuzzer/proc.go#L69-L99. The Generate() function doesn't seem to reduce the number of syscalls, but I've also realised that it is rarely ever called. It seems to me that most of the time, we are handling workqueue items and not generating/mutating the programs. I was wondering if you could point me in the right direction to understand what is shortening the length of the syscall sequence despite the (much higher) number of recommended calls. 

Thank you so much for your attention and help provided!

Cheers,

Xing Yi

[Dmitry Vyukov]

On Wed, 17 Jan 2024 at 02:56, Xing Yi Han <hxing...@gmail.com> wrote:
>
> Hey Vyukov,
>

> Thanks so much for your help! The async thing works well for my use case :) I've been able to add my modifications and it runs well.
>
> I've been trying to make the syscall sequences much longer, as that would be more likely to incur data races. However, I've noticed that even though prog.recommendedCalls = 30, most syscall progams end up only having 1 or 2 syscalls, with the occasional 12+ number of syscalls.
>
> I'm currently looking at this: https://github.com/google/syzkaller/blob/2a7bcc7f9f751f2ec13586584e8e25b8f6817e8c/syz-fuzzer/proc.go#L69-L99. The Generate() function doesn't seem to reduce the number of syscalls, but I've also realised that it is rarely ever called. It seems to me that most of the time, we are handling workqueue items and not generating/mutating the programs. I was wondering if you could point me in the right direction to understand what is shortening the length of the syscall sequence despite the (much higher) number of recommended calls.

That's probably Minimize function in the prog package.

> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/0f13713f-f527-40b3-b1c3-0336a79f62e8n%40googlegroups.com.