Some questions for syzbot

[Xu, Pengfei]

Hi Dmitry and Aleksandr,

 

Greeting!

 

I am a user of syzkaller, thank you for creating this great fuzzing tool, found many valuable bugs for the Linux kernel and other operating systems.

This is a great fuzzing tool!

 

For the issue link in “linux-xfs” community: https://lore.kernel.org/all/20230526205...@mit.edu/

 

I think there are 3 points that led me to find some bugs that syzbot didn't find, thanks for corrections or comments if any.

1. Kconfig difference like IOMMUFD.
2. Syzkaller is a KCOV-oriented random fuzzing test tool: different syzkaller instances may fill the syscall parameters with different random values and use different combinations of syscalls to find different problems with a certain probability.
3. Platform specific like Alder lake hybrid platforms for perf related issues.

 

 

If possible, can I get the following answers, without infringing on the confidentiality of your company:

1. Kconfig:  does syzbot enable IOMMUFD in general tests on Intel x86 platform?

May I suggest to syzbot some new kconfig settings for syzbot in future or should I figure out a way to cover these kconfig configurations, based on the interests of the respective companies?

If IOMMUFD is enabled as follow in kconfig for guest:

CONFIG_IOMMUFD=y

CONFIG_IOMMUFD_TEST=y

CONFIG_IOMMUFD_VFIO_CONTAINER=y

CONFIG_FAULT_INJECTION=y

 

And then below non-IOMMUFD VFIO_CONTAINER will be set to n automatically if above CONFIG_IOMMUFD_VFIO_CONTAINER is set to y.

CONFIG_CONFIG_VFIO_CONTAINER=n    // Maybe there are 2 kconfigs IOMMUFD and legacy VFIO settings we should cover.

 

2. A maintainer asked me privately how many machines syzbot uses to run syzkaller?

Why I can find some bugs that syzbot didn't find, they think that as long as there are enough platforms, it should be able to make up for such a gap.

Currently I used Intel x86 platforms like Alder lake-s, Raptor lake-p/s, Tiger lake and so on platforms, about 8 machines are running syzkaller fuzzing test on Intel x86 platforms.

 

3. Platform specific, may I ask which types of Intel x86 platforms syzbot are using for syzkaller fuzzing?

 

4. Ted mentioned that:

“

If we could import the fuzzing seed for this and similar reports into Google's syzkaller instance, it seems to

me that this would be a Good Thing.

”

And if my private syzkaller found some bugs that syzbot didn’t find in mainline kernel.

 

Could I import the fuzzing report seed to syzbot like Ted mentioned?

Is syzbot willing to accept the above way of releasing bugs from other private syzkaller to the Linux kernel community?

I also have the following concerns: whether it will affect the interests of their respective companies and the efficiency of reporting in this way.

 

 

Thanks!

BR

Pengfei

 

 

 

[Aleksandr Nogikh]

Hi Pengfei,

Thank you for your interest in syzkaller and syzbot!

Please find my comments below.

On Sun, Jun 4, 2023 at 5:07 AM Xu, Pengfei <pengf...@intel.com> wrote:
>
> Hi Dmitry and Aleksandr,
>
> Greeting!
>
> I am a user of syzkaller, thank you for creating this great fuzzing tool, found many valuable bugs for the Linux kernel and other operating systems.
>
> This is a great fuzzing tool!
>
> For the issue link in “linux-xfs” community: https://lore.kernel.org/all/20230526205...@mit.edu/
>
> I think there are 3 points that led me to find some bugs that syzbot didn't find, thanks for corrections or comments if any.
>
> Kconfig difference like IOMMUFD.
> Syzkaller is a KCOV-oriented random fuzzing test tool: different syzkaller instances may fill the syscall parameters with different random values and use different combinations of syscalls to find different problems with a certain probability.

Yes, kernel fuzzing is very probabilistic and we've also observed that
every time syzkaller is run, it finds slightly different sets of bugs.
So at least in some cases your instances might have just had more luck
than those of syzbot :)

> Platform specific like Alder lake hybrid platforms for perf related issues.
>
> If possible, can I get the following answers, without infringing on the confidentiality of your company:
>
> Kconfig: does syzbot enable IOMMUFD in general tests on Intel x86 platform?

The upstream Linux kernel configurations we fuzz are public (not all
files there are being used, but we only use configs from this folder):
https://github.com/google/syzkaller/tree/master/dashboard/config/linux

We tried to cover as many options as we could, but there are of course
still many good-to-test configs that we didn't notice/know of. If you
have any suggestions as to what else could be enabled, feel free to
share with us.

> May I suggest to syzbot some new kconfig settings for syzbot in future or should I figure out a way to cover these kconfig configurations, based on the interests of the respective companies?
>
> If IOMMUFD is enabled as follow in kconfig for guest:
>
> CONFIG_IOMMUFD=y
>
> CONFIG_IOMMUFD_TEST=y
>
> CONFIG_IOMMUFD_VFIO_CONTAINER=y
>
> CONFIG_FAULT_INJECTION=y

Out of these configuration options, we currently only don't set
CONFIG_IOMMUFD_VFIO_CONTAINER=y:

https://github.com/google/syzkaller/blob/058b3a5a6a945a55767811552eb7b9f4a20307f8/dashboard/config/linux/upstream-apparmor-kasan.config#L7138-L7140

I can enable it, that's not a problem at all.

> And then below non-IOMMUFD VFIO_CONTAINER will be set to n automatically if above CONFIG_IOMMUFD_VFIO_CONTAINER is set to y.
>
> CONFIG_CONFIG_VFIO_CONTAINER=n // Maybe there are 2 kconfigs IOMMUFD and legacy VFIO settings we should cover.

I think it should be fine since we don't set
CONFIG_CONFIG_VFIO_CONTAINER=y at the moment anyway.

But, apart from Kconfig, we should also somehow make iommu available
to the fuzzed kernel/VM, right? Otherwise fuzzing won't go very deep
into the code. Qemu seems to have a "-device intel-iommu" option. Are
you aware of any other alternatives?

> A maintainer asked me privately how many machines syzbot uses to run syzkaller?

(The figures below are only about upstream Linux kernel fuzzing)

We have a pretty diverse setup in terms of fuzzed kernel trees /
different sanitizers, but not so diverse in terms of CPU platforms.

These use e2-standard-2 VMs for fuzzing:
ci-upstream-kasan-gce -- 10 VMs
ci-upstream-kasan-gce-root -- 10 VMs
ci-upstream-kasan-gce-selinux-root -- 10 VMs
ci-upstream-kasan-gce-smack-root -- 10 VMs
ci-upstream-linux-next-kasan-gce-root -- 10 VMs
ci-upstream-bpf-kasan-gce -- 7 VMs
ci-upstream-bpf-next-kasan-gce -- 10 VMs
ci-upstream-net-this-kasan-gce -- 7 VMs
ci-upstream-net-kasan-gce -- 10 VMs
ci-upstream-kasan-gce-386 -- 5 VMs
ci-upstream-gce-leak -- 10 VMs
ci-upstream-kmsan-gce -- 10 VMs
ci-upstream-kmsan-gce-386 -- 5 VMs
ci-upstream-gce-arm64 -- 8 VMs
ci2-upstream-usb -- 10 VMs
ci2-upstream-kcsan-gce -- 5 VMs
ci2-upstream-fs -- 10 VMs

Around 150 in total?

Also these run on a Intel Ice Lake machine:
ci-qemu-upstream -- 5 qemu VMs
ci-qemu-upstream-386 -- 5 qemu VMs

And these on an AMD Rome machine:
ci-qemu2-arm32 -- 5 emulated qemu VMs
ci-qemu2-arm64 -- 5 emulated qemu VMs
ci-qemu2-arm64-compat -- 5 emulated qemu VMs
ci-qemu2-arm64-mte -- 5 emulated qemu VMs
ci-qemu2-riscv64 -- 10 emulated qemu VMs

All syz-manager instances above are united via syz-hub [1], so they
exchange programs/reproducers and, in some sense, act like one big
syzkaller instance.

[1] https://github.com/google/syzkaller/blob/master/docs/hub.md

> Why I can find some bugs that syzbot didn't find, they think that as long as there are enough platforms, it should be able to make up for such a gap.

If the issues are platform-specific, then that should definitely help.

> Currently I used Intel x86 platforms like Alder lake-s, Raptor lake-p/s, Tiger lake and so on platforms, about 8 machines are running syzkaller fuzzing test on Intel x86 platforms.

By the way, are you running syzkaller on real devices (via e.g.
isolated mode) or do you fuzz kernels in virtual machines that are
running on those platforms?

> Platform specific, may I ask which types of Intel x86 platforms syzbot are using for syzkaller fuzzing?

As I described above, there are some qemu instances on Intel Ice Lake
and the rest are just transient E2 VMs:
https://cloud.google.com/compute/docs/general-purpose-machines#e2_machine_types
(sometimes Intel, sometimes AMD?)

I think we may well consider making our setup more diverse in terms of
platforms, but it'd be great to first understand what available CPU
platforms can enable what Linux kernel feature fuzzing.

> Ted mentioned that:
>
> “
> If we could import the fuzzing seed for this and similar reports into Google's syzkaller instance, it seems to
> me that this would be a Good Thing.
> ”
>
> And if my private syzkaller found some bugs that syzbot didn’t find in mainline kernel.
>
> Could I import the fuzzing report seed to syzbot like Ted mentioned?
>
> Is syzbot willing to accept the above way of releasing bugs from other private syzkaller to the Linux kernel community?
>
> I also have the following concerns: whether it will affect the interests of their respective companies and the efficiency of reporting in this way.

(I'll leave these questions unanswered in this email, we'll send a
separate reply re. this)

--
Best Regards,
Aleksandr

>
>
> Thanks!
>
> BR
>
> Pengfei
>

[Pengfei Xu]

Hi Aleksandr,

On 2023-06-08 at 14:47:51 +0200, Aleksandr Nogikh wrote:
> Hi Pengfei,
>
> Thank you for your interest in syzkaller and syzbot!
>
> Please find my comments below.

Thanks a lot for your comments and my reply is below.

>
> On Sun, Jun 4, 2023 at 5:07 AM Xu, Pengfei <pengf...@intel.com> wrote:
> >
> > Hi Dmitry and Aleksandr,
> >
> > Greeting!
> >
> > I am a user of syzkaller, thank you for creating this great fuzzing tool, found many valuable bugs for the Linux kernel and other operating systems.
> >
> > This is a great fuzzing tool!
> >
> > For the issue link in “linux-xfs” community: https://lore.kernel.org/all/20230526205...@mit.edu/
> >
> > I think there are 3 points that led me to find some bugs that syzbot didn't find, thanks for corrections or comments if any.
> >
> > Kconfig difference like IOMMUFD.
> > Syzkaller is a KCOV-oriented random fuzzing test tool: different syzkaller instances may fill the syscall parameters with different random values and use different combinations of syscalls to find different problems with a certain probability.
>
> Yes, kernel fuzzing is very probabilistic and we've also observed that
> every time syzkaller is run, it finds slightly different sets of bugs.
> So at least in some cases your instances might have just had more luck
> than those of syzbot :)

Thank you for confirming the above point of view!
Good to know that some other private syzkaller instances from different
companies are still necessary and useful. :)

>
> > Platform specific like Alder lake hybrid platforms for perf related issues.
> >
> > If possible, can I get the following answers, without infringing on the confidentiality of your company:
> >
> > Kconfig: does syzbot enable IOMMUFD in general tests on Intel x86 platform?
>
> The upstream Linux kernel configurations we fuzz are public (not all
> files there are being used, but we only use configs from this folder):
> https://github.com/google/syzkaller/tree/master/dashboard/config/linux
>
> We tried to cover as many options as we could, but there are of course
> still many good-to-test configs that we didn't notice/know of. If you
> have any suggestions as to what else could be enabled, feel free to
> share with us.

Thanks for your comments, yes I will.

>
> > May I suggest to syzbot some new kconfig settings for syzbot in future or should I figure out a way to cover these kconfig configurations, based on the interests of the respective companies?
> >
> > If IOMMUFD is enabled as follow in kconfig for guest:
> >
> > CONFIG_IOMMUFD=y
> >
> > CONFIG_IOMMUFD_TEST=y
> >
> > CONFIG_IOMMUFD_VFIO_CONTAINER=y
> >
> > CONFIG_FAULT_INJECTION=y
>
> Out of these configuration options, we currently only don't set
> CONFIG_IOMMUFD_VFIO_CONTAINER=y:
>
> https://github.com/google/syzkaller/blob/058b3a5a6a945a55767811552eb7b9f4a20307f8/dashboard/config/linux/upstream-apparmor-kasan.config#L7138-L7140
>
> I can enable it, that's not a problem at all.
>
> > And then below non-IOMMUFD VFIO_CONTAINER will be set to n automatically if above CONFIG_IOMMUFD_VFIO_CONTAINER is set to y.
> >
> > CONFIG_CONFIG_VFIO_CONTAINER=n // Maybe there are 2 kconfigs IOMMUFD and legacy VFIO settings we should cover.
>
> I think it should be fine since we don't set
> CONFIG_CONFIG_VFIO_CONTAINER=y at the moment anyway.
>
> But, apart from Kconfig, we should also somehow make iommu available
> to the fuzzed kernel/VM, right? Otherwise fuzzing won't go very deep
> into the code. Qemu seems to have a "-device intel-iommu" option. Are
> you aware of any other alternatives?

"CONFIG_CONFIG_VFIO_CONTAINER=y" kconfig seems to have no effect on my
syzkaller fuzzing, I think the key kconfig is "CONFIG_IOMMUFD=y" and
"CONFIG_IOMMUFD_TEST=y" for IOMMUFD fuzzing test.

Yes, using the "-device intel-iommu" option can cover more intel IOMMU(Intel
VT-d) related kernel code, and provide better performance and device security,
but it's better to ensure that the host hardware support Intel VT-d option.

Thanks a lot for your info sharing! Yes, syzbot has many more VM instances
than I'm using for syzkaller fuzzing.

>
> All syz-manager instances above are united via syz-hub [1], so they
> exchange programs/reproducers and, in some sense, act like one big
> syzkaller instance.
>
> [1] https://github.com/google/syzkaller/blob/master/docs/hub.md

Thanks for your suggestion, it's a good idea to share corpus in different
local syzkaller instances! I will have a try for syzhub function as one big
syzkaller instance.

>
> > Why I can find some bugs that syzbot didn't find, they think that as long as there are enough platforms, it should be able to make up for such a gap.
>
> If the issues are platform-specific, then that should definitely help.

Thanks for confirmation.

>
> > Currently I used Intel x86 platforms like Alder lake-s, Raptor lake-p/s, Tiger lake and so on platforms, about 8 machines are running syzkaller fuzzing test on Intel x86 platforms.
>
> By the way, are you running syzkaller on real devices (via e.g.
> isolated mode) or do you fuzz kernels in virtual machines that are
> running on those platforms?

I'm running syzkaller syz-manager on bare metal client platforms and then
syzkaller boot 4 virtual machines for fuzzing in each platform. I'm using
probably the most common and simple way.

And I have a confusion: will the following two different ways affect the
results of syzkaller fuzzing between below docker and non-docker way:
1. Use docker to deploy the syzkaller environment on different platforms and
run syzkaller "VM fuzzing in docker".
2. Deploy syzkaller directly on bare metal platforms and running syzkaller
fuzz tests in VMs(Current way I'm using).

From the information of your previous email, it seems that syzbot is using
e2_machine VMs from google cloud and docker way to deploy and run each fuzzing
in VMs directly, becaues from E2 machine introduction, E2 could not support
nested VMs.

Looking forward to your comments if anything is wrong or inaccurate. :)

>
> > Platform specific, may I ask which types of Intel x86 platforms syzbot are using for syzkaller fuzzing?
>
> As I described above, there are some qemu instances on Intel Ice Lake
> and the rest are just transient E2 VMs:
> https://cloud.google.com/compute/docs/general-purpose-machines#e2_machine_types
> (sometimes Intel, sometimes AMD?)
>
> I think we may well consider making our setup more diverse in terms of
> platforms, but it'd be great to first understand what available CPU
> platforms can enable what Linux kernel feature fuzzing.

Yes, and I think there are very few platform-specific problems.
The difference in VM functionality between different platforms is actually
very small for syzkaller fuzzing.

From the results of syzkaller fuzzing I collected: I only found that PMU
(Performance Monitoring Unit) and perf related bugs between non-hybrid and
hybrid platforms, and most of bugs are found from common platforms.

>
> > Ted mentioned that:
> >
> > “
> > If we could import the fuzzing seed for this and similar reports into Google's syzkaller instance, it seems to
> > me that this would be a Good Thing.
> > ”
> >
> > And if my private syzkaller found some bugs that syzbot didn’t find in mainline kernel.
> >
> > Could I import the fuzzing report seed to syzbot like Ted mentioned?
> >
> > Is syzbot willing to accept the above way of releasing bugs from other private syzkaller to the Linux kernel community?
> >
> > I also have the following concerns: whether it will affect the interests of their respective companies and the efficiency of reporting in this way.
>
> (I'll leave these questions unanswered in this email, we'll send a
> separate reply re. this)

Got it, thanks.

Best Regards,

-Pengfei

[Dmitry Vyukov]

How can we check if host support VT-d?

Currently we use the following qemu flags:

ci-qemu-upstream
"qemu_args": "-enable-kvm -machine
q35,nvdimm,accel=kvm,kernel-irqchip=split -cpu max,migratable=off -smp
4,maxcpus=8,cores=2,threads=2,sockets=2 -device
ioh3420,id=pcie.1,chassis=1 -device
intel-iommu,intremap=on,device-iotlb=on -vga virtio -usb -usbdevice
tablet -m 4G,slots=4,maxmem=16G -object
memory-backend-file,id=pmem1,share=off,mem-path=/dev/zero,size=64M
-device nvdimm,id=nvdimm1,memdev=pmem1 -net nic,model=e1000e -fdb
{{TEMPLATE}}/fd -cdrom {{TEMPLATE}}/cdrom -virtfs
local,path={{TEMPLATE}}/virtfs,mount_tag=syz,security_model=mapped-xattr,id=syz"

ci-qemu-upstream-386
"qemu_args": "-enable-kvm -machine
q35,nvdimm,accel=kvm,kernel-irqchip=split -cpu max,migratable=off -smp
4,maxcpus=8,cores=2,threads=2,sockets=2 -device
ioh3420,id=pcie.1,chassis=1 -device
intel-iommu,intremap=on,device-iotlb=on -vga std -usb -usbdevice
tablet -m 2G,slots=4,maxmem=16G -object
memory-backend-file,id=pmem1,share=off,mem-path=/dev/zero,size=64M
-device nvdimm,id=nvdimm1,memdev=pmem1 -net nic,model=e1000e -fdb
{{TEMPLATE}}/fd -cdrom {{TEMPLATE}}/cdrom -virtfs
local,path={{TEMPLATE}}/virtfs,mount_tag=syz,security_model=mapped-xattr,id=syz"


You can also check coverage reports for these instances here:
https://syzkaller.appspot.com/upstream


If you can suggest some additional options/configs to increase
coverage, that's welcome.

> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/ZIQbVebzYPnrtge3%40xpf.sh.intel.com.

[Pengfei Xu]

Hi Dmitry,

Thank you and Aleksandr for your previous help.
My feedback is as below.

In my opinion we could check whether these strings "DMAR: Intel(R)
Virtualization Technology for Directed I/O" are included in dmesg to confirm
whether the host supports Intel VT-d technology, and Intel VT-d technology can
be enabled/disabled in bios also.

In the past ten years or even earlier, Intel platforms support Intel VT-d,
such as ICL server Aleksandr mentioned could support Intel VT-d also.

>
> Currently we use the following qemu flags:
>
> ci-qemu-upstream
> "qemu_args": "-enable-kvm -machine
> q35,nvdimm,accel=kvm,kernel-irqchip=split -cpu max,migratable=off -smp
> 4,maxcpus=8,cores=2,threads=2,sockets=2 -device
> ioh3420,id=pcie.1,chassis=1 -device
> intel-iommu,intremap=on,device-iotlb=on -vga virtio -usb -usbdevice
> tablet -m 4G,slots=4,maxmem=16G -object
> memory-backend-file,id=pmem1,share=off,mem-path=/dev/zero,size=64M
> -device nvdimm,id=nvdimm1,memdev=pmem1 -net nic,model=e1000e -fdb
> {{TEMPLATE}}/fd -cdrom {{TEMPLATE}}/cdrom -virtfs
> local,path={{TEMPLATE}}/virtfs,mount_tag=syz,security_model=mapped-xattr,id=syz"
>
> ci-qemu-upstream-386
> "qemu_args": "-enable-kvm -machine
> q35,nvdimm,accel=kvm,kernel-irqchip=split -cpu max,migratable=off -smp
> 4,maxcpus=8,cores=2,threads=2,sockets=2 -device
> ioh3420,id=pcie.1,chassis=1 -device
> intel-iommu,intremap=on,device-iotlb=on -vga std -usb -usbdevice
> tablet -m 2G,slots=4,maxmem=16G -object
> memory-backend-file,id=pmem1,share=off,mem-path=/dev/zero,size=64M
> -device nvdimm,id=nvdimm1,memdev=pmem1 -net nic,model=e1000e -fdb
> {{TEMPLATE}}/fd -cdrom {{TEMPLATE}}/cdrom -virtfs
> local,path={{TEMPLATE}}/virtfs,mount_tag=syz,security_model=mapped-xattr,id=syz"
>

Actually I used the simple one in syzkaller fuzzing as below:
"qemu_args": "-cpu host -net nic,model=e1000 -enable-kvm",
I didn't pass through any PCI into VM and I didn't use "-device intel-iommu"
in my syzkaller.

I think the discrepancy in finding issues like IOMMUFD is from syzkaller
runtime and luck.

There are very few problems related to the specific platform(here only refers
to the x86 platform). I found that I only found a few hybrid-specific perf and
pmu related problems that are specific to the hybrid platform, and they have
been fixed in intel internal.
Most of the problems found by syskaller are common and platform-independent
problems.

>
> You can also check coverage reports for these instances here:
> https://syzkaller.appspot.com/upstream
>
>
> If you can suggest some additional options/configs to increase
> coverage, that's welcome.

Yes, I will, except IOMMUFD, I think there is no special kconfig in my private
syzkaller.
The kconfig for syzkaller I used is in link:
https://raw.githubusercontent.com/xupengfe/kconfig_diff/main/config-5.18rc4i_kvm

Thanks!
BR.

[Dmitry Vyukov]

+syzkaller mailing list

On Tue, 14 Oct 2025 at 04:46, Wei Fang <void...@gmail.com> wrote:

> Are these parameters manually set, or are there any automated tools?
> For example, qemu supports some sound devices (AC97, adlib, cs4231a, etc.). Will adding these devices to qemu_args help improve coverage?
> Also, are there any dependencies or conflicts between these devices?