Performance Testing

[Adamson, Ryan]

All,

 

I had a question related to testing the efficiency of syzkaller.  If I were to make changes to the code or change our syzkaller infrastructure to incorporate new harware, what would be the most appropriate metric(s) to measure to see how performance is impacted? Another closely related question is what should the testing methodology be?  Should I start a series of test by clearing /resetting the accumulated kernel corpus?  Should I clear the working directory between tests or keep it as it is?  I am assuming that I should use a consistent kernel and syzkaller version, etc.

 

Thanks!

 

Ryan

[Aleksandr Nogikh]

Hi Ryan,

On Wed, May 7, 2025 at 8:13 PM 'Adamson, Ryan' via syzkaller
<syzk...@googlegroups.com> wrote:
>
> All,
>
>
>
> I had a question related to testing the efficiency of syzkaller. If I were to make changes to the code or change our syzkaller infrastructure to incorporate new harware, what would be the most appropriate metric(s) to measure to see how performance is impacted? Another closely related question is what should the testing methodology be? Should I start a series of test by clearing /resetting the accumulated kernel corpus? Should I clear the working directory between tests or keep it as it is?

I wish we knew the right answers to these questions ourselves :)

When we do code changes that may affect fuzzing performance, we often
use https://github.com/google/syzkaller/blob/master/docs/syz_testbed.md
to compare two (or more) syzkallers side by side. Having ~20 runs of
each syzkaller version for e.g. 24 hours is usually enough to see
significant changes in their performance: how much coverage/signal
they reached, how many different bugs they triggered, etc. In many
cases it's just unclear what conclusions to draw from the results.

Regarding the corpus - at least for us, it's more practical to take
the already saturated corpus from syzbot and let each instance start
from that exact corpus. Since we're doing continuous fuzzing, we are
not that much interested in the speed with which it could go from 0
corpus to saturated one, but rather how far it can bring an already
existing non-zero corpus.

> I am assuming that I should use a consistent kernel and syzkaller version, etc.

If you are testing the effect of syzkaller changes on fuzzing
performance, then yes, you'd better keep everything else exactly the
same.

--
Aleksandr

>
>
>
> Thanks!
>
>
>
> Ryan
>

[Adamson, Ryan]

I was looking at syz-hub documentation since I am exploring distributed fuzzing (https://github.com/google/syzkaller/blob/master/docs/hub.md) and was wondering if the syz-manager processes do any coordination through the hub to prevent the duplication of fuzzing work (i.e. execution of the same or very similar programs on more than one fuzzer). I understand that very small mutations potentially expose other code pathways that were previously unknown to the fuzzer and so maybe it’s not actually an issue given enough randomness in creating mutations.

 

Thanks!

 

Ryan

[Dmitry Vyukov]

On Fri, 16 May 2025 at 21:50, 'Adamson, Ryan' via syzkaller
<syzk...@googlegroups.com> wrote:
>
> I was looking at syz-hub documentation since I am exploring distributed fuzzing (https://github.com/google/syzkaller/blob/master/docs/hub.md) and was wondering if the syz-manager processes do any coordination through the hub to prevent the duplication of fuzzing work (i.e. execution of the same or very similar programs on more than one fuzzer). I understand that very small mutations potentially expose other code pathways that were previously unknown to the fuzzer and so maybe it’s not actually an issue given enough randomness in creating mutations.

Hi Ryan,

There is no scheme to avoid duplicate/similar work in syz-hub, nor in
standalone syz-manager even.

The reason is as you noted: it's very hard to understand what is "similar" work.

[Dmitry Vyukov]

On Tue, 27 May 2025 at 16:43, Adamson, Ryan <adam...@ornl.gov> wrote:
>
> Thanks for your reply! What is the largest fuzzing ‘farm’ that you have come across in terms of executions per second?

+syzkaller mailing list (please keep in CC)

The only I come across is out syzbot. You can see execs/sec here:

https://syzbot.org/upstream/graph/fuzzing?Instances=ci-qemu-gce-upstream-auto&Instances=ci-qemu-native-arm64-kvm&Instances=ci-snapshot-upstream-root&Instances=ci-upstream-kasan-gce-root&Metrics=ExecsPerSec&Months=1


> Ryan
>
>
>
> From: Dmitry Vyukov <dvy...@google.com>
> Date: Monday, May 19, 2025 at 2:03 AM
> To: Adamson, Ryan <adam...@ornl.gov>
> Cc: syzk...@googlegroups.com <syzk...@googlegroups.com>
> Subject: [EXTERNAL] Re: Syz-manager coordination
>
> On Fri, 16 May 2025 at 21: 50, 'Adamson, Ryan' via syzkaller <syzkaller@ googlegroups. com> wrote: > > I was looking at syz-hub documentation since I am exploring distributed fuzzing (https: //urldefense. us/v2/url?u=https-3A__github. com_google_syzkaller_blob_master_docs_hub. md&d=DwIFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=XRxVLvNemnSfctPEl3YuK8DjYBcrdklSgfcp01e08Rs&m=JzF9cOwDWm037mdPxUUyPFi5-LqJIW8FoD8Kujd9TSkQTAzrIqm4CQ_qZlVZKPOv&s=cSo_s1A-5kJvlJeSAoIsTR-nEPYtuCh1u3QE7wi7Xq4&e=)

>
> On Fri, 16 May 2025 at 21:50, 'Adamson, Ryan' via syzkaller
>
> <syzk...@googlegroups.com> wrote:
>
> >
>

> > I was looking at syz-hub documentation since I am exploring distributed fuzzing (https://urldefense.us/v2/url?u=https-3A__github.com_google_syzkaller_blob_master_docs_hub.md&d=DwIFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=XRxVLvNemnSfctPEl3YuK8DjYBcrdklSgfcp01e08Rs&m=JzF9cOwDWm037mdPxUUyPFi5-LqJIW8FoD8Kujd9TSkQTAzrIqm4CQ_qZlVZKPOv&s=cSo_s1A-5kJvlJeSAoIsTR-nEPYtuCh1u3QE7wi7Xq4&e=) and was wondering if the syz-manager processes do any coordination through the hub to prevent the duplication of fuzzing work (i.e. execution of the same or very similar programs on more than one fuzzer). I understand that very small mutations potentially expose other code pathways that were previously unknown to the fuzzer and so maybe it’s not actually an issue given enough randomness in creating mutations.

[Adamson, Ryan]

Thanks for the link to syzbot dashboard. I am noticing in my experimentation that the number of execs/s will drop over time, sometimes by an order of magnitude or more. Is this expected as coverage becomes more saturated?

 

Thanks!

 

Ryan

 

From: Dmitry Vyukov <dvy...@google.com>
Date: Friday, May 30, 2025 at 1:59 AM
To: Adamson, Ryan <adam...@ornl.gov>, syzkaller <syzk...@googlegroups.com>
Subject: Re: [EXTERNAL] Re: Syz-manager coordination

On Tue, 27 May 2025 at 16: 43, Adamson, Ryan <adamsonrm@ ornl. gov> wrote: > > Thanks for your reply! What is the largest fuzzing ‘farm’ that you have come across in terms of executions per second? +syzkaller mailing list (please keep

On Tue, 27 May 2025 at 16:43, Adamson, Ryan <adam...@ornl.gov> wrote:

> 

> Thanks for your reply! What is the largest fuzzing ‘farm’ that you have come across in terms of executions per second?

+syzkaller mailing list (please keep in CC)

The only I come across is out syzbot. You can see execs/sec here:

https://urldefense.us/v2/url?u=https-3A__syzbot.org_upstream_graph_fuzzing-3FInstances-3Dci-2Dqemu-2Dgce-2Dupstream-2Dauto-26Instances-3Dci-2Dqemu-2Dnative-2Darm64-2Dkvm-26Instances-3Dci-2Dsnapshot-2Dupstream-2Droot-26Instances-3Dci-2Dupstream-2Dkasan-2Dgce-2Droot-26Metrics-3DExecsPerSec-26Months-3D1&d=DwIFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=XRxVLvNemnSfctPEl3YuK8DjYBcrdklSgfcp01e08Rs&m=x8a0l9z-e_Vq-RSofH4bkn3AJkl2eNgIi0MEXf9lwUyB8PrNYUzRsKuve35D0au5&s=2qPIzaWfy5JLbwa8LW8HdKpPNQvUV7NZy5B8qlsgDvA&e=

> Ryan

> 

> 

> 

> From: Dmitry Vyukov <dvy...@google.com>

> Date: Monday, May 19, 2025 at 2:03 AM

> To: Adamson, Ryan <adam...@ornl.gov>

> Cc: syzk...@googlegroups.com <syzk...@googlegroups.com>

> Subject: [EXTERNAL] Re: Syz-manager coordination

> 

> On Fri, 16 May 2025 at 21: 50, 'Adamson, Ryan' via syzkaller <syzkaller@ googlegroups. com> wrote: > > I was looking at syz-hub documentation since I am exploring distributed fuzzing (https: //urldefense. us/v2/url?u=https-3A__github. com_google_syzkaller_blob_master_docs_hub. md&d=DwIFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=XRxVLvNemnSfctPEl3YuK8DjYBcrdklSgfcp01e08Rs&m=JzF9cOwDWm037mdPxUUyPFi5-LqJIW8FoD8Kujd9TSkQTAzrIqm4CQ_qZlVZKPOv&s=cSo_s1A-5kJvlJeSAoIsTR-nEPYtuCh1u3QE7wi7Xq4&e=)

> 

> On Fri, 16 May 2025 at 21:50, 'Adamson, Ryan' via syzkaller

> 

> <syzk...@googlegroups.com> wrote:

> 

> >

> 

> > I was looking at syz-hub documentation since I am exploring distributed fuzzing (https://urldefense.us/v2/url?u=https-3A__github.com_google_syzkaller_blob_master_docs_hub.md&d=DwIFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=XRxVLvNemnSfctPEl3YuK8DjYBcrdklSgfcp01e08Rs&m=JzF9cOwDWm037mdPxUUyPFi5-LqJIW8FoD8Kujd9TSkQTAzrIqm4CQ_qZlVZKPOv&s=cSo_s1A-5kJvlJeSAoIsTR-nEPYtuCh1u3QE7wi7Xq4&e=)and was wondering if the syz-manager processes do any coordination through the hub to prevent the duplication of fuzzing work (i.e. execution of the same or very similar programs on more than one fuzzer). I understand that very small mutations potentially expose other code pathways that were previously unknown to the fuzzer and so maybe it’s not actually an issue given enough randomness in creating mutations.

[Aleksandr Nogikh]

Hi Ryan,

On Thu, Jun 5, 2025 at 5:27 PM 'Adamson, Ryan' via syzkaller
<syzk...@googlegroups.com> wrote:
>
> Thanks for the link to syzbot dashboard. I am noticing in my experimentation that the number of execs/s will drop over time, sometimes by an order of magnitude or more. Is this expected as coverage becomes more saturated?

Yes, we observe that too. During the corpus triage stage, syzkaller
intentionally goes from the smaller to bigger programs from its
corpus.db file, which naturally leads to a very big slow down during
the process. After the corpus is fully triaged and the tool is just
normally fuzzing, it still acquires more and more complex programs
over time, which are also slower.

The slowdown in exec/sec also heavily depends on the fuzzed system
calls. There are very slow (e.g. mount/syz_mount_image) and very fast
ones (networking, simple file operations). If you restrict the
fuzzer's enable_syscalls only to the lightweight ones, its observed
performance will be very different.

--
Aleksandr

> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/syzkaller/SA1PR09MB7840D4644539B551A933A3B2D16FA%40SA1PR09MB7840.namprd09.prod.outlook.com.