How to know if fuzzing test is running fine?

[Tarun Kumar]

Hi,

I'm doing fuzzing for my project for the first time. How do we get to know if the testing is going correctly? What are the ways to analyze the test results?

In fuzz testing, a lot of things happen under the hood specifically when we use an off the shelf fuzzer like AFL++.  Other than building afl++ with the target and supplying an initial corpus, there is very little we do and rest is taken care by AFL++.  I am using AFL++ in two different ways to test my Firmware:

Method 1: I used AFL++ gcc plugin to add instrumentation to firmware. Used a mechanism to send out coverage guidance from FW to the host. On the host I use afl-proxy to receive the guidance from FW and then send the next iteration of the test command (afl-proxy is compiled with a test application which can send various test commands to FW and receive a completion code of the command). The test command is nothing but the input corpus which afl-proxy can mutate using the coverage guidance received from the FW. This goes on in a loop.

Method 2: I compile a test application with afl++. This test application has the code to send different test command to FW and receive the completion code of the command. Command and its data are received from the input corpus generated by afl-fuzz (initial input corpus is user provided). Thus, the command sent to firmware are random. Although in this case, FW instrumentation and coverage guidance is not used but application can send the different test commands generated by fuzzer. That’s how the fuzzing takes place. We are adding sanitizers in FW during test.

Questions

1. In both the above scenarios I am not getting any specific issues from the firmware. What are different criteria to conclude if the fuzz testing is running well and is complete?
2. What are the deliverables when fuzzing is considered complete?

Thanks and regards,

Tarun

[Marc]

Method 1 is of course better. you know it is "working" if it discovers
new paths. if it doesnt then something is wrong either with your
coverage gathering or your fuzz input doesnt reach the functionality.

Regards,
Marc

On 11/14/22 12:45, Tarun Kumar wrote:
> Hi,
>
> I'm doing fuzzing for my project for the first time. How do we get to
> know if the testing is going correctly? What are the ways to analyze the
> test results?
>
> In fuzz testing, a lot of things happen under the hood specifically when
> we use an off the shelf fuzzer like AFL++.  Other than building afl++
> with the target and supplying an initial corpus, there is very little we
> do and rest is taken care by AFL++.  I am using AFL++ in two different
> ways to test my Firmware:
>

> *Method 1:* I used AFL++ gcc plugin to add instrumentation to firmware.

> Used a mechanism to send out coverage guidance from FW to the host. On
> the host I use afl-proxy to receive the guidance from FW and then send
> the next iteration of the test command (afl-proxy is compiled with a
> test application which can send various test commands to FW and receive
> a completion code of the command). The test command is nothing but the
> input corpus which afl-proxy can mutate using the coverage guidance
> received from the FW. This goes on in a loop.

> *
> *
> *Method 2:* I compile a test application with afl++. This test

> application has the code to send different test command to FW and
> receive the completion code of the command. Command and its data are
> received from the input corpus generated by afl-fuzz (initial input
> corpus is user provided). Thus, the command sent to firmware are random.
> Although in this case, FW instrumentation and coverage guidance is not
> used but application can send the different test commands generated by
> fuzzer. That’s how the fuzzing takes place. We are adding sanitizers in
> FW during test.

> *
> *
> *Questions*
>
> 1. In both the above scenarios I am not getting any specific issues

> from the firmware. What are different criteria to conclude if the
> fuzz testing is running well and is complete?

> 2. What are the deliverables when fuzzing is considered complete?
>
> Thanks and regards,
> Tarun
>

[Tarun Kumar]

Thanks  Marc

Regards,

Tarun

--

Tarun kumar