Is it possible to tell libfuzzer to not add a certain file to a corpous, even if it yielded new coverage?

[Vincent Ulitzsch]

Hi,

I was wondering if it is possible to tell libfuzzer to not add a certain testcase to the current corpus, even if it yielded new coverage. go-fuzz [1], for example, supports this by leveraging the return value. From the documentation of go-fuzz:

"The function must return 1 if the fuzzer should increase priority of the given input during subsequent fuzzing (for example, the input is lexically correct and was parsed successfully); -1 if the input must not be added to corpus even if gives new coverage; and 0 otherwise; other values are reserved for future use."

Is there something similar for libfuzzer and libprotobuf? If not, do you think this would be an interesting feature to implement via return values?

Thanks,
Vincent

[Konstantin Serebryany]

Hi Vincent, 

I don't think we have such functionality. 

Also, having a return value in the fuzz target was probably a mistake - users find all kinds of ways to use it incorrectly :) 

What is your use case for not adding a corpus element? 

--kcc 

--
You received this message because you are subscribed to the Google Groups "libfuzzer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to libfuzzer+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/libfuzzer/d9caf16a-b6d0-4fc8-b979-326379186353%40googlegroups.com.

[Vincent]

I am trying to fuzz a programming language interpreter and want to reject testcases which yielded an exception.

My hope is that this will yield a test corpus which only consists of semantically valid testcases.
The problem is, in order to know whether a testcase yields an exception, I need to run it against the interpreter - so I need some way of rejecting a testcase although it yielded new coverage.

I ended up hacking on libfuzzer/libprotobuf-mutator to implement this behavior via return value and it seems to be working, but I am still open to any suggestions on other ways to achieve this.

Thank you,
Vincent

On Tuesday, 18 February 2020 19:14:23 UTC+1, Konstantin Serebryany wrote:

Hi Vincent, 

I don't think we have such functionality. 

Also, having a return value in the fuzz target was probably a mistake - users find all kinds of ways to use it incorrectly :) 

What is your use case for not adding a corpus element? 

--kcc 

On Sun, Feb 16, 2020 at 4:16 AM Vincent Ulitzsch <vincent...@gmail.com> wrote:

Hi,

I was wondering if it is possible to tell libfuzzer to not add a certain testcase to the current corpus, even if it yielded new coverage. go-fuzz [1], for example, supports this by leveraging the return value. From the documentation of go-fuzz:

"The function must return 1 if the fuzzer should increase priority of the given input during subsequent fuzzing (for example, the input is lexically correct and was parsed successfully); -1 if the input must not be added to corpus even if gives new coverage; and 0 otherwise; other values are reserved for future use."

Is there something similar for libfuzzer and libprotobuf? If not, do you think this would be an interesting feature to implement via return values?

Thanks,
Vincent

--
You received this message because you are subscribed to the Google Groups "libfuzzer" group.

To unsubscribe from this group and stop receiving emails from it, send an email to libf...@googlegroups.com.

[Konstantin Serebryany]

On Tue, Feb 18, 2020 at 10:22 AM Vincent <vincent....@gmail.com> wrote:

I am trying to fuzz a programming language interpreter and want to reject testcases which yielded an exception.

My hope is that this will yield a test corpus which only consists of semantically valid testcases.
The problem is, in order to know whether a testcase yields an exception, I need to run it against the interpreter - so I need some way of rejecting a testcase although it yielded new coverage.

Got it. Do you think it actually improves the fuzzing progress? 

(I can totally see how it makes the corpus smaller, which typically indirectly improves the fuzzing progress)

 

I ended up hacking on libfuzzer/libprotobuf-mutator to implement this behavior via return value and it seems to be working, but I am still open to any suggestions on other ways to achieve this.

 

Thank you,
Vincent

On Tuesday, 18 February 2020 19:14:23 UTC+1, Konstantin Serebryany wrote:

Hi Vincent, 

I don't think we have such functionality. 

Also, having a return value in the fuzz target was probably a mistake - users find all kinds of ways to use it incorrectly :) 

What is your use case for not adding a corpus element? 

--kcc 

On Sun, Feb 16, 2020 at 4:16 AM Vincent Ulitzsch <vincent...@gmail.com> wrote:

Hi,

I was wondering if it is possible to tell libfuzzer to not add a certain testcase to the current corpus, even if it yielded new coverage. go-fuzz [1], for example, supports this by leveraging the return value. From the documentation of go-fuzz:

"The function must return 1 if the fuzzer should increase priority of the given input during subsequent fuzzing (for example, the input is lexically correct and was parsed successfully); -1 if the input must not be added to corpus even if gives new coverage; and 0 otherwise; other values are reserved for future use."

Is there something similar for libfuzzer and libprotobuf? If not, do you think this would be an interesting feature to implement via return values?

Thanks,
Vincent

--
You received this message because you are subscribed to the Google Groups "libfuzzer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to libf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/libfuzzer/d9caf16a-b6d0-4fc8-b979-326379186353%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "libfuzzer" group.

To unsubscribe from this group and stop receiving emails from it, send an email to libfuzzer+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/libfuzzer/d2cd5ce6-be5c-454b-b338-8f692e1fc924%40googlegroups.com.

[Vincent]

I have yet to conduct a thorough investigation and measurements, but I think it indeed does. My intuition here is this:

Consider the following javascript code:

var x = 5;
x.push(10);
[Some other code]

Now, the javascript interpreter under test would abort execution after line 2, but we would save the testcase also containing other code. I am afraid the fuzzer would waste time mutating the other code below line 2, which will never get executed and thus does not have any chance to yield new coverage.

On Tuesday, 18 February 2020 21:20:01 UTC+1, Konstantin Serebryany wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/libfuzzer/d2cd5ce6-be5c-454b-b338-8f692e1fc924%40googlegroups.com.

[Vincent]

Quick note on that: I conducted first measurements which seem to indicate that this indeed helped fuzzer performance, but unfortunately I am lacking the time at the moment to conduct this with enough rigor for the results to really be meaningful.

[Alex Groce]

Vincent, are you using a custom mutator here? If not already using one, you might want to look at the function use_mutation_tool in https://github.com/agroce/afl-compiler-fuzzer/blob/master/afl-fuzz.c -- it's worked quite well fuzzing Solidity, which is a fairly JavaScript-ish language, for us. Very experimental, but the idea is instead of writing custom mutators for every language, just use mutation operators from mutation testing, adjusted to work on plain source that doesn't necessarily compile (or even parse).

[Alex Groce]

We've only done this with AFL, but libFuzzer for in-line compilers/parsers/interpreters is an obvious interesting target too.

[Vincent]

Interesting, thank you very much! I am using a protobuf description, so I do not need to write custom mutators, but a custom protobuf grammar, which also turns out to be a lot of work.

What exactly the use_mutation_tool for at the moment, fuzzing the compiler and Solidity scripts themselves?

[Alex Groce]

Right now, it's just a variation of AFL that looks interface-wise just like AFL, but is tuned to C-like source code. The solidity compiler is already fuzzed using AFL fairly extensively, but with this we've turned up a few previously-undiscovered issues, most of which we haven't been able to reproduce without the new AFL mutation approach. For solc, it's already set up to fuzz via AFL, you just build with afl-clang/afl-clang++ and then fuzz the solfuzzer binary. I believe that's what they already do.