Replaying only existing corpus entries

[Josh Bleecher Snyder]

Here is an implementation detail I hadn't seen discussed anywhere.

Right now, on start-up, go-fuzz re-runs all existing corpus entries.
However, it also tries to re-minimize them at the same time, which is
slow. This is useful if you're going to fuzz for a while. If you're
just going to fuzz for a few seconds, though (or not at all, if you're
only testing the existing entries), this is unhelpful. On the other
hand, this re-minimization is helpful when the underlying instrumented
code changes, as it will.

What is the right way to handle this?

-josh

[Damian Gryski]

AFL has corpus minimizing as a seperate command: afl-cmin.

On a slightly related note, it would be interesting to benchmark go-fuzz's current minimization logic against the ddmin algorithm: https://github.com/dgryski/go-ddmin

Damian

[thepudds]

Hi all,

A few quick comments, all in the context of #19109:

   * The proposal recently changed so that it no longer generates new mutation-based inputs during go test if the -fuzz flag is not supplied.  As part of that, if you run `go test ./...`, presumably it just rips over the inputs from the corpus to see if the fuzz function fails without any minimization.

  * I think the proposal does not say if the default online minimization still happens at the start of a fuzzing run (e.g., `go test -fuzz=.` or similar). I would guess yes? But not sure.

  * The proposal includes a new `-fuzzminimize` flag (where the equivalent does not exist in go-fuzz).  

  * It might be able to at least in theory maintain metadata about whether or not -fuzzminimize was run to completion vs. any other state changes (e.g., code under test changed), which could then mean skipping the default online minimization? Not sure. cmd/go already caches and invalidates things like test results, so maybe at least plausible in a Go context to track whether or not the corpus was minimized?

Regards,

thepudds

[Dmitry Vyukov]

Since it's an internal detail, I don't care too much at this stage ;)

Running all inputs first can make sense.
syzkaller has much more complex prioritization and queueing logic for
triage, minimization, new external inputs, smashing, distribution. But
it may not matter when you are hammering at 200K hits per second.

[thepudds]

> Right now, on start-up, go-fuzz re-runs all existing corpus entries. 
> However, it also tries to re-minimize them at the same time, which is 
> slow. This is useful if you're going to fuzz for a while. If you're 
> just going to fuzz for a few seconds, though (or not at all, if you're 
> only testing the existing entries), this is unhelpful. On the other 
> hand, this re-minimization is helpful when the underlying instrumented 
> code changes, as it will. 
> What is the right way to handle this? 

Here is a possible alternative to tracking explicit state or computing a hash of the code to see if it has changed, etc.

One way to handle the question might be to take the time spent so far as a hint.

If someone fuzzes for 10 seconds, they probably are looking for a quick test and don't want to spend time on minimizing.

On the other hand, if someone is fuzzing for multiple hours, they would benefit from the online minimization having happened.

So simple but maybe "effective enough" logic could be to pick a duration somewhere in-between those two extremes (say, maybe 30 seconds? 60 seconds?).  

A fuzzing run such as `go test -fuzz=FuzzFunc some/pkg` could then start by:

  1. run through all the input corpus, but don't do the online minimization

  2. start mutating inputs

  3. after 60 seconds, do the online minimization (more-or-less guessing it is worthwhile to settle in for a longer fuzzing session), and continue from there.

Regards,

thepudds