fuzzing rich signatures beyond Fuzz([]byte) int?

[thepudds]

Hi there,

Just a quick comment that the fzgo prototype now supports fuzzing rich signatures such as:

  func FuzzRegexRichArgs(re string, input []byte, posix bool) (bool, error)

  

which you can then fuzz via:

  fzgo test -fuzz=FuzzRegexRichArgs

The way it works is when fzgo is asked to fuzz a set of functions, it checks each function to see if it a rich signature (that is, not a classic 'Fuzz([]byte) int' signature). 

If it finds a rich signature, such as in the example above, fzgo behind the scenes automatically generates a wrapper that creates random values for the rich parameters based on a []byte handed in by dvyukov/go-fuzz. There is a fzgo.randparam package that is built on top of github.com/google/gofuzz. fzgo.randparam has some more control of the randomness to more directly pull values from the []byte from dvyukov/go-fuzz (mainly so that dvyukov/go-fuzz literal injection works end-to-end, which wouldn't work if just directly using github.com/google/gofuzz, as well as in order to not waste as many input bytes in the original []byte handed in, and to control what happens when you run out of bytes in the input []byte, etc.).

Once the fuzzing starts, dvyukov/go-fuzz is still in charge, including the recorded corpus stores the serialized arguments that were used to fuzz the rich signatures, and dvyukov/go-fuzz still does its genetic mutations, etc.

For example, this gets guessed within a few seconds of fuzzing:

    func FuzzHardToGuessNumber(guessMe int64, list []string) {

        if guessMe == 0x123456789 {

            panic("bingo")

        }

    }

The rich signature support in fzgo is still at an early stage, but it seems to work, and it is fun! 

    

It only fills in public struct elements, and can't automatically fill in interfaces or a passed-in func pointer, but it should handle a decent variety of rich signatures.

FWIW, I think supporting rich signatures is really the killer feature of the proposal. 

More than anything, I think cmd/go supporting rich signatures for fuzzing moves it from "Hmm, fuzzing sounds interesting to do someday" to instead being something that is much more natural to get started with, and more akin to unit tests.

That said, it is likely still very reasonable for the official proposal to say that support for rich signatures is a desired future direction rather than required for the first version, which is more-or-less what I think the proposal says now. (Incremental progress, and all that).

Separately, I am hopefully about to post in some of the other fuzz proposal threads a little later today with some other questions/comments, as well as I am going to try to post an updated at least partial list of open questions regarding the proposal.

Regards,

thepudds

[Josh Bleecher Snyder]

Awesome!

Any reason not to upstream this to go-fuzz right now?

If/when fuzz signatures grow a fuzzing.F, that’ll go at the beginning? Any prospective conflicts with that, as it stands?

-j

--
You received this message because you are subscribed to the Google Groups "golang-fuzzing-proposal" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-fuzzing-pr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-fuzzing-proposal/c9eeed25-3cf4-4365-836c-273ed4f30ea3%40googlegroups.com.

[thepudds]

> Awesome!

> 

> Any reason not to upstream this to go-fuzz right now?

Hi josharian,

I took a look last night whether or not it might be workable to upstream rich signature support to dvyukov/go-fuzz.

It looks feasible so far. I'd be happy to try to put together a PR.

I don't think there would be any issue with later adding a fuzzing.F parameter if/when that ends up happening. For example, it could just check to see if the user function's signature includes that, and use it if found. 

Some other things to think about:

Upstreaming this would currently require adding two dependencies to dvyukov/go-fuzz:

  * github.com/google/gofuzz (for recursively walking data structures)

  * golang.org/x/tools/imports (the library, not the tool; it is currently used to resolve import paths if a rich signature is using a type from outside the package being fuzzed). 

  

Would that be reasonable? 

dvyukov/go-fuzz already depends on other things in x/tools (e.g,. x/tools/go/packages). It also vendors some other thing things like elazarl/go-bindata-assetfs, so maybe it is not crazy to add 1-2 more dependencies, but probably a good topic to at least try to settle upfront.

Also, if it was to land upstream, would it be reasonable to label it "experimental" at first, or something like that? 

That would be for a few different reasons. First, it would help communicate that it is fairly fresh functionality. More importantly, though, the semantics of how it interprets an input data []byte and transforms that into arguments would probably be tweaked based on early experience, which means the interpretation of an on-disk corpus could change. If that happens, probably better to set people's expectations up front. After some additional experience, or better yet, if the "fuzzing as a first class citizen" proposal lands, the on-disk format could be versioned, and at some point there could be a statement that backwards compatibility with any on-disk corpus would be maintained... but promising backwards compatibility seems like it might be overkill for the very first thing to get merged? 

Regards,

thepudds

On Sunday, August 4, 2019 at 4:21:49 PM UTC-4, Josh Bleecher Snyder wrote:

Awesome!

Any reason not to upstream this to go-fuzz right now?

If/when fuzz signatures grow a fuzzing.F, that’ll go at the beginning? Any prospective conflicts with that, as it stands?

-j

On Sun, Aug 4, 2019 at 12:05 PM thepudds <thepudds1....> wrote:

Hi there,

Just a quick comment that the fzgo prototype now supports fuzzing rich signatures such as:

  func FuzzRegexRichArgs(re string, input []byte, posix bool) (bool, error)

  

which you can then fuzz via:

  fzgo test -fuzz=FuzzRegexRichArgs

The way it works is when fzgo is asked to fuzz a set of functions, it checks each function to see if it a rich signature (that is, not a classic 'Fuzz([]byte) int' signature). 

If it finds a rich signature, such as in the example above, fzgo behind the scenes automatically generates a wrapper that creates random values for the rich parameters based on a []byte handed in by dvyukov/go-fuzz. There is a fzgo.randparam package that is built on top of github.com/google/gofuzz. fzgo.randparam has some more control of the randomness to more directly pull values from the []byte from dvyukov/go-fuzz (mainly so that dvyukov/go-fuzz literal injection works end-to-end, which wouldn't work if just directly using github.com/google/gofuzz, as well as in order to not waste as many input bytes in the original []byte handed in, and to control what happens when you run out of bytes in the input []byte, etc.).

Once the fuzzing starts, dvyukov/go-fuzz is still in charge, including the recorded corpus stores the serialized arguments that were used to fuzz the rich signatures, and dvyukov/go-fuzz still does its genetic mutations, etc.

For example, this gets guessed within a few seconds of fuzzing:

    func FuzzHardToGuessNumber(guessMe int64, list []string) {

        if guessMe == 0x123456789 {

            panic("bingo")

        }

    }

The rich signature support in fzgo is still at an early stage, but it seems to work, and it is fun! 

    

It only fills in public struct elements, and can't automatically fill in interfaces or a passed-in func pointer, but it should handle a decent variety of rich signatures.

FWIW, I think supporting rich signatures is really the killer feature of the proposal. 

More than anything, I think cmd/go supporting rich signatures for fuzzing moves it from "Hmm, fuzzing sounds interesting to do someday" to instead being something that is much more natural to get started with, and more akin to unit tests.

That said, it is likely still very reasonable for the official proposal to say that support for rich signatures is a desired future direction rather than required for the first version, which is more-or-less what I think the proposal says now. (Incremental progress, and all that).

Separately, I am hopefully about to post in some of the other fuzz proposal threads a little later today with some other questions/comments, as well as I am going to try to post an updated at least partial list of open questions regarding the proposal.

Regards,

thepudds

--
You received this message because you are subscribed to the Google Groups "golang-fuzzing-proposal" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-fuzzing-proposal+unsub...@googlegroups.com.

[Josh Bleecher Snyder]

It looks feasible so far. I'd be happy to try to put together a PR.

SGTM. I’m happy to help review, but in the end I’ll defer to Dmitry about the direction of go-fuzz. So maybe we should wait for him to weigh in here first, on this and the other questions below.

Some other things to think about:

Upstreaming this would currently require adding two dependencies to dvyukov/go-fuzz:

  * github.com/google/gofuzz (for recursively walking data structures)

  * golang.org/x/tools/imports (the library, not the tool; it is currently used to resolve import paths if a rich signature is using a type from outside the package being fuzzed). 

  

Would that be reasonable? 

I think so.

We do have a problem with dependencies and modules that this will make slightly worse, but (a) we already have a problem and (b) I think that we have the means to a solution if so desired, namely fixing https://github.com/dvyukov/go-fuzz/issues/252 and then using Go modules for go-fuzz itself.

Also, if it was to land upstream, would it be reasonable to label it "experimental" at first, or something like that? 

I'm not sure that the label is critical, one way or the other. But I agree that we should make super visible and clear in the README that the on-disk format is unstable, so there is a likelihood of data loss. I think we need a story someday for this, but being able to make steady incremental progress is more important (IMO).

And IIUC, Russ's take on accepting this into the Go core is that he wants all these kinds of questions to be experimented with and worked out in go-fuzz. So let's do it! :)

-josh

P.S. I've cc'd here people from the two Go fuzzing companies I'm aware of, in case they want to weigh in. And Daniel Marti. I won't do it again; I'll let you subscribe to the group, if desired.

[Dmitry Vyukov]

On Mon, Aug 12, 2019 at 1:12 AM Josh Bleecher Snyder
<josh...@gmail.com> wrote:
>
>> It looks feasible so far. I'd be happy to try to put together a PR.
>
>
> SGTM. I’m happy to help review, but in the end I’ll defer to Dmitry about the direction of go-fuzz. So maybe we should wait for him to weigh in here first, on this and the other questions below.
>
>
>> Some other things to think about:
>>
>> Upstreaming this would currently require adding two dependencies to dvyukov/go-fuzz:
>> * github.com/google/gofuzz (for recursively walking data structures)
>> * golang.org/x/tools/imports (the library, not the tool; it is currently used to resolve import paths if a rich signature is using a type from outside the package being fuzzed).
>>
>> Would that be reasonable?

How much do we need from github.com/google/gofuzz? Does it do exactly
what we need in a way we need it? "recursively walking data
structures" sounds like very moderate amount of simple code. If that's
the case, I would consider redoing it from scratch.

> I think so.
>
> We do have a problem with dependencies and modules that this will make slightly worse, but (a) we already have a problem and (b) I think that we have the means to a solution if so desired, namely fixing https://github.com/dvyukov/go-fuzz/issues/252 and then using Go modules for go-fuzz itself.
>
>
>> Also, if it was to land upstream, would it be reasonable to label it "experimental" at first, or something like that?
>
>
> I'm not sure that the label is critical, one way or the other. But I agree that we should make super visible and clear in the README that the on-disk format is unstable, so there is a likelihood of data loss. I think we need a story someday for this, but being able to make steady incremental progress is more important (IMO).

Changing on-disk format initially is fine.
But it would be good to figure out some consistent long term story for
the format over time.

> And IIUC, Russ's take on accepting this into the Go core is that he wants all these kinds of questions to be experimented with and worked out in go-fuzz. So let's do it! :)
>
> -josh
>
> P.S. I've cc'd here people from the two Go fuzzing companies I'm aware of, in case they want to weigh in. And Daniel Marti. I won't do it again; I'll let you subscribe to the group, if desired.
>
>

> --
> You received this message because you are subscribed to the Google Groups "golang-fuzzing-proposal" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to golang-fuzzing-pr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-fuzzing-proposal/CAFAcib8RuwbsHiDq%2BK-UBe1F3UfLzQRJFRv0F8M_3x7tsypGTQ%40mail.gmail.com.

[Dmitry Vyukov]

On Mon, Aug 12, 2019 at 11:20 AM Yevgeny Pats <y...@fuzzit.dev> wrote:
>
> Maybe I missed the previous discussion. But just to make sure, the idea is support something like property based testing?

Yes. The idea is to natively support arbitrary types rather than just
single []byte. E.g.:

func FuzzRegexp(re, str string, posix bool) {...}

[t hepudds]

Hello there,

The google group should be public, but it is also well known that properly configuring a google group is one of the bigger challenges in the world of technology.

Does this link work for you?

https://groups.google.com/d/msgid/golang-fuzzing-proposal/c9eeed25-3cf4-4365-836c-273ed4f30ea3%40googlegroups.com?utm_medium=email&utm_source=footer

Regards, 

thepudds 

On Aug 12, 2019, at 7:25 PM, Everest Munro-Zeisberger <eve...@fuzzbuzz.io> wrote:

Thanks for copying me into this! Side note: I can't seem to find the "golang-fuzzing-proposal" group anywhere - is it private or something? Searching google groups for it doesn't seem to bring anything up. I'd love to follow along.

This is a great direction for go-fuzz, which will hopefully make building complex fuzz targets much simpler. I'm all onboard!

--

Everest Munro-Zeisberger

Co-Founder @ Fuzzbuzz

[Daniel Martí]

For what it's worth, that link worked for me, and I was able to navigate
the UI and join the group :)

Thanks for the CC; I'll follow along with enthusiasm. I can't wait to
get rid of that mess of a script I wrote to make go-fuzz work with
modules:

https://github.com/mvdan/sh/blob/fuzz/fuzz

Otherwise happy to help with reviews etc. I can't commit proper coding
hours until after summer, I'm afraid.