Should `go test` without `-fuzz` ever be non-deterministic?
17 views
Skip to first unread message
thepudds
May 9, 2019, 9:35:46 AM5/9/19
to Golang Fuzzing
Hi all,
We'll see if this is useful, but I am starting a new thread on item 6 from here:
This might be a quick topic, or might be a meatier topic.
In this post, I will just quote what was said on item 6 in that bigger "List of possible modifications to the March 2017 proposal" thread, which was:
> -------------------------------------------------------------------
> 6. Should `go test` without `-fuzz` ever be non-deterministic?
> -------------------------------------------------------------------
>
> It seems reasonable to expect non-determinism if you explicitly ask for fuzzing, such as by supplying `-fuzz`.
>
> However, if you are just doing something like `go test .` (without a `-fuzz` flag), then any non-determinism should be carefully evaluated given people might not expect non-determinism in that situation.
>
> One example would be in a normal (non-fuzz) invocation of something like `go test ./...`. If that fails in something like travis, can you repeat that failure?
>
> There are multiple forms of non-determinism, including:
>
> 6a. Non-determinism from `rand` or similar
>
> The March 2017 proposal says:
>
> "go test runs fuzz functions as unit tests. ... Fuzz functions are executed on all inputs from the corpus and on some amount of newly generated inputs (for 1s in normal mode and for 0.1s in short mode)"
>
> If there are newly generated inputs, are those random inputs? Or deterministic in some way? Or random, but with a seed printed if there is a failure (so that the same seed can be re-used to recreate the failure if needed)?
>
> 6b. Execution time-based non-determinism
>
> If the execution time is "1s in normal mode and for 0.1s in short mode", does that mean you might get a failure on a powerful machine that does not occur within the same wall clock time on a less powerful machine?
---------------------------------------------------------------------
Romain responded:
> I would add that in this case, fuzzs will behave the same way as benchmarks: non of them are deterministic, and this is exactly why they are not the default (in addition to the time consumption thing).
Dmitry also responded:
> Provided that "go test -fuzz=. ./..." will also work (which seems to
> be what we want), then I think we better stick to simpler and safer
> deterministic load in normal "go test".
> There should be an easy way to "promote" a crashing input to a
> unit-test, e.g. by going:
> cp $GOAPTH/pkg/.../INPUT testdata/fuzz/
> and then _these_ inputs will run during normal go test.
>
Regards,
thepudds
thepudds
May 9, 2019, 9:37:26 AM5/9/19
to Golang Fuzzing
Hi Dmitry,
You wrote:
> Provided that "go test -fuzz=. ./..." will also work (which seems to
> be what we want), then I think we better stick to simpler and safer
> deterministic load in normal "go test".
> There should be an easy way to "promote" a crashing input to a
> unit-test, e.g. by going:
> cp $GOAPTH/pkg/.../INPUT testdata/fuzz/
> and then _these_ inputs will run during normal go test.
Can you expand slightly on your comment?
The simplest way to eliminate all non-determinism is to just run the corpus as unit tests if someone runs `go test ./...` (without a `-fuzz` flag), and eliminate any mutation-based generation of new inputs in that case.
I think that also substantially simplifies the proposal discussion with the core Go team.
Personally, that is what I would vote for.
The proposal currently reads:
"go test runs fuzz functions as unit tests. Fuzz functions are selected with -run flag on par with tests (i.e. all by default). Fuzz functions are executed on all inputs from the corpus and on some amount of newly generated inputs (for 1s in normal mode and for 0.1s in short mode)."
So for example this could be struck:
"and on some amount of newly generated inputs (for 1s in normal mode and for 0.1s in short mode)."
However, I wanted to confirm that's what you meant, vs. maybe you had something else in mind (e.g., trying to have deterministic mutations for 0.1 or 1 sec and then dealing with the execution time based non-determinism, or perhaps something else).
And of course, mutation-based generation of inputs during `go test ./...` (without a `-fuzz` flag) could be a future extension after there is more real-world experience.
Regards,
thepudds
Dmitry Vyukov
May 9, 2019, 10:28:33 AM5/9/19
to thepudds, Golang Fuzzing
I think it's good to have 1 thread per 1 question.
Yes, by "some amount of newly generated inputs" in the proposal I
meant non-deterministically generated inputs.
But now I think we should drop this part of the proposal, and only
execute inputs from the corpus (deterministic).
meant non-deterministically generated inputs.
But now I think we should drop this part of the proposal, and only
execute inputs from the corpus (deterministic).
Dmitry Vyukov
May 9, 2019, 10:37:58 AM5/9/19
to thepudds, Golang Fuzzing
organize and discover the corpus.
Traditionally, for fuzzer corpus is a single directory with files, all
of which are under control of the fuzzer (i.e. it can
add/remove/change/etc them as necessary).
But I thinking towards splitting corpus into "fixed" part and "dynamic" part.
"Dynamic" part is the traditional definition of the corpus (lots of
random inputs that fuzzer generates and changes).
"Fixed" part is controlled by humans and is checked in into VCS
(generally) (say, this will be located in testdata/fuzz). There are 2
ways how inputs appear in the "fixed" part: (1) a developer copies a
single crashing input from the dynamic corpus, which now becomes a
regression test; (2) a developer explicitly writes a new input (or
copies them from some existing base of tests, e.g. different real jpeg
pictures), this is also useful for so-called bootstrapping of the
corpus.
Now, fuzzer will also take, execute and mutate inputs from the "fixed"
part, but only do any changes (add/remove/change) to the "dynamic"
part.
Now, go test may run only fixed inputs or both fixed and dynamic.
Running only fixed is fully deterministic. Running both may lead to
unwanted failures. Consider, you run a fuzzer, it produced a crashing
input in the corpus. Now, you do some unrelated change to the code and
re-run tests. Tests crash because you have that crashing input in the
dynamic corpus now (other people don't, it's only on your disk for
now).
Romain Baugue
May 9, 2019, 10:48:48 AM5/9/19
to Dmitry Vyukov, thepudds, Golang Fuzzing
Splitting inputs seems to allow us a deterministic `go test`, which is good. But I would worry about having the user manually copy anything from $GOPATH/pkg, it seems to me that this directory should stay "hidden" from the user.
What about two directories in `testdata`? With a suffix to differentiate the fuzzer-corpus from the test-corpus ?
--
You received this message because you are subscribed to the Google Groups "Golang Fuzzing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-fuzzing-pr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-fuzzing-proposal/CACT4Y%2Bafg4068GUBzyohcNi9D5Tcx4LSJmnLoCbBjbeB_e%2BAyw%40mail.gmail.com.
Dmitry Vyukov
May 9, 2019, 11:01:01 AM5/9/19
to Romain Baugue, thepudds, Golang Fuzzing
> Splitting inputs seems to allow us a deterministic `go test`, which is good. But I would worry about having the user manually copy anything from $GOPATH/pkg, it seems to me that this directory should stay "hidden" from the user.
>
> What about two directories in `testdata`? With a suffix to differentiate the fuzzer-corpus from the test-corpus ?
This conflicts with "fuzzer must not dirty vcs state".
>
> What about two directories in `testdata`? With a suffix to differentiate the fuzzer-corpus from the test-corpus ?
I think the fuzzer artifacts dir must be visible to user to some degree.
Fuzzer will store crash outputs there too. And fuzzer will say
"crashed on input X" and user must be able to locate the input X.
Do we have any other options besides $GOPATH/pkg?
Dmitry Vyukov
May 9, 2019, 11:03:19 AM5/9/19
to Romain Baugue, thepudds, Golang Fuzzing
From: Dmitry Vyukov <dvy...@google.com>
Date: Thu, May 9, 2019 at 5:00 PM
To: Romain Baugue
Cc: thepudds, Golang Fuzzing
> > Splitting inputs seems to allow us a deterministic `go test`, which is good. But I would worry about having the user manually copy anything from $GOPATH/pkg, it seems to me that this directory should stay "hidden" from the user.
> >
> > What about two directories in `testdata`? With a suffix to differentiate the fuzzer-corpus from the test-corpus ?
>
> This conflicts with "fuzzer must not dirty vcs state".
>
> I think the fuzzer artifacts dir must be visible to user to some degree.
> Fuzzer will store crash outputs there too. And fuzzer will say
> "crashed on input X" and user must be able to locate the input X.
> Do we have any other options besides $GOPATH/pkg?
Or, how strict is the rule that /pkg must stay hidden?
Ian mentioned GOROOT/pkg several times here:
https://github.com/golang/go/issues/19109#issuecomment-280133900
Date: Thu, May 9, 2019 at 5:00 PM
To: Romain Baugue
Cc: thepudds, Golang Fuzzing
> > Splitting inputs seems to allow us a deterministic `go test`, which is good. But I would worry about having the user manually copy anything from $GOPATH/pkg, it seems to me that this directory should stay "hidden" from the user.
> >
> > What about two directories in `testdata`? With a suffix to differentiate the fuzzer-corpus from the test-corpus ?
>
> This conflicts with "fuzzer must not dirty vcs state".
>
> I think the fuzzer artifacts dir must be visible to user to some degree.
> Fuzzer will store crash outputs there too. And fuzzer will say
> "crashed on input X" and user must be able to locate the input X.
> Do we have any other options besides $GOPATH/pkg?
Ian mentioned GOROOT/pkg several times here:
https://github.com/golang/go/issues/19109#issuecomment-280133900
thepudds
May 9, 2019, 11:19:55 AM5/9/19
to Golang Fuzzing
> Ian mentioned GOROOT/pkg several times here:
> https://github.com/golang/go/issues/19109#issuecomment-280133900
> https://github.com/golang/go/issues/19109#issuecomment-280133900
I'm not 100% sure what Ian meant there.
He might have been talking in the context of when fuzzing the standard library itself? But not sure.
A normal user though does have things in the "pkg" directory that lives immediately under GOPATH, including GOPATH/pkg/mod is the location of the new modules cache (including downloaded dependencies and VCS cache directories).
Side note: in some of these discussions, when someone writes "GOPATH/pkg", I am not always sure if the person means the actual directory named "pkg" that lives immediately under GOPATH, or is "GOPATH/pkg" intended to be shorthand for "GOPATH/src/some/import/path/to/some/pkg".
thepudds
> >> To unsubscribe from this group and stop receiving emails from it, send an email to golang-fuzzing-proposal+unsub...@googlegroups.com.
Dmitry Vyukov
May 9, 2019, 12:00:09 PM5/9/19
to thepudds, Golang Fuzzing
> > Ian mentioned GOROOT/pkg several times here:
> > https://github.com/golang/go/issues/19109#issuecomment-280133900
>
> I'm not 100% sure what Ian meant there.
>
> He might have been talking in the context of when fuzzing the standard library itself? But not sure.
>
> A normal user though does have things in the "pkg" directory that lives immediately under GOPATH, including GOPATH/pkg/mod is the location of the new modules cache (including downloaded dependencies and VCS cache directories).
>
> Side note: in some of these discussions, when someone writes "GOPATH/pkg", I am not always sure if the person means the actual directory named "pkg" that lives immediately under GOPATH, or is "GOPATH/pkg" intended to be shorthand for "GOPATH/src/some/import/path/to/some/pkg".
I don't know what Ian meant. Using src path with break "go fuzz must
> > https://github.com/golang/go/issues/19109#issuecomment-280133900
>
> I'm not 100% sure what Ian meant there.
>
> He might have been talking in the context of when fuzzing the standard library itself? But not sure.
>
> A normal user though does have things in the "pkg" directory that lives immediately under GOPATH, including GOPATH/pkg/mod is the location of the new modules cache (including downloaded dependencies and VCS cache directories).
>
> Side note: in some of these discussions, when someone writes "GOPATH/pkg", I am not always sure if the person means the actual directory named "pkg" that lives immediately under GOPATH, or is "GOPATH/pkg" intended to be shorthand for "GOPATH/src/some/import/path/to/some/pkg".
not dirty vcs state".
So we could use either "GOPATH/pkg/some/import/path/{corpus,crashes}"
or "GOPATH/pkg/fuzz/some/import/path/{corpus,crashes}".
, but I only see how this can work if we use
"GOPATH/src/some/import/path/to/some/pkg".
We can either use "GOPATH/src/some/import/path/to/some/pkg"
>> > >> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-fuzzing-proposal/CACT4Y%2Bafg4068GUBzyohcNi9D5Tcx4LSJmnLoCbBjbeB_e%2BAyw%40mail.gmail.com.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Golang Fuzzing" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-fuzzing-pr...@googlegroups.com.
> You received this message because you are subscribed to the Google Groups "Golang Fuzzing" group.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-fuzzing-proposal/17c63736-d650-4dad-8dbe-7e8a58422c8a%40googlegroups.com.
Romain Baugue
May 11, 2019, 10:21:03 AM5/11/19
to Golang Fuzzing
I see what the point about not dirtying the VCS state is, but I'm not sure I completely agree with it. Things like code generation directives already exist and they dirty the VCS state at compile-time, why should a testing feature whose purpose is to generate a corpus not do it?
In addition, using GOPATH/pkg/fuzz/xxx, I'm not sure about the way the user should promote a generated input to the checked-in corpus. Having to do a manual copy seems clumsy, error-prone at best and too much arcane for the "standard user" I imagine. We would have to add tooling for this and I'm not covinced this would be better.
>> > >> To unsubscribe from this group and stop receiving emails from it, send an email to golang-fuzzing-proposal+unsub...@googlegroups.com.
>> > >> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-fuzzing-proposal/CACT4Y%2Bafg4068GUBzyohcNi9D5Tcx4LSJmnLoCbBjbeB_e%2BAyw%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups "Golang Fuzzing" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-fuzzing-proposal+unsub...@googlegroups.com.
Dmitry Vyukov
May 16, 2019, 10:40:55 AM5/16/19
to Romain Baugue, Golang Fuzzing
On Sat, May 11, 2019 at 4:21 PM Romain Baugue <romain...@gmail.com> wrote:
>
> I see what the point about not dirtying the VCS state is, but I'm not sure I completely agree with it. Things like code generation directives already exist and they dirty the VCS state at compile-time, why should a testing feature whose purpose is to generate a corpus not do it?
>
> In addition, using GOPATH/pkg/fuzz/xxx, I'm not sure about the way the user should promote a generated input to the checked-in corpus. Having to do a manual copy seems clumsy, error-prone at best and too much arcane for the "standard user" I imagine. We would have to add tooling for this and I'm not covinced this would be better.
Let's keep the discussion about dirtying in one place:
>
> I see what the point about not dirtying the VCS state is, but I'm not sure I completely agree with it. Things like code generation directives already exist and they dirty the VCS state at compile-time, why should a testing feature whose purpose is to generate a corpus not do it?
>
> In addition, using GOPATH/pkg/fuzz/xxx, I'm not sure about the way the user should promote a generated input to the checked-in corpus. Having to do a manual copy seems clumsy, error-prone at best and too much arcane for the "standard user" I imagine. We would have to add tooling for this and I'm not covinced this would be better.
https://groups.google.com/d/msg/golang-fuzzing-proposal/K75JJ6fI--o/vfHPF0o7BAAJ
>> >> > >> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-fuzzing-proposal/CACT4Y%2Bafg4068GUBzyohcNi9D5Tcx4LSJmnLoCbBjbeB_e%2BAyw%40mail.gmail.com.
>> >
>> > --
>> > You received this message because you are subscribed to the Google Groups "Golang Fuzzing" group.
>> > To unsubscribe from this group and stop receiving emails from it, send an email to golang-fuzzing-pr...@googlegroups.com.
>> >
>> > --
>> > You received this message because you are subscribed to the Google Groups "Golang Fuzzing" group.
>> > To view this discussion on the web visit https://groups.google.com/d/msgid/golang-fuzzing-proposal/17c63736-d650-4dad-8dbe-7e8a58422c8a%40googlegroups.com.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Golang Fuzzing" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-fuzzing-pr...@googlegroups.com.
> You received this message because you are subscribed to the Google Groups "Golang Fuzzing" group.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-fuzzing-proposal/228004ad-8b94-4e6f-9b47-3bf068e0e588%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages