Summary of the situation & roadmap

64 views
Skip to first unread message

Romain Baugue

unread,
May 8, 2019, 12:50:47 PM5/8/19
to Golang Fuzzing
Before anything, one should refer to the original issue: https://github.com/golang/go/issues/19109.
Also, reading the actual proposal that was made is a good idea: https://docs.google.com/document/u/1/d/1zXR-TFL3BfnceEAWytV8bnzB2Tfp6EPFinWVJ5V4QC8/pub.

Incoming: details on what's being done and what's to do.

Romain Baugue

unread,
May 8, 2019, 5:51:43 PM5/8/19
to Golang Fuzzing
What's been done:

- Multiple fuzzing functions have been added to the stdlib: see encoding/csv, encoding/json, html, image/png, golang.org/x/image/tiff
- A repository implementing a tool based on the proposal is in progress in https://github.com/thepudds/fzgo

t hepudds

unread,
May 8, 2019, 6:48:04 PM5/8/19
to Golang Fuzzing
If your name happens to be Dmitry or Josh, this is not particularly interesting post, but here is an attempt to summarize the current situation for the #19109 proposal:

  #19109 "cmd/go: make fuzzing a first class citizen, like tests or benchmarks"

Right now, the #19109 proposal is in "Proposal-Hold" status.

That means that anyone doing work to move the proposal along is (at least partly) doing speculative work that might not actually result in `go test -fuzz=.` ever existing. There is reason for optimism that the proposal would be approved, but it is still speculation as to what will actually happen.

Based on having both watched this particular proposal process as well as having done some work to help nudge it along, my personal take is the most important things to aim for are:
 
  * Complete the prototyping work asked for by the core Go team as part of the #19109 discussion.
  * Get the proposal into a state where it is ready to be re-reviewed.
  * Ask the core Go team to re-review the proposal.
  * Get a thumbs up or thumbs down on the proposal from the core Go team.

Obviously, there will be iteration and those are not strictly linear steps, and there is an interplay between those steps (especially for any prototyping work and any updates to the proposal), and discussion and thinking is required as part of all of those steps.

Getting a thumbs up (or thumbs down) on the proposal is obviously an important milestone itself, but more importantly knowing the proposal is approved (or not) enables people to make more concrete decisions about whether or not to invest time in the work needed to turn the proposal into reality.

----------------------------------------------------------------------------------------------------
Additional Background on the #19109 Proposal
----------------------------------------------------------------------------------------------------

This background has been sitting in the "Background" section of #30719 (so no need to re-read this "Additional Background" section if you've already read that ;-).

After some discussion in #19109, the core Go team asked for a prototype before evaluating the proposal. For example, comments from Russ in #19109 included:

   "it still seems like the right next step is to make 'go-fuzz' the separate command as close to 'go fuzz' the proposed standard command as possible, and to add fuzz tests to at least the x subrepos and maybe the standard library, so that we can understand the implications of having them in the source repos (including how much space we're going to spend on testdata/fuzz corpus directories)."

In order to break things down into more manageable chunks of work (including in an attempt to break out items that must be done by someone on the core Go team vs. could be done by someone from the broader community),  the following steps were suggested in https://github.com/golang/go/issues/19109#issuecomment-441442080:

> Draft Goals for a Prototype
> 
> To be done before an evaluation can be made by the Go proposal review committee:
> 
> 1. Prototype proposed CLI, including interaction with existing 'go test'.
> 2. Add some sample fuzz tests to at least the x subrepos and maybe the standard library.
> 3. Start an initial set of corpus directories for the x repos and maybe the standard library (for example, earlier, the proposal suggested "For the standard library it is proposed to check in corpus into golang.org/x/fuzz repo").
> 4. Understand how much space is used in corpus directories for x subrepos and/or standard library based on those sample fuzz tests.
> 5. Add a new fuzzing signature (or change the existing `Fuzz(data []byte) int` signature) to work with `testing.TB`.

----------------------------------------------------------------------------------------------------
Progress on those 5 "Draft Goals for a Prototype"
----------------------------------------------------------------------------------------------------

Some progress on Step 1 is thepudds/fzgo, a simple work-in-progress prototype that currently implements the majority of the proposed commands from the March 2017 proposal document.

For step 2, there are now four go-fuzz compatible Fuzz functions in the standard library, and one in x/image. (Thanks acln0 and
elwinar!). Related issues:

  
Step 3 is tracked in:

  #31215 -- x/fuzzdata: new repository for fuzzing corpus data 
  
That seems to have some consensus. I was planning on bumping it in the hopes of getting a final decision once the freeze deadline is slightly further past. (There is something like ~2 week window beyond freeze for landing things already in-flight, so people on the core Go team are still pretty focused on the freeze, but might have a few more mental cycles shortly).

Step 4 is mostly dependent on Step 3.

For step 5, Josh has been moving things forward on testing.F/fuzz.F with some input from Dmitry. See for example dvyukov/go-fuzz#218 and dvyukov/go-fuzz#223. There is a good chance the go-fuzz version of testing.F/fuzz.F won't implement testing.TB, which is probably OK for a prototype.
  
  
----------------------------------------------------------------------------------------------------
Compiler and/or coverage integration
----------------------------------------------------------------------------------------------------

Josh has done some experimentation regarding compiler level integration, including some related discussion in #29430 and some experimentation with the new `-trimpath` functionality in 1.13 for much better caching behavior.

Josh has also tried using cmd/internal/edit, including he wrote in https://github.com/dvyukov/go-fuzz/issues/156#issuecomment-490635905:

  "I believe the best fix here is to rewrite our coverage instruction to use cmd/internal/edit, at which point we can benefit from improvements made to cmd/cover. Related: golang/go#29824. Ultimately it would be nice to merge the two of them, but using the same basic mechanism is a good start towards that goal. Unfortunately, switching to cmd/internal/edit is not as simple as it sounds. I tried to do it as part of the switch to go/packages and eventually gave up. It is possible...just complicated."

There is an open question as to whether it is better to aim for tighter or looser integration with the compiler itself. See #29430 (especially the comments from Dmitry and Josh), or some related discussion in https://groups.google.com/d/msg/golang-fuzzing-proposal/9iLIwpglppw/rHdWHmW4AgAJ.

----------------------------------------------------------------------------------------------------
oss-fuzz integration
----------------------------------------------------------------------------------------------------

oss-fuzz integration is important, but there is a strong desire to let that run on a separate parallel track.

See for example:


----------------------------------------------------------------------------------------------------

Finally, the comments above are almost entirely focused on the #19109 proposal.

However, that of course doesn't mean that's the only valuable thing to do in the world of go-fuzz. 

Josh in particular has been making some very, very nice improvements to go-fuzz itself. In Josh's words:

  "FYI, I have been working on go-fuzz some recently. ... I have been doing so with one eye on #19109, but mainly with an emphasis on making it more useful and usable now."
  
Some very nice improvements have been to speed on multiple dimensions, cutting over to golang.org/x/tools/go/packages, making all arguments to `go-fuzz` and `go-fuzz-build` optional, many bug fixes, etc.

In any event, that is an attempt to summarize the current situation and recent progress...

Regards,
thepudds

Dmitry Vyukov

unread,
May 9, 2019, 6:00:17 AM5/9/19
to t hepudds, Golang Fuzzing
> If your name happens to be Dmitry or Josh, this is not particularly interesting post, but here is an attempt to summarize the current situation for the #19109 proposal:

My name is Dmitry, but I am losing track of things too :)
Thanks for putting this all together.
It would be useful to have a condensed bullet-point-list version of
this in the proposal doc (with links to more info, e.g. this thread).
Then we could keep it up-to-date and anybody could easily see the
current status.
> --
> You received this message because you are subscribed to the Google Groups "Golang Fuzzing" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-fuzzing-pr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-fuzzing-proposal/d3f6b842-0a38-46ed-8256-baadcefefa85%40googlegroups.com.

thepudds

unread,
May 9, 2019, 6:32:44 PM5/9/19
to Golang Fuzzing
> My name is Dmitry, but I am losing track of things too :) 
> Thanks for putting this all together. 
> It would be useful to have a condensed bullet-point-list version of 
> this in the proposal doc (with links to more info, e.g. this thread). 
> Then we could keep it up-to-date and anybody could easily see the 
> current status. 

Hi Dmitry,

That makes sense. I will take a stab at that.

Regards,
thepudds
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-fuzzing-proposal+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages