Path location for ClusterFuzz tests

[sch...@chromium.org]

Where should ClusterFuzz tests live in the source tree?

ClusterFuzz finds bugs by analyzing crashes of procedurally generated random tests, e.g. randomized layout tests. Some fuzzers already exist but are closed source. New fuzzers should be open source (and old ones migrated) to keep with the open source nature of the chromium project and to support external security researchers.

In conversation with Abhishek about adding ClusterFuzz tests to the chromium repo he requested they be next to the code being tested. e.g. Web Bluetooth tests in the .../modules/bluetooth/web_bluetooth_fuzzer directory. Here's a patch in progress for that.

Kentaro suggested Tools/Scripts on that patch, raising good points about the previous tests in WebKit in a conversation:

current convention is:
1) put C++ tests next to the source files
2) put layout tests to LayoutTests/
3) put other scripted test things to Tools/Scripts/.

Personally, I think multiple parallel directory trees should be avoided. I believe the same logic for unit and browser tests should apply, even though ClusterFuzz tests use another language.

I'd like consensus here, though, because ideally we establish a pattern and encourage more Web Platform APIs to author their ClusterFuzz tests in open source.

[dan...@chromium.org]

On Wed, Jul 20, 2016 at 1:02 PM, <sch...@chromium.org> wrote:

Where should ClusterFuzz tests live in the source tree?

ClusterFuzz finds bugs by analyzing crashes of procedurally generated random tests, e.g. randomized layout tests. Some fuzzers already exist but are closed source. New fuzzers should be open source (and old ones migrated) to keep with the open source nature of the chromium project and to support external security researchers.

In conversation with Abhishek about adding ClusterFuzz tests to the chromium repo he requested they be next to the code being tested. e.g. Web Bluetooth tests in the .../modules/bluetooth/web_bluetooth_fuzzer directory. Here's a patch in progress for that.

Kentaro suggested Tools/Scripts on that patch, raising good points about the previous tests in WebKit in a conversation:

current convention is:
1) put C++ tests next to the source files
2) put layout tests to LayoutTests/
3) put other scripted test things to Tools/Scripts/.

Personally, I think multiple parallel directory trees should be avoided. I believe the same logic for unit and browser tests should apply, even though ClusterFuzz tests use another language.

The scope of layout tests and unit tests feels different to me. Unit tests (incl browser tests) are usually written to test a single class, and then live beside it. Some unit tests are more like integration tests and then they don't have a class to live beside (I'm thinking of layer_tree_host_unittest_xyz.cc tests in src/cc/ as an example, so they just kinda live somewhere. LayoutTests don't appear to be strongly tied to any given class and just test the overall system, as integration tests.

So I guess I'm wondering what do clusterfuzz tests look more like in this regard?

 

I'd like consensus here, though, because ideally we establish a pattern and encourage more Web Platform APIs to author their ClusterFuzz tests in open source.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/e0e86452-97e5-41d0-93a4-8b8957d83558%40chromium.org.

[Abhishek Arya]

On Wed, Jul 20, 2016 at 1:02 PM, <sch...@chromium.org> wrote:

Where should ClusterFuzz tests live in the source tree?

ClusterFuzz finds bugs by analyzing crashes of procedurally generated random tests, e.g. randomized layout tests. Some fuzzers already exist but are closed source. New fuzzers should be open source (and old ones migrated) to keep with the open source nature of the chromium project and to support external security researchers.

We can't just open source everything without understanding the implications of security vulnerabilities. We are moving more towards the open-source path, but this needs more thought as well.

 

In conversation with Abhishek about adding ClusterFuzz tests to the chromium repo he requested they be next to the code being tested. e.g. Web Bluetooth tests in the .../modules/bluetooth/web_bluetooth_fuzzer directory. Here's a patch in progress for that.

Kentaro suggested Tools/Scripts on that patch, raising good points about the previous tests in WebKit in a conversation:

current convention is:
1) put C++ tests next to the source files

This seems fine, currently all libFuzzer based fuzzers reside next to source, just like unittests.

 

2) put layout tests to LayoutTests/

This also makes sense, but in this case, this is more like an input seed for the fuzzer. 

 

3) put other scripted test things to Tools/Scripts/.

I don't understand why we should put fuzzer scripts there. Right now, I don't see any component scripts in there other than layout test controller helper scripts. 

[Vincent Scheib]

On Wed, Jul 20, 2016 at 1:10 PM, <dan...@chromium.org> wrote:

On Wed, Jul 20, 2016 at 1:02 PM, <sch...@chromium.org> wrote:

Where should ClusterFuzz tests live in the source tree?

ClusterFuzz finds bugs by analyzing crashes of procedurally generated random tests, e.g. randomized layout tests. Some fuzzers already exist but are closed source. New fuzzers should be open source (and old ones migrated) to keep with the open source nature of the chromium project and to support external security researchers.

In conversation with Abhishek about adding ClusterFuzz tests to the chromium repo he requested they be next to the code being tested. e.g. Web Bluetooth tests in the .../modules/bluetooth/web_bluetooth_fuzzer directory. Here's a patch in progress for that.

Kentaro suggested Tools/Scripts on that patch, raising good points about the previous tests in WebKit in a conversation:

current convention is:
1) put C++ tests next to the source files
2) put layout tests to LayoutTests/
3) put other scripted test things to Tools/Scripts/.

Personally, I think multiple parallel directory trees should be avoided. I believe the same logic for unit and browser tests should apply, even though ClusterFuzz tests use another language.

The scope of layout tests and unit tests feels different to me. Unit tests (incl browser tests) are usually written to test a single class, and then live beside it. Some unit tests are more like integration tests and then they don't have a class to live beside (I'm thinking of layer_tree_host_unittest_xyz.cc tests in src/cc/ as an example, so they just kinda live somewhere. LayoutTests don't appear to be strongly tied to any given class and just test the overall system, as integration tests.

So I guess I'm wondering what do clusterfuzz tests look more like in this regard?

For example this Web Bluetooth fuzzer will test all the .idl exposed in modules/bluetooth. It does not do it one idl file at a time, so it is 'spanning' a bit. But it won't test anything outside of the bluetooth module in a concerted way.

So, it's scope is essentially the same as modules/bluetooth -- it should cover all of it, and not intentionally more.

[dan...@chromium.org]

On Wed, Jul 20, 2016 at 1:14 PM, Vincent Scheib <sch...@chromium.org> wrote:

On Wed, Jul 20, 2016 at 1:10 PM, <dan...@chromium.org> wrote:

On Wed, Jul 20, 2016 at 1:02 PM, <sch...@chromium.org> wrote:

Where should ClusterFuzz tests live in the source tree?

ClusterFuzz finds bugs by analyzing crashes of procedurally generated random tests, e.g. randomized layout tests. Some fuzzers already exist but are closed source. New fuzzers should be open source (and old ones migrated) to keep with the open source nature of the chromium project and to support external security researchers.

In conversation with Abhishek about adding ClusterFuzz tests to the chromium repo he requested they be next to the code being tested. e.g. Web Bluetooth tests in the .../modules/bluetooth/web_bluetooth_fuzzer directory. Here's a patch in progress for that.

Kentaro suggested Tools/Scripts on that patch, raising good points about the previous tests in WebKit in a conversation:

current convention is:
1) put C++ tests next to the source files
2) put layout tests to LayoutTests/
3) put other scripted test things to Tools/Scripts/.

Personally, I think multiple parallel directory trees should be avoided. I believe the same logic for unit and browser tests should apply, even though ClusterFuzz tests use another language.

The scope of layout tests and unit tests feels different to me. Unit tests (incl browser tests) are usually written to test a single class, and then live beside it. Some unit tests are more like integration tests and then they don't have a class to live beside (I'm thinking of layer_tree_host_unittest_xyz.cc tests in src/cc/ as an example, so they just kinda live somewhere. LayoutTests don't appear to be strongly tied to any given class and just test the overall system, as integration tests.

So I guess I'm wondering what do clusterfuzz tests look more like in this regard?

For example this Web Bluetooth fuzzer will test all the .idl exposed in modules/bluetooth. It does not do it one idl file at a time, so it is 'spanning' a bit. But it won't test anything outside of the bluetooth module in a concerted way.

So, it's scope is essentially the same as modules/bluetooth -- it should cover all of it, and not intentionally more.

Thx, IMHO it should live where we would make it live if it was written in C++, which means probably the modules/bluetooth directory, in a file named something_something_fuzztest.xyz or the like.

[Kentaro Hara]

OK, then I'm fine with putting the tests in modules/bluetooth/.

I'd prefer putting them under modules/bluetooth/testing/ though. Blink has conventionally put testing files in xxx/testing/.

--

Kentaro Hara, Tokyo, Japan