wasm compile fuzzer questions

[Sam Parker-Haynes]

Hi,

I'm trying to find a codegen bug in aarch64, so I've been looking at the wasm_compile_fuzzer in the hope that it can help me. I have a number of questions about the current behaviour of the fuzz target. (sorry in advance for the list!)

1) What set of commands is best to use? I've noticed on the default setting a single, constant, instruction is generated and I'm not sure how useful that is. I've currently using -len_control=10 to get to the, hopefully, juicy stuff quickly.

2) Viewing the generated modules is difficult. I'm using `DumpModule` to output any valid module and there seems to be two error types that prevent my available tools from working. A common output from the WABT tools is: `error: unexpected type form (got -0x30)`. wasm-objdump tries harder but then often falls over with `error: expected valid local type`. I'm using the latest version of WABT, does anyone know what type(s) the fuzz target generates that could cause this issue?

3) For the modules that I have successfully viewed, I've often noticed long chains of the same operation, i32.eqz being a very popular one. Is there any explanation for this? In general, I still haven't got my head around how the input from libfuzzer is used to generate the module...

4) Is there any memory attached to the instance when it runs? And if there is, there doesn't seem to be an attempt to ensure addresses are in range. So, do most of the memory operations just crash the program? The differential testing seems to only test that the return of `main` is equal, but what about the contents of memory?

Thanks!

Sam

[Andreas Haas]

On Wed, Jul 5, 2023 at 3:21 PM Sam Parker-Haynes <sam.p...@arm.com> wrote:

Hi,

I'm trying to find a codegen bug in aarch64, so I've been looking at the wasm_compile_fuzzer in the hope that it can help me. I have a number of questions about the current behaviour of the fuzz target. (sorry in advance for the list!)

1) What set of commands is best to use? I've noticed on the default setting a single, constant, instruction is generated and I'm not sure how useful that is. I've currently using -len_control=10 to get to the, hopefully, juicy stuff quickly.

Typically I use `v8_wasm_compile_fuzzer` without parameters. Libfuzzer should automatically adjust the input size. 

2) Viewing the generated modules is difficult. I'm using `DumpModule` to output any valid module and there seems to be two error types that prevent my available tools from working. A common output from the WABT tools is: `error: unexpected type form (got -0x30)`. wasm-objdump tries harder but then often falls over with `error: expected valid local type`. I'm using the latest version of WABT, does anyone know what type(s) the fuzz target generates that could cause this issue?

If you compile `v8_simple_wasm_compile_fuzzer` in V8, then you can run it with `v8_simple_wasm_compile_fuzzer --wasm-fuzzer-gen-test`, which will generate an mjsunit test that is equivalent to the generated test.

 

3) For the modules that I have successfully viewed, I've often noticed long chains of the same operation, i32.eqz being a very popular one. Is there any explanation for this? In general, I still haven't got my head around how the input from libfuzzer is used to generate the module...

The fuzzer builds the module top down. It generates the outermost instruction randomly, and then generates inputs for the instruction randomly, but of the correct type. For those inputs it also generates the inputs of the correct types, and so on. Maybe `i32.eqz` gets generated with a `0`, which makes it more likely by the fuzzer because the fuzzer generates long strings of `0`?

 

4) Is there any memory attached to the instance when it runs? And if there is, there doesn't seem to be an attempt to ensure addresses are in range. So, do most of the memory operations just crash the program? The differential testing seems to only test that the return of `main` is equal, but what about the contents of memory?

A memory is attached, I think there is some logic to prefer useful memory addresses, but I don't know the details/current state. The fuzzer could also realize that low memory indices test better, and prefer low indices.

 

Thanks!

Sam

--
--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/2ad6b36e-9a5b-4dba-9d76-abb2e95d9f58n%40googlegroups.com.

--

Andreas Haas

Software Engineer

ah...@google.com

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

    

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[Jakob Kummerow]

For disassembling binary Wasm modules, I recommend:

- Binaryen's wasm-dis (especially if you like "nested s-expression" syntax). See https://github.com/WebAssembly/binaryen#building.

- V8's own Wasm Module Inspector, compile it with `tools/dev/gm.py x64.release wami` and see `out/x64.release/wami --help`. It reuses V8's production decoder, so it's guaranteed to provide accurate insight into what V8 thinks those bits mean. I use it, among other things, to investigate cases where it's not immediately clear whether the module is invalid or V8 is decoding it incorrectly.

[Matthias Liedtke]

To extend Andreas' comments a bit:

2) If the mjsunit format isn't readable enough (immediate support is somewhat limited on the test generator), you can also decode the module from --dump-wasm-module using wami, e.g. wami --full-wat the-wasm-file.wasm. Wami is part of the V8 repository and also gets updated with new wasm features in v8.
3) If the fuzzer "runs out" of input bytes, it will just fill with 0s. So every randomized decision will take the 0-branch. This also helps to prevent endless recursions, so it's by design. 
As a first experiment (assuming a linux environment) you could try `head -c 255 /dev/urandom > fuzzer_input ; ./out/x64.optdebug/v8_simple_wasm_compile_fuzzer fuzzer_input --wasm-fuzzer-gen-test > test.js`. With 255 random bytes it should be able to generate meaningful instructions. Some bytes will already be used for static decisions or for decisions that are not part of the function body (types, globals, signatures, ...), so if the random input should have a certain size to create meaningful fuzzing examples. Note that the max input size is 512 bytes, so the input shouldn't be larger than that either!

4) There might be some possible gains in making the fuzzer smarter on generating better addresses etc., so that it doesn't trap that often. It is a bit more difficult however because the fuzzer will generate other instructions to generate an offset (e.g. a local.get) and it is hard to predict which value range is generated by that. One could possibly generate a few more instructions (a modulo with the memory size?) with a certain probability to reduce the chance of oob memory accesses.

Best regards,
Matthias

On Wed, Jul 5, 2023 at 3:52 PM 'Andreas Haas' via v8-dev <v8-...@googlegroups.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/CAELSTvcdk2Kng_R9u4qn-eyEtr4if66k0zwQYSJ9SxXVCaNbmg%40mail.gmail.com.

--

Matthias Liedtke

Software Engineer

mlie...@google.com

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls Sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde. 

[Sam Parker-Haynes]

Thanks all, I will try those other disassemblers. I wanted to use --wasm-fuzzer-gen-test but it seems I can't do that when running the proper fuzzer instance - and I just want to dump out a load of generated cases for inspection.

Could you point me to where the memory is added? I'm still unfamiliar with the API and all I can find is this which, naively, suggests there isn't any and I'd like to inspect it.

I like the idea of using modulo memory size for memory arguments! From what I've seen memory arguments are generated just like any other I32 input, except that specific effort is made around alignment. I'd like to ensure that more memory accesses are in bounds to, hopefully, make generated programs more useful.

[Clemens Backes]

On Wed, Jul 5, 2023 at 4:20 PM Sam Parker-Haynes <sam.p...@arm.com> wrote:

Thanks all, I will try those other disassemblers. I wanted to use --wasm-fuzzer-gen-test but it seems I can't do that when running the proper fuzzer instance - and I just want to dump out a load of generated cases for inspection.

Could you point me to where the memory is added? I'm still unfamiliar with the API and all I can find is this which, naively, suggests there isn't any and I'd like to inspect it.

The memory is added implicitly by the `WasmModuleBuilder`: https://source.chromium.org/chromium/chromium/src/+/refs/heads/main:v8/src/wasm/wasm-module-builder.cc;l=739;drc=e510fa298843c8ff43b538742e4fa16306da265f

Ensuring that more memory accesses are in-bounds sounds like a good strategy to make the fuzzer execute more code before trapping.

Regarding comparing the memory contents of the reference run and the regular run, I am not sure if that's worth it as it would probably make the fuzzer significantly slower. Changes in memory content can also be observed in other ways, e.g. by simply reading back the memory content and returning it from the main function. So the fuzzer has a chance to find differences in memory contents even without explicitly checking for that. That only works if the execution does return regularly instead of trapping. It might be worth checking how often we actually trap instead of returning. If we trap in most runs, then we should indeed do something about this.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/6ecf0e1f-6040-4264-9fee-4409574f9895n%40googlegroups.com.

--

Clemens Backes

Software Engineer

clem...@google.com

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian   

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

[Sam Parker-Haynes]

Ah, thanks! Using wami, I now also see a memory in the module.

From my limited viewing of the code, the huge offsets that are generated for the memory operations look suspicious. I will try to get some concrete numbers for successful accesses vs traps.

For analysing the memory afterwards, I was hoping that memcmp would be fast enough... With the current setup, of only checking the return from main, it would suggest there could be many paths that don't affect the result.

But now I'm wondering in how many cases the target would generate a store - if we're generating the inputs, how many instructions require a 'void' input? 

[Clemens Backes]

On Wed, Jul 5, 2023 at 5:17 PM Sam Parker-Haynes <sam.p...@arm.com> wrote:

Ah, thanks! Using wami, I now also see a memory in the module.

From my limited viewing of the code, the huge offsets that are generated for the memory operations look suspicious. I will try to get some concrete numbers for successful accesses vs traps.

For analysing the memory afterwards, I was hoping that memcmp would be fast enough... With the current setup, of only checking the return from main, it would suggest there could be many paths that don't affect the result.

Ack, maybe it's fast enough. Feel free to test and send over a CL. Note that we limit the memory size to 32 pages, so the maximum memory to check is 2MB large.

 

But now I'm wondering in how many cases the target would generate a store - if we're generating the inputs, how many instructions require a 'void' input? 

Generating a `void` should not be that uncommon. It's the initial thing we generate for functions that do not return anything, and when generating any other type we always have the option to generate one or several instructions producing `void` before or after the instruction generating the requested type. Look for uses of `WasmGenerator::sequence`: https://source.chromium.org/chromium/chromium/src/+/refs/heads/main:v8/test/fuzzer/wasm-compile.cc;l=972;drc=e510fa298843c8ff43b538742e4fa16306da265f

 

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/6bc78704-adce-4c7d-9842-2ee2e9344cf5n%40googlegroups.com.