Failed to build fuchsia image with syzkaller

123 views
Skip to first unread message

Behouba Manassé Kouamé

unread,
May 7, 2022, 7:21:03 PM5/7/22
to discuss

Hello,

I am trying to experiment fuzzing fuchsia with syzkaller by following the documentation provided here: https://github.com/google/syzkaller/blob/master/docs/fuchsia/README.md.

 I get the following output when trying to build a fuchsia image:

 

fx --dir "out/arm64" set core.arm64   --with-base "//bundles:tools"   --with-base "//src/testing/fuzzing/syzkaller"   --args=syzkaller_dir='"/home/behouba/Desktop/kos-fuzzing/syzkaller"'

WARNING: Please opt in or out of fx metrics collection.

You will receive this warning until an option is selected.

To check what data we collect, run `fx metrics`

To opt in or out, run `fx metrics <enable|disable>

 

ERROR at //build/go/go_library.gni:43:3 (//build/toolchain:host_x64): Assertion failed.

  assert(defined(invoker.sources), "sources is required for go_library")

  ^-----

sources is required for go_library

See //src/testing/fuzzing/syzkaller/BUILD.gn:107:3: whence it was called.

  go_library("syzkaller-go") {

  ^---------------------------

See //src/testing/fuzzing/syzkaller/BUILD.gn:86:5: which caused the file to be included.

    ":run-sysgen($host_toolchain)",

    ^-----------------------------

ERROR: error running gn gen: exit status 1

 

Can someone help please )


Shai Barack

unread,
May 7, 2022, 8:17:32 PM5/7/22
to Behouba Manassé Kouamé, discuss, Marco Vanotti, Chris Palmer
I think fxrev.dev/611320 broke this workflow back in November, and we never noticed because it's not tested in our presubmit. Thank you Behouba for bringing this to our attention.

I prepared a possible workaround for you. In your Fuchsia checkout (where you typed `fx ...`), please type the following:
git fetch https://fuchsia.googlesource.com/fuchsia refs/changes/65/677365/1 && git cherry-pick FETCH_HEAD
Afterwards, try the `fx ...` command again and it should get past the error that you reported.

+Marco Vanotti +Chris Palmer could help update the docs on syzkaller github or the build targets on Fuchsia to fix this workflow. I suggest also finding a way to test this in presubmit if feasible, so the workflow doesn't break again.

--
All posts must follow the Fuchsia Code of Conduct https://fuchsia.dev/fuchsia-src/CODE_OF_CONDUCT or may be removed.
---
You received this message because you are subscribed to the Google Groups "discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@fuchsia.dev.
To view this discussion on the web visit https://groups.google.com/a/fuchsia.dev/d/msgid/discuss/b85d5c70-0b71-4b5e-af19-c24a62b47110n%40fuchsia.dev.

Behouba Manassé Kouamé

unread,
May 8, 2022, 3:36:07 AM5/8/22
to discuss, Shai Barack, discuss, Marco Vanotti, Chris Palmer, Behouba Manassé Kouamé
Thank you Shai,
I tried the workaround and I am now getting a new error:

fx --dir "out/arm64" set core.arm64   --with-base "//bundles:tools"   --with-base "//src/testing/fuzzing/syzkaller"   --args=syzkaller_dir='"/home/behouba/Desktop/kos-fuzzing/syzkaller"'

WARNING: Please opt in or out of fx metrics collection.
You will receive this warning until an option is selected.
To check what data we collect, run `fx metrics`
To opt in or out, run `fx metrics <enable|disable>

ERROR Unresolved dependencies.
//src/media/drivers/amlogic_decoder:amlogic_decoder-package.pkg.pm(//build/toolchain/fuchsia:arm64)
  needs //src/media/drivers/amlogic_decoder:amlogic_decoder_firmware(//build/toolchain/fuchsia:arm64)
//src/media/drivers/amlogic_decoder:amlogic_decoder-package.pkg_manifest_dist(//build/toolchain/fuchsia:arm64)
  needs //src/media/drivers/amlogic_decoder:amlogic_decoder_firmware(//build/toolchain/fuchsia:arm64)
//src/media/drivers/amlogic_decoder:amlogic_decoder-package.pkg_manifest_dist__hermetic_inputs(//build/toolchain/fuchsia:arm64)
  needs //src/media/drivers/amlogic_decoder:amlogic_decoder_firmware(//build/toolchain/fuchsia:arm64)
//src/media/drivers/amlogic_decoder:amlogic_decoder-package.pkg_manifest_dist_generate(//build/toolchain/fuchsia:arm64)
  needs //src/media/drivers/amlogic_decoder:amlogic_decoder_firmware(//build/toolchain/fuchsia:arm64)


ERROR: error running gn gen: exit status 1

Shai Barack

unread,
May 8, 2022, 11:35:27 AM5/8/22
to Behouba Manassé Kouamé, discuss, Marco Vanotti, Chris Palmer
I couldn't reproduce your error.
Please try checking out recent sources by running `jiri update`.
If this doesn't help, please enter `fx status` and quote the results here.

Suraj Malhotra

unread,
May 8, 2022, 12:24:32 PM5/8/22
to Shai Barack, Behouba Manassé Kouamé, discuss, Marco Vanotti, Chris Palmer
Try commenting out this line: https://cs.opensource.google/fuchsia/fuchsia/+/main:src/media/drivers/amlogic_decoder/BUILD.gn;l=295

I believe that it should added as a dep if _has_decoder_firmware = true, but someone forgot to add that clause.

Behouba Manassé Kouamé

unread,
May 8, 2022, 6:29:25 PM5/8/22
to discuss, Suraj Malhotra, Behouba Manassé Kouamé, discuss, Marco Vanotti, Chris Palmer, Shai Barack
Thank you again, 
After doing what you suggested, the command was successful, but when I run the build command, I get this error when the build is about to complete:

fx clean-build
...
In file included from /home/behouba/Desktop/kos-fuzzing/syzkaller/executor/executor.cc:193:
In file included from /home/behouba/Desktop/kos-fuzzing/syzkaller/executor/common.h:467:
/home/behouba/Desktop/kos-fuzzing/syzkaller/executor/common_fuchsia.h:262:2: error: use of undeclared identifier 'zx_clock_get'; did you mean 'zx_clock_read'?
        zx_clock_get(ZX_CLOCK_MONOTONIC, &now);
        ^~~~~~~~~~~~
        zx_clock_read
gen/include/zircon/syscalls/internal/cdecls.inc:112:1: note: 'zx_clock_read' declared here
_ZX_SYSCALL_DECL(clock_read, zx_status_t, /* no attributes */, 2,
^
gen/zircon/public/sysroot/cpp/include/zircon/syscalls.h:19:31: note: expanded from macro '_ZX_SYSCALL_DECL'
  extern attrs __LEAF_FN type zx_##name prototype;                     \
                              ^
<scratch space>:68:1: note: expanded from here
zx_clock_read
^
In file included from /home/behouba/Desktop/kos-fuzzing/syzkaller/executor/executor.cc:390:
host_x64/gen/src/testing/fuzzing/syzkaller/sysgen/executor/syscalls.h:2496:52: error: use of undeclared identifier 'zx_clock_get'
    {"zx_clock_get", 0, {0, 0, 0, 1, }, (syscall_t)zx_clock_get},
                                                   ^
host_x64/gen/src/testing/fuzzing/syzkaller/sysgen/executor/syscalls.h:2565:46: error: use of undeclared identifier 'zx_socket_shutdown'
    {"zx_socket_shutdown", 0, {}, (syscall_t)zx_socket_shutdown},
                                             ^
host_x64/gen/src/testing/fuzzing/syzkaller/sysgen/executor/syscalls.h:2586:42: error: use of undeclared identifier 'zx_vcpu_resume'
    {"zx_vcpu_resume", 0, {}, (syscall_t)zx_vcpu_resume},
                                         ^
/home/behouba/Desktop/kos-fuzzing/syzkaller/executor/executor.cc:852:19: error: invalid application of 'sizeof' to an incomplete type 'const call_t[]'
                if (call_num >= ARRAY_SIZE(syscalls))
                                ^~~~~~~~~~~~~~~~~~~~
/home/behouba/Desktop/kos-fuzzing/syzkaller/executor/executor.cc:44:30: note: expanded from macro 'ARRAY_SIZE'
#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
                             ^~~
5 errors generated.
[37685/59173] CXX obj/src/sys/test_runners/legacy_test/legacy_test_runner.suite.cc.o
ninja: build stopped: subcommand failed.
Hint: run `fx build` with the option `--log LOGFILE` to generate a debug log if you are reporting a bug.


Maybe some issue with headers files.

Shai Barack

unread,
May 8, 2022, 6:43:25 PM5/8/22
to Behouba Manassé Kouamé, discuss, Suraj Malhotra, Marco Vanotti, Chris Palmer
It looks like the Fuchsia headers for syzkaller broke with a change that rolled out in 2021 if I recall correctly. +Chris Palmer might know what to do.

Shai Barack

unread,
May 9, 2022, 7:08:06 PM5/9/22
to Aaron Green, Chris Palmer, Cameron Finucane, Ange Albertini, Behouba Manassé Kouamé, discuss, Suraj Malhotra, Marco Vanotti
Can we update the upstream syzkaller Fuchsia docs to indicate that this is known not to work? 

On Mon, May 9, 2022, 4:06 PM Aaron Green <aaron...@google.com> wrote:
This is correct; a resourcing decision was made more than a year ago with the consequence that the increasing costs of syzkaller couldn't be maintained. The situation is starting to look a little better, and I'm optimistic we'll get back to system call fuzzing soon. In the meantime, the short term action is probably to make it clearer that syzkaller is deprecated and its replacement is not ready yet.

Aaron

On Mon, May 9, 2022 at 3:29 PM Chris Palmer <pal...@google.com> wrote:
+Cameron Finucane and +Aaron Green and +Ange Albertini — in general, as I understand it, syzkaller is known to not work anymore. (Alas!) We know this is a problem and we are working on it, but the solution will probably be to have a new system call fuzzing framework; syzkaller was not working for us for a few reasons. Behouba, if you have ideas for what you'd like to see in a system call fuzzing framework, we'd love to hear them. :)

Aaron Green

unread,
May 9, 2022, 7:20:15 PM5/9/22
to Chris Palmer, Shai Barack, Cameron Finucane, Ange Albertini, Behouba Manassé Kouamé, discuss, Suraj Malhotra, Marco Vanotti
This is correct; a resourcing decision was made more than a year ago with the consequence that the increasing costs of syzkaller couldn't be maintained. The situation is starting to look a little better, and I'm optimistic we'll get back to system call fuzzing soon. In the meantime, the short term action is probably to make it clearer that syzkaller is deprecated and its replacement is not ready yet.

Aaron

On Mon, May 9, 2022 at 3:29 PM Chris Palmer <pal...@google.com> wrote:
+Cameron Finucane and +Aaron Green and +Ange Albertini — in general, as I understand it, syzkaller is known to not work anymore. (Alas!) We know this is a problem and we are working on it, but the solution will probably be to have a new system call fuzzing framework; syzkaller was not working for us for a few reasons. Behouba, if you have ideas for what you'd like to see in a system call fuzzing framework, we'd love to hear them. :)

On Sun, May 8, 2022 at 3:43 PM Shai Barack <sha...@google.com> wrote:

Aaron Green

unread,
May 9, 2022, 7:20:15 PM5/9/22
to Shai Barack, Chris Palmer, Cameron Finucane, Ange Albertini, Behouba Manassé Kouamé, discuss, Suraj Malhotra, Marco Vanotti
We definitely should and will.

Chris Palmer

unread,
May 9, 2022, 7:20:15 PM5/9/22
to Shai Barack, Cameron Finucane, Aaron Green, Ange Albertini, Behouba Manassé Kouamé, discuss, Suraj Malhotra, Marco Vanotti
+Cameron Finucane and +Aaron Green and +Ange Albertini — in general, as I understand it, syzkaller is known to not work anymore. (Alas!) We know this is a problem and we are working on it, but the solution will probably be to have a new system call fuzzing framework; syzkaller was not working for us for a few reasons. Behouba, if you have ideas for what you'd like to see in a system call fuzzing framework, we'd love to hear them. :)

On Sun, May 8, 2022 at 3:43 PM Shai Barack <sha...@google.com> wrote:

Behouba Manassé Kouamé

unread,
May 10, 2022, 6:00:53 AM5/10/22
to discuss, Chris Palmer, Behouba Manassé Kouamé, discuss, Suraj Malhotra, Marco Vanotti, Shai Barack, e...@google.com, Aaron Green, Ange Albertini
Well, as part of a university project, we are trying to find a way to use syzkaller to fuzz a kernel similar to fuchsia. A kernel without Golang runtime and it is a bit challenging given our lack of experience in the domain. I am was trying to understand the approach used to fuzz fuchsia with syzkaller.)

Aaron Green

unread,
May 10, 2022, 5:33:55 PM5/10/22
to Pedro Falcato, discuss, Shai Barack, e...@google.com, Ange Albertini, Behouba Manassé Kouamé, Suraj Malhotra, Marco Vanotti, Chris Palmer
Sure! Although I'll start with one clarification:

why exactly did you end up abandoning syzkaller?

While it's definitely on pause, I want to be clear that we haven't abandoned system call fuzzing forever! It's high on our list of areas we want to spend more time on, albeit not necessarily with an unmodified out-of-the-box syzkaller. We paused syzakller work for several reasons a major one being the decision by the project leadership to not formally support Go (https://fuchsia.dev/fuchsia-src/contribute/governance/policy/programming_languages#go). This meant, for example, the Fuchsia toolchain team would not provide a Fuchsia-ready Go runtime or support us if/when we encountered issues.

in response to this, we spent a fair bit of effort working to implement a "host" mode, in which the Go portions ran on a Linux platform and then transferred the ELF objects to a Fuchsia platform to be executed against the Zircon kernel. This worked, albeit with mediocre performance, but runs contrary to multiple efforts to prevent unsigned, arbitrary code from executing within Fuchsia. It's possible to create a special, one-off instance that avoids a lot of these aspects of Fuchsia, like the Component Framework, but much of our build and test infra obviously expects those to be present. Similarly, this approach depends on remote shell access via SSH that is also increasingly restricted (in favor of, e.g., FIDL over overnet).

We didn't have a lot of bandwidth to create and maintain special runtimes and infra. In fact, we were struggling even to keep syzbot building and running Fuchsia reliably. On top of all that, there wasn't a solid kernel coverage story for Zircon, so system call fuzzing couldn't be as coverage guided as we wanted. For this reason, we decided to focus what limited resources we had on other taks first, e.g. implementing kcov for Zircon. 

It just got to be too many costs related to language runtimes, infrastructure, missing kernel features, and OS architecture... i.e. a lot of things that are easy to take for granted on Linux, but need a lot of effort to make work on this OS.

Hope that helps,
Aaron


On Tue, May 10, 2022 at 2:01 PM Pedro Falcato <pedro....@gmail.com> wrote:
I'm curious, why exactly did you end up abandoning syzkaller? I know you mentioned maintenance costs, but most projects seem to roll with it just fine.

Tim Newsham

unread,
May 11, 2022, 12:03:22 AM5/11/22
to Aaron Green, Pedro Falcato, discuss, Shai Barack, e...@google.com, Ange Albertini, Behouba Manassé Kouamé, Suraj Malhotra, Marco Vanotti, Chris Palmer
On Tue, May 10, 2022 at 11:33 AM 'Aaron Green' via discuss <dis...@fuchsia.dev> wrote:
 guided as we wanted. For this reason, we decided to focus what limited resources we had on other taks first, e.g. implementing kcov for Zircon. 

Is kcov in kernel now?

Hope that helps,
Aaron

--
Tim Newsham

Pedro Falcato

unread,
May 11, 2022, 12:03:47 AM5/11/22
to discuss, Aaron Green, Shai Barack, e...@google.com, Ange Albertini, Behouba Manassé Kouamé, discuss, Suraj Malhotra, Marco Vanotti, Chris Palmer
I'm curious, why exactly did you end up abandoning syzkaller? I know you mentioned maintenance costs, but most projects seem to roll with it just fine.

On Tuesday, May 10, 2022 at 12:20:15 AM UTC+1 Aaron Green wrote:
Reply all
Reply to author
Forward
0 new messages