Re: [syz] USB fuzzing

[Dmitry Vyukov]

On Wed, May 20, 2020 at 8:25 AM Maxime Villard <m...@m00nbsd.net> wrote:
>
> CC'ing dvyukov, and asking: does syzbot perform a NetBSD distribution build each
> time it wants to build the executor? It looks like it has an old build that
> doesn't have the vhci.h header. How can we refresh that?
>
> Thanks
>
>
> Le 20/05/2020 à 08:11, Maxime Villard a écrit :
> > Looks like the syzkaller build is failing:
> >
> > https://syzkaller.appspot.com/text?tag=CrashLog&x=128b424a100000
> >
> > executor/common_usb_netbsd.h:10:10: fatal error: dev/usb/vhci.h: No such file or directory
> > #include <dev/usb/vhci.h>
> > ^~~~~~~~~~~~~~~~
> >
> > I have re-tested just now in my local setup, it works. The vhci header is
> > there.
> >
> > Is there stale data on the instances? Can we force a rebuild of the
> > distribution to make sure the header is installed?

+syzkaller mailing list

Hi,

The host toolchain is updated manually. I've updated it now to
8b99e2a6c01eba58bb960ddc6e7b05f530bc2d48, the new version is available
at:
https://storage.googleapis.com/syzkaller/netbsd-dest.tar.gz

executor build succeeded now.
Though, 2 netbsd builds failed:

May 20 10:27:46 ci2 syz-ci[25585]: 2020/05/20 10:27:46
ci2-netbsd-kmsan: kernel build failed:
sys/ufs/ffs/ffs_extattr.c:124:16: error: redefinition of typedef
'__accmode_t' is a C11 feature [-Werror,-Wtypedef-redefinition]
May 20 10:27:46 ci2 syz-ci[25585]: ERROR: Failed to make all in
"sys/arch/amd64/compile/obj/GENERIC_SYZKALLER"

May 20 10:35:38 ci2 syz-ci[25585]: 2020/05/20 10:35:38
ci2-netbsd-kubsan: kernel build failed: sys/uvm/uvm_aobj.c:1015:14:
error: 'pageidx' may be used uninitialized in this function
[-Werror=maybe-uninitialized]
May 20 10:35:38 ci2 syz-ci[25585]: sys/uvm/uvm_aobj.c:997:12: error:
'swslot' may be used uninitialized in this function
[-Werror=maybe-uninitialized]
May 20 10:35:38 ci2 syz-ci[25585]: ERROR: Failed to make all in
"sys/arch/amd64/compile/obj/GENERIC_SYZKALLER"

[Dmitry Vyukov]

On Wed, May 20, 2020 at 8:37 PM Maxime Villard <m...@m00nbsd.net> wrote:
>
> Le 20/05/2020 à 19:41, Dmitry Vyukov a écrit :
> > On Wed, May 20, 2020 at 7:18 PM Maxime Villard <m...@m00nbsd.net> wrote:
> >>
> >> > executor build succeeded now.
> >>
> >> Thank you. I would like to submit an updated vusb.txt but am unable to get
> >> "make extract" to work.
> >>
> >> $ make extract TARGETOS=netbsd SOURCEDIR=/netbsd-syz/src/
> >> GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=4afdfa205b55633e7eb9db03a9d099d7aa324801 -X 'github.com/google/syzkaller/prog.gitRevisionDate=Wed May 20 15:39:22 2020 +0200'" -o bin/syz-extract ./sys/syz-extract
> >> bin/syz-extract -build -os=netbsd -sourcedir=/netbsd-syz/src/
> >> generating netbsd/amd64...
> >> fs.txt: input file sys/netbsd/fs.txt is missing
> >> ipc.txt: input file sys/netbsd/ipc.txt is missing
> >> lwp.txt: input file sys/netbsd/lwp.txt is missing
> >> mm.txt: input file sys/netbsd/mm.txt is missing
> >> socket.txt: input file sys/netbsd/socket.txt is missing
> >> socket_inet.txt: input file sys/netbsd/socket_inet.txt is missing
> >> socket_inet6.txt: input file sys/netbsd/socket_inet6.txt is missing
> >> socket_unix.txt: input file sys/netbsd/socket_unix.txt is missing
> >> sys.txt: input file sys/netbsd/sys.txt is missing
> >> vnet.txt: input file sys/netbsd/vnet.txt is missing
> >> vusb.txt: input file sys/netbsd/vusb.txt is missing
> >> make: *** [Makefile:204: extract] Error 1
> >>
> >> The input files are _not_ missing. What am I supposed to do here?
> >>
> >> Thanks
> >
> > Turns out I broke it recently.
> > Please sync to head to pick up:
> > https://github.com/google/syzkaller/commit/204f4fde068172170c5bc22bcffd60753d583c35
> > it should fix it.
>
> Thanks, it indeed fixed it. I'll push the new vusb.txt once the current PR is
> closed.
>
> We will have to add the "vhci" pseudo-device in the instances. The commands
> to create it are:
>
> # cd /dev
> # ./MAKEDEV vhci
>
> I suspect this has to be done manually?

I hope you don't mean that for every VM booted by the fuzzer after
each crash, the process needs to halt and wait for a human to ssh,
type some commands and resume the process :)

Here we do some image setup:
https://github.com/google/syzkaller/blob/master/pkg/build/netbsd.go#L135-L140
That's the plece to do it, right? Please send a PR.

[Dmitry Vyukov]

On Sun, May 24, 2020 at 9:06 AM Maxime Villard <m...@m00nbsd.net> wrote:
>
> Le 21/05/2020 à 11:54, Dmitry Vyukov a écrit :
> > $ go test ./pkg/report
> >
> > or to run subset of tests:
> >
> > $ go test -v -run TestParse/netbsd ./pkg/report
>
> That worked, thanks

>
>
> Le 21/05/2020 à 09:14, Dmitry Vyukov a écrit :
> >> We will have to add the "vhci" pseudo-device in the instances. The commands
> >> to create it are:
> >>
> >> # cd /dev
> >> # ./MAKEDEV vhci
> >>
> >> I suspect this has to be done manually?
> >
> > I hope you don't mean that for every VM booted by the fuzzer after
> > each crash, the process needs to halt and wait for a human to ssh,
> > type some commands and resume the process :)
> >
> > Here we do some image setup:
> > https://github.com/google/syzkaller/blob/master/pkg/build/netbsd.go#L135-L140
> > That's the plece to do it, right? Please send a PR.
>

> Not sure; we should be creating /dev/vhci the same way we created
> /dev/kcov, and we don't create /dev/kcov here.
>
> I think it was created only once:
>
> https://github.com/R3x/netbsd-fuzzing-aids/blob/master/install_netbsd.sh
>
> There's a "./MAKEDEV kcov" in here. So there should be a "./MAKEDEV vhci"
> as well. But I'm not sure where/how this script gets executed?

It may be better to create both devices in pkg/build/netbsd.go
instead. This way we depend less on external things/special setup and
any new changes will be deployed automatically with 0 cost. Updating
prepackaged archives is expensive.

[Dmitry Vyukov]

On Tue, May 26, 2020 at 1:03 PM Maxime Villard <m...@m00nbsd.net> wrote:

>
> Le 24/05/2020 à 12:19, Dmitry Vyukov a écrit :
> >> I think it was created only once:
> >>
> >> https://github.com/R3x/netbsd-fuzzing-aids/blob/master/install_netbsd.sh
> >>
> >> There's a "./MAKEDEV kcov" in here. So there should be a "./MAKEDEV vhci"
> >> as well. But I'm not sure where/how this script gets executed?
> >
> > It may be better to create both devices in pkg/build/netbsd.go
> > instead. This way we depend less on external things/special setup and
> > any new changes will be deployed automatically with 0 cost. Updating
> > prepackaged archives is expensive.
>

> It looks like my fix didn't work:
>
> https://github.com/google/syzkaller/pull/1762
>
> The instances still report:
>
> USB emulation: /dev/vhci does not exist
>
> How can I debug this?

Where is that "USB emulation: /dev/vhci does not exist" in the logs?
Does that build include the commit?

I noticed strings.Join(commands, ";") there and ; discards errors,
right? Not discarding errors looks like a good step.

The image/key it uses are available here:
https://github.com/google/syzkaller/tree/master/docs/netbsd#syzbot
So you may try to repeat everything it's doing step by step.

[Dmitry Vyukov]

On Thu, May 28, 2020 at 7:59 AM Siddharth Muralee
<siddhart...@gmail.com> wrote:
>
> Hey,
>
>>
>> Thanks for uploading, I've checked your image, it looks good to me.
>
>
> Great!
>
>>
>> Are you using Ubuntu? I'm using Fedora, maybe there's some problem with
>> it.
>
>
> Yex - Linux 5.3.0-46-generic #38~18.04.1-Ubuntu SMP Tue Mar 31 04:17:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I've updated the image on the machine.
It's also available as:
https://storage.googleapis.com/syzkaller/netbsd-image.tar.gz

It's possible I gave links to an older image.
I see the GCS bucket contained an older image+key files separately and
a newer tar.gz that contained both image and key. The docs pointer to
the older files, while syzbot probably used newer tar.gz.
I've updated the docs to point to the newer tar.gz now:
https://github.com/google/syzkaller/blob/master/docs/netbsd/README.md#syzbot

It will take some time for the instance to build a new image as it's
currently busy with some bisection. But if we see a newer build, it
should be already using the new image.

[Dmitry Vyukov]

Turns out we also need this:
https://github.com/google/syzkaller/commit/7e7ceb21ba30cacae05f29b2f574b9175ccfd425
I think ssh-keygen should have been created it with the right perms,
but somehow it got mangled along the way...

[Maxime Villard]

Thanks for your changes.

It appears the instances haven't built in two days, but there is no error
indicating what is the cause. Is there still a problem with the image? Or
is it because since the kMSan instance is down, syzbot is waiting for it
to work again before doing an update of the images on all the instances
at once?

Thanks,
Maxime

[Dmitry Vyukov]

Turns out the previous fix wasn't enough, because the permission check
failed in the pkg/build/netbsd.go itself when it tried to run qemu.
I've added:
https://github.com/google/syzkaller/commit/6f3e1c7c67bc16c53b8c778984c068c342ec5274

The error did not surface as syzbot bug because we are specifically
picky about what errors during build process are reported and what are
not reported. E.g. if we fail to create some temp file due to ENOSPSC,
we don't want to send email to all linux kernel developers (some of
them may be actively not happy about this). However, we have not seen
lots of infra failures and silent errors are bad too. So maybe we need
to tune this policy.

[Dmitry Vyukov]

Filed https://github.com/google/syzkaller/issues/1777 to track this.

[Dmitry Vyukov]

On Thu, Jun 11, 2020 at 9:01 AM Maxime Villard <m...@m00nbsd.net> wrote:

> Hi Dmitry,
> Is it possible to update the host toolchain? There are two separate
> changes I want to make in syzkaller, but they need fresh toolchain.
>
> Thanks,
> Maxime

Hi Maxime,

I've build a new toolchain on:

commit 0b8bb860b8bdcdaece1de92252bbdc67941c4613
Date: Thu Jun 11 09:02:04 2020 +0000
regen

and deployed to syzbot.

It's also available at
https://storage.googleapis.com/syzkaller/netbsd-toolchain.tar.gz