syzkaller for freebsd again 2

[timo.v...@fh-muenster.de]

Hi,

I'm wondering what is the current status of FreeBSD support. In the last few days, I tried to run syzkaller on a ubuntu host (Ubuntu 18.04.1 LTS x86_64) and a FreeBSD qemu VM (installed from FreeBSD-12.0-ALPHA8-amd64-20180928-r338991-disc1.iso) with no success.

I set up the FreeBSD VM such that I'm able to ssh as root from ubuntu host by using RSA key authentication. I tried to follow the documentation

https://github.com/google/syzkaller/blob/master/docs/freebsd.md

and did a 

make manager fuzzer execprog TARGETOS=freebsd

on the host. After that I copied the executor/* files to the FreeBSD VM and tried to compile it. Since the executor/executor_freebsd.cc disappeared, the documentation seems to be partly obsolete. I noted the GitRevision from the make command output on the host and used this command to build the executor

c++ -o syz-executor executor/executor.cc -pthread -Wall -Wframe-larger-than=8192 -Wparentheses -Werror -O2 -m64 -static -DGOOS_freebsd=1 -DGOARCH_amd64=1 -DGIT_REVISION=\"5b11ac2c91d99eababa8f04c7e2f9484c0763b53\"

which failed because endian.h could not be found. I figured that on FreeBSD the include command in executor/common.h must be

#include <sys/endian.h>

After I changed that, I was able to compile the executor. Then, I copied the built syz-executor file to syzkaller/bin/freebsd_amd64/ on the host and gave syz-manager a first try with the following command

sudo bin/syz-manager -config freebsd.cfg -debug

freebsd.cfg file content:

{
    "target": "freebsd/amd64",
    "http": "127.0.0.1:56741",
    "workdir": "/home/msvoelker/src/gopath/src/github.com/google/syzkaller/workdir",
    "image": "/home/msvoelker/qemu/freebsd/freebsd.img",
    "sshkey": "/home/msvoelker/qemu/freebsd/freebsd_id_rsa",
    "syzkaller": "/home/msvoelker/src/gopath/src/github.com/google/syzkaller",
    "procs": 8,
    "type": "qemu",
    "vm": {
        "count": 4,
        "cpu": 2,
        "mem": 2048
    }
}

It complained about a missing kcov support in the VM. Because of that and because executor/executor_freebsd.cc disappeared, I assume there is a supported kcov implementation for FreeBSD now. The only implementation I found is this

https://reviews.freebsd.org/D14599

I used it to patch the FreeBSD kernel in my VM and gave syz-manager another try. Now it crashed on a ioctl call. I figured that in executor/executor_bsd.h the definitions of KIOSETBUFSIZE and KIOENABLE does not match to the definitions in the FreeBSD kcov patch. I changed it in executor_bsd.h to

#define KIOSETBUFSIZE        _IOWINT('c', 4)

#define KIOENABLE        _IOWINT('c', 2)

(It might be better to include <sys/kcov.h> instead)

and changed the corresponding ioctl calls such that the value instead of the address is passed (I removed the & before kCoverSize and kcov_mode). Now, when I start syz-manager, I get another error.

spawned worker pid 739
#0 [28ms] -> mmap(0x20000000, 0x1000, 0x3, 0x12, 0xffffffffffffffff, 0x0)
SIGSEGV on 0xffffffffffffffff, exiting
panic: kcov_mmap_cleanup: Cleanup while running

Thats the point where I doubt to be on the right track. Could someone please lead me to the right track or write about the current status of FreeBSD support.

Thanks

Timo

[Dmitry Vyukov]

Hi Timo,

Sorry for your experience. I think you were generally on the right
track. FreeBSD support is not finished, and in particular we don't
have testing so things fall apart.
I've added some people who were involved in FreeBSD support to CC. As
far as I understand syzkaller is working on FreeBSD somehow for some
people, but I don't know if they have local patches, or use an old
version, or something else:
https://twitter.com/ed_maste/status/1045409421973753858

One quick thing you may try is to add:
"cover": false
to syz-manager config. This will disable (well, at least should,
because it's also not tested for FreeBSD at the moment) everything
related to coverage. If that works, at least we know that coverage is
the only missing piece in the equation.

Do you mind to send a pull request with the docs update and missing
include? That would be useful.

Ultimately we need to setup FreeBSD on syzbot:
https://github.com/google/syzkaller/blob/master/docs/syzbot.md
Besides continuous kernel testing, it will also provide continuous
testing for syzkaller/freebsd support. FWIW OpenBSD is already on
syzbot:
https://syzkaller.appspot.com/#openbsd

Thanks

[Timo Völker]

Hi Dmitry,

> On 16. Oct 2018, at 11:36, Dmitry Vyukov <dvy...@google.com> wrote:
>
> Hi Timo,
>
> Sorry for your experience. I think you were generally on the right
> track. FreeBSD support is not finished, and in particular we don't
> have testing so things fall apart.
> I've added some people who were involved in FreeBSD support to CC. As
> far as I understand syzkaller is working on FreeBSD somehow for some
> people, but I don't know if they have local patches, or use an old
> version, or something else:
> https://twitter.com/ed_maste/status/1045409421973753858
>
> One quick thing you may try is to add:
> "cover": false
> to syz-manager config. This will disable (well, at least should,
> because it's also not tested for FreeBSD at the moment) everything
> related to coverage. If that works, at least we know that coverage is
> the only missing piece in the equation.

It does not work out of the box with "cover":false. It seems syzkaller tries to use mmap for a quick test. This crashes, because the flag MAP_ANONYMOUS is used, which is not defined in the mmap description for freebsd (sys/freebsd/posix_mm.txt). If I add it and run

make extract TARGETOS=freebsd SOURCEDIR=~/src/freebsd/
(~/src/freebsd/ contains the FreeBSD sources

the flag is added as comment to posix_mm_amd64.const with the note: "is not set". I couldn't find the reason for that. For a test, I changed the flag MAP_ANONYMOUS in sys/targets/common.go to MAP_ANON. After that, fuzzing with "cover":false worked.

>
> Do you mind to send a pull request with the docs update and missing
> include? That would be useful.

I created a fork from main syzkaller repository, created a branch in that fork, and pushed my changes in small commits. However, I was not able to successfully run make presubmit. It failed with the following message.

powerpc64le-linux-gnu-gcc -o ./bin/linux_ppc64le/syz-executor executor/executor.cc \
-pthread -Wall -Wframe-larger-than=8192 -Wparentheses -Werror -O2 -D__powerpc64__ -static \
-DGOOS_linux=1 -DGOARCH_ppc64le=1 -DGIT_REVISION=\"4b2418d456189b6c1182a452b89fbc66395bc01a\"
GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/sys.GitRevision=4b2418d456189b6c1182a452b89fbc66395bc01a" -o ./bin/syz-manager github.com/google/syzkaller/syz-manager
...
GOOS=openbsd GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/sys.GitRevision=4b2418d456189b6c1182a452b89fbc66395bc01a" -o ./bin/syz-upgrade github.com/google/syzkaller/tools/syz-upgrade
In file included from executor/executor.cc:137:0:
executor/common.h:19:10: fatal error: sys/endian.h: No such file or directory
#include <sys/endian.h> // for htobe*.
^~~~~~~~~~~~~~
compilation terminated.
Makefile:108: recipe for target 'executor' failed

Because of that, I did not a pull request. You find my changes in the freebsd-again-2 branch of my syzkaller repository

https://github.com/msvoelker/syzkaller/tree/freebsd-again-2

>
> Ultimately we need to setup FreeBSD on syzbot:
> https://github.com/google/syzkaller/blob/master/docs/syzbot.md
> Besides continuous kernel testing, it will also provide continuous
> testing for syzkaller/freebsd support. FWIW OpenBSD is already on
> syzbot:
> https://syzkaller.appspot.com/#openbsd
>
> Thanks

With my changes, it becomes necessary to have kcov support in FreeBSD. In FreeBSD with sources (head 335299 2018-06-17 19:44:24Z) in /usr/src, I followed these steps to add kcov support.

su
cd /usr/src
wget https://reviews.freebsd.org/file/data/bn3p5gnfimbwjxccjayo/PHID-FILE-pij5qnystwmhfs34aiw2/D14599.diff
patch < D14599.diff
make kernel
shutdown -r now
su
cd /usr/src
make includes

The last step is necessary to generate the /usr/include/sys/kcov.h file.

Timo

[Dmitry Vyukov]

On Fri, Oct 19, 2018 at 1:59 PM, Timo Völker
<timo.v...@fh-muenster.de> wrote:
> Hi Dmitry,
>
>> On 16. Oct 2018, at 11:36, Dmitry Vyukov <dvy...@google.com> wrote:
>>
>> Hi Timo,
>>
>> Sorry for your experience. I think you were generally on the right
>> track. FreeBSD support is not finished, and in particular we don't
>> have testing so things fall apart.
>> I've added some people who were involved in FreeBSD support to CC. As
>> far as I understand syzkaller is working on FreeBSD somehow for some
>> people, but I don't know if they have local patches, or use an old
>> version, or something else:
>> https://twitter.com/ed_maste/status/1045409421973753858
>>
>> One quick thing you may try is to add:
>> "cover": false
>> to syz-manager config. This will disable (well, at least should,
>> because it's also not tested for FreeBSD at the moment) everything
>> related to coverage. If that works, at least we know that coverage is
>> the only missing piece in the equation.
>
> It does not work out of the box with "cover":false. It seems syzkaller tries to use mmap for a quick test. This crashes, because the flag MAP_ANONYMOUS is used, which is not defined in the mmap description for freebsd (sys/freebsd/posix_mm.txt).

Okay, so there are more breakages.
I've pushed this change that fixes MAP_ANONYMOUS, and hopefully should
prevent such cases in future:
https://github.com/google/syzkaller/commit/ecb386fe6f6849c451955e16556d04b388b1fde1

> If I add it and run
>
> make extract TARGETOS=freebsd SOURCEDIR=~/src/freebsd/
> (~/src/freebsd/ contains the FreeBSD sources
>
> the flag is added as comment to posix_mm_amd64.const with the note: "is not set". I couldn't find the reason for that.

That's because MAP_ANONYMOUS is defined in kernel headers under
#ifndef _KERNEL and we define _KERNEL in sys/syz-extract/freebsd.go

> For a test, I changed the flag MAP_ANONYMOUS in sys/targets/common.go to MAP_ANON. After that, fuzzing with "cover":false worked.
>
>>
>> Do you mind to send a pull request with the docs update and missing
>> include? That would be useful.
>
> I created a fork from main syzkaller repository, created a branch in that fork, and pushed my changes in small commits. However, I was not able to successfully run make presubmit. It failed with the following message.
>
> powerpc64le-linux-gnu-gcc -o ./bin/linux_ppc64le/syz-executor executor/executor.cc \
> -pthread -Wall -Wframe-larger-than=8192 -Wparentheses -Werror -O2 -D__powerpc64__ -static \
> -DGOOS_linux=1 -DGOARCH_ppc64le=1 -DGIT_REVISION=\"4b2418d456189b6c1182a452b89fbc66395bc01a\"
> GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/sys.GitRevision=4b2418d456189b6c1182a452b89fbc66395bc01a" -o ./bin/syz-manager github.com/google/syzkaller/syz-manager
> ...
> GOOS=openbsd GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/sys.GitRevision=4b2418d456189b6c1182a452b89fbc66395bc01a" -o ./bin/syz-upgrade github.com/google/syzkaller/tools/syz-upgrade
> In file included from executor/executor.cc:137:0:
> executor/common.h:19:10: fatal error: sys/endian.h: No such file or directory
> #include <sys/endian.h> // for htobe*.
> ^~~~~~~~~~~~~~
> compilation terminated.
> Makefile:108: recipe for target 'executor' failed

This happens because we build freebsd version on linux too, to give it
at least some testing. I understand that strictly saying, it does not
have to work. But it worked to date and was useful for testing.

Perhaps we can do:

#if GOOS_freebsd && !__linux__

#include <sys/endian.h> // for htobe*.

This will exclude building freebsd version on linux.

[Timo Völker]

>> Hi Dmitry,
>>
>>> On 16. Oct 2018, at 11:36, Dmitry Vyukov <dvy...@google.com> wrote:
>>>
>>> Hi Timo,
>>>
>>> Sorry for your experience. I think you were generally on the right
>>> track. FreeBSD support is not finished, and in particular we don't
>>> have testing so things fall apart.
>>> I've added some people who were involved in FreeBSD support to CC. As
>>> far as I understand syzkaller is working on FreeBSD somehow for some
>>> people, but I don't know if they have local patches, or use an old
>>> version, or something else:
>>> https://twitter.com/ed_maste/status/1045409421973753858
>>>
>>> One quick thing you may try is to add:
>>> "cover": false
>>> to syz-manager config. This will disable (well, at least should,
>>> because it's also not tested for FreeBSD at the moment) everything
>>> related to coverage. If that works, at least we know that coverage is
>>> the only missing piece in the equation.
>>
>> It does not work out of the box with "cover":false. It seems syzkaller tries to use mmap for a quick test. This crashes, because the flag MAP_ANONYMOUS is used, which is not defined in the mmap description for freebsd (sys/freebsd/posix_mm.txt).
>
> Okay, so there are more breakages.
> I've pushed this change that fixes MAP_ANONYMOUS, and hopefully should
> prevent such cases in future:
> https://github.com/google/syzkaller/commit/ecb386fe6f6849c451955e16556d04b388b1fde1

Great, that works.

>
>
>> If I add it and run
>>
>> make extract TARGETOS=freebsd SOURCEDIR=~/src/freebsd/
>> (~/src/freebsd/ contains the FreeBSD sources
>>
>> the flag is added as comment to posix_mm_amd64.const with the note: "is not set". I couldn't find the reason for that.
>
> That's because MAP_ANONYMOUS is defined in kernel headers under
> #ifndef _KERNEL and we define _KERNEL in sys/syz-extract/freebsd.go

OK, thats probably the reason.

>
>> For a test, I changed the flag MAP_ANONYMOUS in sys/targets/common.go to MAP_ANON. After that, fuzzing with "cover":false worked.
>>
>>>
>>> Do you mind to send a pull request with the docs update and missing
>>> include? That would be useful.
>>
>> I created a fork from main syzkaller repository, created a branch in that fork, and pushed my changes in small commits. However, I was not able to successfully run make presubmit. It failed with the following message.
>>
>> powerpc64le-linux-gnu-gcc -o ./bin/linux_ppc64le/syz-executor executor/executor.cc \
>> -pthread -Wall -Wframe-larger-than=8192 -Wparentheses -Werror -O2 -D__powerpc64__ -static \
>> -DGOOS_linux=1 -DGOARCH_ppc64le=1 -DGIT_REVISION=\"4b2418d456189b6c1182a452b89fbc66395bc01a\"
>> GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/sys.GitRevision=4b2418d456189b6c1182a452b89fbc66395bc01a" -o ./bin/syz-manager github.com/google/syzkaller/syz-manager
>> ...
>> GOOS=openbsd GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/sys.GitRevision=4b2418d456189b6c1182a452b89fbc66395bc01a" -o ./bin/syz-upgrade github.com/google/syzkaller/tools/syz-upgrade
>> In file included from executor/executor.cc:137:0:
>> executor/common.h:19:10: fatal error: sys/endian.h: No such file or directory
>> #include <sys/endian.h> // for htobe*.
>> ^~~~~~~~~~~~~~
>> compilation terminated.
>> Makefile:108: recipe for target 'executor' failed
>
> This happens because we build freebsd version on linux too, to give it
> at least some testing. I understand that strictly saying, it does not
> have to work. But it worked to date and was useful for testing.
>
> Perhaps we can do:
>
> #if GOOS_freebsd && !__linux__
> #include <sys/endian.h> // for htobe*.
>
> This will exclude building freebsd version on linux.

I added !__linux__. Also, I had to change pkg/csource/csource_test.go in order to skip tests, when target is freebsd.

For my fix in executor/executor_bsd.h, I had to change the Makefile in order to skip build executor for freebsd. For both, I created a pull request (#777).

I made no process in debugging FreeBSD with kcov (when cover is set to true in syzkaller config). Still, the following lines is the last output I see before syz-manager restarts the VM. Looks like a FreeBSD kernel panic. Do you have any idea whether this is a bigger problem or not?

[10ms] exec opts: procid=0 threaded=1 collide=1 cover=0 comps=0 dedup=1 fault=0/0/0 prog=0
spawned worker pid 728
#0 [15ms] -> mmap(0x20000000, 0x1000, 0x3, 0x1012, 0xffffffffffffffff, 0x0)

SIGSEGV on 0xffffffffffffffff, exiting
panic: kcov_mmap_cleanup: Cleanup while running

cpuid = 0
time = 1540209971
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x47/frame 0xfffffe0000446480
vpanic() at vpanic+0x1ff/frame 0xfffffe00004464e0
panic() at panic+0x43/frame 0xfffffe0000446540
kcov_mmap_cleanup() at kcov_mmap_cleanup+0x68/frame 0xfffffe0000446560
devfs_destroy_cdevpriv() at devfs_destroy_cdevpriv+0xb5/frame 0xfffffe0000446590
devfs_close_f() at devfs_close_f+0x7b/frame 0xfffffe00004465c0
_fdrop() at _fdrop+0x3a/frame 0xfffffe00004465f0
closef() at closef+0x27d/frame 0xfffffe0000446680
fdescfree_fds() at fdescfree_fds+0xb6/frame 0xfffffe00004466d0
fdescfree() at fdescfree+0x59a/frame 0xfffffe0000446790
exit1() at exit1+0x781/frame 0xfffffe0000446800
sigexit() at sigexit+0xcdf/frame 0xfffffe0000446ae0
postsig() at postsig+0x413/frame 0xfffffe0000446bb0
ast() at ast+0x47d/frame 0xfffffe0000446bf0
doreti_ast() at doreti_ast+0x1f/frame 0x7fffffffec80
KDB: enter: panic
[ thread pid 727 tid 100083 ]
Stopped at kdb_enter+0x6b: movq $0,kdb_why
db>
2018/10/22 14:06:21 vm-0: crash: panic: kcov_mmap_cleanup: Cleanup while running

[Dmitry Vyukov]

I can't help here. Looks like a bug in kcov kernel patch.

Ed, are there any plans to fix and upstream the kcov patch?
Out-of-tree patches are always pain and maintenance cost.

[Ed Maste]

Adding Andrew Turner who is working on updating/integrating the
FreeBSD kcov work. Sorry I have been out for various conferences and
other things for the last while.

We definitely do want to get this fixed up and integrated.

[Dmitry Vyukov]

Hi Andrew,

It would be good to integrate kcov into the kernel and fix the
userspace interface.
Then we could proceed with syzbot integration (mainly building kernel/image).
Then it can be incrementally improved from there to better test
various subsystems, etc.