An error when run syzkaller targeted at Android devices(pixel4)

[Zheng Zhang]

Hi all:

        I tried to run syzkaller to fuzz the pixel4 phone, there is a crash at the very beginning: "panic: LINE, dump: LINE". I could not find useful information via google. Could you give me any suggestions? I believe that I strictly follow the instructions in https://github.com/google/syzkaller/blob/master/docs/linux/setup_linux-host_android-device_arm64-kernel.md. 

        Thanks very much!

[Dmitry Vyukov]

Hi Zheng,

syzkaller seems to be working as intended as found a bug. You need to
fix found bugs to find more bugs with fuzzing.

[Zheng Zhang]

Hi Dmitry:

      Thanks for replying!

      It seems that syzkaller doesn't find new bug in pixel4. I set "enable_syscalls" option to limit fuzz in several syscalls ( clock_gettime, for example), this error still exists. Thus it seems that syzkaller doesn't work well. I checked the log and report, it seems that the fuzzer runs normally but immediately the connection is closed. 

      Here are some details of my installation:

      wget https://dl.google.com/go/go1.14.2.linux-amd64.tar.gz

      tar -xf go1.14.2.linux-amd64.tar.gz 

      mv go goroot; mkdir gopath; export GOPATH=`pwd`/gopath; export GOROOT=`pwd`/goroot; export PATH=$GOPATH/bin:$PATH; export PATH=$GOROOT/bin:$PATH

      go get -u -d github.com/google/syzkaller/prog

      cd gopath/src/github.com/google/syzkaller/
      make TARGETOS=linux TARGETARCH=arm64

      ./bin/syz-manager -config=my.cfg

       I think I followed the instructions strictly, thus I'm a little confused. In the workdir/crashes/subdir/description, it shows "panic:LINE, dump:LINE", but I get little information by searching it via google. 

[Dmitry Vyukov]

On Tue, May 5, 2020 at 10:41 AM Zheng Zhang <zzha...@ucr.edu> wrote:
>
> Hi Dmitry:
> Thanks for replying!
> It seems that syzkaller doesn't find new bug in pixel4. I set "enable_syscalls" option to limit fuzz in several syscalls ( clock_gettime, for example), this error still exists. Thus it seems that syzkaller doesn't work well. I checked the log and report, it seems that the fuzzer runs normally but immediately the connection is closed.
> Here are some details of my installation:
> wget https://dl.google.com/go/go1.14.2.linux-amd64.tar.gz
> tar -xf go1.14.2.linux-amd64.tar.gz
> mv go goroot; mkdir gopath; export GOPATH=`pwd`/gopath; export GOROOT=`pwd`/goroot; export PATH=$GOPATH/bin:$PATH; export PATH=$GOROOT/bin:$PATH
> go get -u -d github.com/google/syzkaller/prog
> cd gopath/src/github.com/google/syzkaller/
> make TARGETOS=linux TARGETARCH=arm64
> ./bin/syz-manager -config=my.cfg
>
> I think I followed the instructions strictly, thus I'm a little confused. In the workdir/crashes/subdir/description, it shows "panic:LINE, dump:LINE", but I get little information by searching it via google.

Sometimes it's enough to just boot a kernel to find bugs. If it's that
bad, it's pointless to fuzz to find more bugs. I would suggest fixing
all bugs that happen without fuzzing and then switch to fuzzing to
find more.

> On Monday, May 4, 2020 at 11:35:24 PM UTC-7, Dmitry Vyukov wrote:
>>
>> On Tue, May 5, 2020 at 7:35 AM Zheng Zhang <zzha...@ucr.edu> wrote:
>> >
>> > Hi all:
>> > I tried to run syzkaller to fuzz the pixel4 phone, there is a crash at the very beginning: "panic: LINE, dump: LINE". I could not find useful information via google. Could you give me any suggestions? I believe that I strictly follow the instructions in https://github.com/google/syzkaller/blob/master/docs/linux/setup_linux-host_android-device_arm64-kernel.md.
>> > Thanks very much!
>>
>> Hi Zheng,
>>
>> syzkaller seems to be working as intended as found a bug. You need to
>> fix found bugs to find more bugs with fuzzing.
>

> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/6b24ea3a-f091-4213-acc9-33460dcdbb61%40googlegroups.com.

[Zheng Zhang]

Hi Dmitry:

     Thanks a lot for replying.

     Sorry, I'm a little confused about "just boot a kernel ", do you mean I load the kernel with syzkaller but not boot the fuzzer? How could I achieve it? Is there any reference tutorial I can learn from?

     Besides, I checked the debug log, when the error happens, the fuzzer has been booted. However I don't find details about the error, it seems that the connection is broken when the fuzzer works normally. 

     I have tried the fuzzing with a different version of Android/kernel/syzkaller, and get the same error. It seems to have no relationship with a specific version. I also tried with pixel3 and get the same result.

     I wonder if it's because I use the Android compiled by myself. I tried to use the official release of pixel3(https://developers.google.com/android/images#blueline), then I flash the boot.img with kcov. Then I use syzkaller to fuzz it, this error disappears.  Thus this hypothesis seems to be true. I'm confused since I strictly followed the instructions in the official tutorial.

     Could you give me any suggestions? When you fuzz the pixel phones, do you compile and flash the whole android or only flash boot.img? 

     Thank you very much!

> To unsubscribe from this group and stop receiving emails from it, send an email to syzk...@googlegroups.com.

[Zheng Zhang]

These are all the commands that I used for compiling Android:  https://docs.google.com/document/d/1sr0eHqVfIc30P4tFNbdiv1BkeRYt7s2Ya1c7dGbJfbU/edit?usp=sharing. Is there any error?

All outputs of syzkaller are in https://drive.google.com/file/d/145YvPLOD22ZS2iTJsNrvKXA3-hMqJJkv/view?usp=sharing

[Dmitry Vyukov]

On Sat, May 9, 2020 at 12:46 AM Zheng Zhang <zzha...@ucr.edu> wrote:
>
> Hi Dmitry:

> Thanks a lot for replying.
> Sorry, I'm a little confused about "just boot a kernel ", do you mean I load the kernel with syzkaller but not boot the fuzzer? How could I achieve it? Is there any reference tutorial I can learn from?

I meant you just boot your device and you see bugs in dmesg.

> Besides, I checked the debug log, when the error happens, the fuzzer has been booted. However I don't find details about the error, it seems that the connection is broken when the fuzzer works normally.

Please post a full syz-manager log with -debug flag.

> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/e12dffb2-1e1a-4886-bfae-f97cf57b9c4a%40googlegroups.com.

[zzha...@ucr.edu]

Hi Dmitry:

     Thanks a lot!

     I have found the reason. There are some normal dmesg outputs that are determined as error message by syzkaller by mistake. For example, "evtlog_status: enable:0, panic:1, dump:2".

     I set it as an exception in pkg/report/report.go  func matchOops, then this issue is solved.

     Thanks for your suggestions!

> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/e12dffb2-1e1a-4886-bfae-f97cf57b9c4a%40googlegroups.com.

[Darby Sauter]

What did you end up doing exactly? I'm not familliar with GO.

I am using a Pixel 3XL on Android 11

When using default settings I was getting a ton of "panic: LINE, dump: LINE"

So I decided to add a syscall with a bug and only fuzz that syscall to make sure my environment is working properly

the syscall will jump to 0xdeadbeef when passed that as an argument (which I have verified with a user program)

but even though the log shows it is calling darby(0xdeadbeef) and shows the PC being 0xdeadbeef it still reports: "panic: LINE, dump: LINE"

I also have: 

[drm:sde_dbg_init:3432] evtlog_status: enable:0, panic:1, dump:2

sde_rotator_evtlog_create_debugfs: evtlog_status: enable:0, panic:1, dump:2

Here is my log: log.txt

report: report.txt

[Dmitry Vyukov]

On Mon, Feb 22, 2021 at 9:08 PM Darby Sauter <quafnd...@gmail.com> wrote:
>
>
> What did you end up doing exactly? I'm not familliar with GO.
>
> I am using a Pixel 3XL on Android 11
>
> When using default settings I was getting a ton of "panic: LINE, dump: LINE"
>
> So I decided to add a syscall with a bug and only fuzz that syscall to make sure my environment is working properly
>
> the syscall will jump to 0xdeadbeef when passed that as an argument (which I have verified with a user program)
>
> but even though the log shows it is calling darby(0xdeadbeef) and shows the PC being 0xdeadbeef it still reports: "panic: LINE, dump: LINE"
>
> I also have:
> [drm:sde_dbg_init:3432] evtlog_status: enable:0, panic:1, dump:2
> sde_rotator_evtlog_create_debugfs: evtlog_status: enable:0, panic:1, dump:2
>
> Here is my log: log.txt
> report: report.txt

Should be fixed by https://github.com/google/syzkaller/pull/2456

> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/7b5d8b26-84ad-4efa-8cf5-b9c9bfe580d8n%40googlegroups.com.

[Darby Sauter]

Thanks! I just added the fix but now I am getting:

It is giving info now, but I still think this isn't the correct description.

I would expect it to have something to do with my syscall I added (darby())

I believe the log is the same.

[Dmitry Vyukov]

On Mon, Feb 22, 2021 at 9:47 PM Darby Sauter <quafnd...@gmail.com> wrote:
>
> Thanks! I just added the fix but now I am getting:
>
>
>

> It is giving info now, but I still think this isn't the correct description.
>
> I would expect it to have something to do with my syscall I added (darby())
>
> I believe the log is the same.

"INFO:" is detected by syzkaller as a kernel bug. If your code prints
"INFO:", don't print it.

> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/d35eba0d-24ec-4248-8ad5-444a3509b3f7n%40googlegroups.com.

[Darby Sauter]

It wasn't my code printing that, I added extra cases and created a PR: https://github.com/google/syzkaller/pull/2457

My syzkaller now reports 2 descriptions:

The first actually shows my bug, the second is expected behavior for Android due to not having suzy-q cable as per setup_linux-host_android-device_arm-kernel.md

This should now work for people running syzkaller on Android out of the box.

Thanks for the help Dmitry!

[Sachin Grover]

Darby,

you can also put these under the ignores section in syzkaller config file:

"ignores": [

            "panic:1, dump:2",

        "INFO: CAM*",

        "INFO:*",

        "WARNING in apply_constraint"

        ]   

Since these warning will keep changing as per android version and oems. Keeping it as part of config file will be beneficial.

Thanks,

Sachin

[Dmitry Vyukov]

On Wed, Feb 24, 2021 at 6:30 AM Darby Sauter <quafnd...@gmail.com> wrote:

It wasn't my code printing that, I added extra cases and created a PR: https://github.com/google/syzkaller/pull/2457

Thanks

 

My syzkaller now reports 2 descriptions:

"in corrupted" means syzkaller couldn't parse stack traces in the crash.

We have some support for parsing arm64 oopses and have some tests:

$ grep arm64 pkg/report/testdata/linux/report/*

Maybe you don't have DEBUG_INFO enabled? Or something wrong with the unwinder? Do they look good? Or there is indeed something wrong with them?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/8b694b0f-9b7f-4695-8561-324222dad713n%40googlegroups.com.

[Darby Sauter]

> Since these warning will keep changing as per android version and oems. Keeping it as part of config file will be beneficial.

Would it not be beneficial to support a Pixel device running an official kernel to be supported?

> "in corrupted" means syzkaller couldn't parse stack traces in the crash.

> We have some support for parsing arm64 oopses and have some tests:

> $ grep arm64 pkg/report/testdata/linux/report/*

> Maybe you don't have DEBUG_INFO enabled? Or something wrong with the unwinder? Do they look good? Or there is indeed something wrong with them?

I have "-e CONFIG_DEBUG_INFO" in my build.config and there are a few lines in the log containing "debug_info" so I don't think that is the issue.

The panic looks like:

[   28.231851] Internal error: Oops - SP/PC alignment exception: 8a000000 [#1] PREEMPT SMP

[   28.231866] Modules linked in:

[   28.231879] CPU: 6 PID: 1942 Comm: syz-executor Not tainted 4.9.232-dirty_audio #1

[   28.231888] Hardware name: Google Inc. MSM sdm845 C1 DVT1.1 (DT)

[   28.231899] task: 00000000bb76f97a task.stack: 00000000fd9408ab

[   28.231909] PC is at 0xffffffffdeadbeef

[   28.231925] LR is at SyS_darby+0x34/0x48

[   28.231935] pc : [<ffffffffdeadbeef>] lr : [<ffffff90080ed944>] pstate: 60400145

[   28.231943] sp : ffffffc01ec87ea0

[   28.231951] x29: ffffffc01ec87ea0 x28: ffffffc055663000 

...

Which doesn't contain "__arm64_" anywhere, I'm not too sure where the array of matches is (pkg/host/syscalls_linux.go:79 or pkg/report/linux.go:914)

How would I go about properly parsing this?

I have also linked full log and report

- log

- report

Thanks!

[Dmitry Vyukov]

On Thu, Feb 25, 2021 at 1:52 AM Darby Sauter <quafnd...@gmail.com> wrote:

> Since these warning will keep changing as per android version and oems. Keeping it as part of config file will be beneficial.

Would it not be beneficial to support a Pixel device running an official kernel to be supported?

That's definitely better, if you are asking me :)

 

> "in corrupted" means syzkaller couldn't parse stack traces in the crash.

> We have some support for parsing arm64 oopses and have some tests:

> $ grep arm64 pkg/report/testdata/linux/report/*

> Maybe you don't have DEBUG_INFO enabled? Or something wrong with the unwinder? Do they look good? Or there is indeed something wrong with them?

I have "-e CONFIG_DEBUG_INFO" in my build.config and there are a few lines in the log containing "debug_info" so I don't think that is the issue.

The panic looks like:

[   28.231851] Internal error: Oops - SP/PC alignment exception: 8a000000 [#1] PREEMPT SMP

[   28.231866] Modules linked in:

[   28.231879] CPU: 6 PID: 1942 Comm: syz-executor Not tainted 4.9.232-dirty_audio #1

[   28.231888] Hardware name: Google Inc. MSM sdm845 C1 DVT1.1 (DT)

[   28.231899] task: 00000000bb76f97a task.stack: 00000000fd9408ab

[   28.231909] PC is at 0xffffffffdeadbeef

[   28.231925] LR is at SyS_darby+0x34/0x48

[   28.231935] pc : [<ffffffffdeadbeef>] lr : [<ffffff90080ed944>] pstate: 60400145

[   28.231943] sp : ffffffc01ec87ea0

[   28.231951] x29: ffffffc01ec87ea0 x28: ffffffc055663000 

...

Which doesn't contain "__arm64_" anywhere, I'm not too sure where the array of matches is (pkg/host/syscalls_linux.go:79 or pkg/report/linux.go:914)

How would I go about properly parsing this?

I have also linked full log and report

- log

- report

Thanks!

Thanks for providing the sample, I"ve added it as a test and fixed parsing a bit:

https://github.com/google/syzkaller/pull/2456/commits/16783c8c8c5246b511464cb339c2e987549b262d

The issue was with the "PC is at 0xffffffffdeadbeef" line. Wild jumps are quite uncommon. We should have some samples for jumps to NULL, but I am not sure about completely wild jumps, definitely not for arm64.

There is still some issue with I think kernel unwinder, see the comment in the test.

But I think it should not affect longer real stacks.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/76b01c34-db51-4fd3-8c22-c96339434f33n%40googlegroups.com.

[Darby Sauter]

I just merged all the changes and now get "Internal error in el0_svc_naked" which looks to be what I assume to be the expected outcome.

Thanks for all the help!

[Dmitry Vyukov]

All is merged now. Thanks.

To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/ce125d6b-899b-4974-84c9-f5c42ac6f53en%40googlegroups.com.