Re: Abnormal execution of Syzkaller on ARM64

[Dmitry Vyukov]

On Mon, 17 Oct 2022 at 10:09, Dongliang Mu <mudongl...@gmail.com> wrote:
>
> Hi Dmitry,
>
> I followed document [1] to install Syzkaller on ARM64. However, when I
> run the following command:
>
> syz-manager -config fuzz_config -debug > log 2>&1
>
> There are lots of errors in the log. I took a look at the log and
> found that ssh failed issue. But I tried manually and I can connect to
> the QEMU ARM64 VM with my private key. Any suggestions to figure out
> the underlying problem? :(
>
> Note that, I only changed the cross compiler from linaro to
> aarch-linux-gnu-gcc-12 shipped on Ubuntu 22.04 since KCOV and KASAN
> can only be enabled on GCC or Clang 12.
>
> [1] https://github.com/google/syzkaller/blob/master/docs/linux/setup_linux-host_qemu-vm_arm64-kernel.md
>
> --
> My best regards to you.
>
> No System Is Safe!
> Dongliang Mu

+syzkaller mailing list

[Dmitry Vyukov]

On Tue, 18 Oct 2022 at 08:07, Dmitry Vyukov <dvy...@google.com> wrote:
>
> On Mon, 17 Oct 2022 at 10:09, Dongliang Mu <mudongl...@gmail.com> wrote:
> >
> > Hi Dmitry,
> >
> > I followed document [1] to install Syzkaller on ARM64. However, when I
> > run the following command:
> >
> > syz-manager -config fuzz_config -debug > log 2>&1
> >
> > There are lots of errors in the log. I took a look at the log and
> > found that ssh failed issue. But I tried manually and I can connect to
> > the QEMU ARM64 VM with my private key. Any suggestions to figure out
> > the underlying problem? :(
> >
> > Note that, I only changed the cross compiler from linaro to
> > aarch-linux-gnu-gcc-12 shipped on Ubuntu 22.04 since KCOV and KASAN
> > can only be enabled on GCC or Clang 12.
> >
> > [1] https://github.com/google/syzkaller/blob/master/docs/linux/setup_linux-host_qemu-vm_arm64-kernel.md
>

> +syzkaller mailing list

Hi Dongliang,

The root cause error is:

SYZFAIL: mmap of output file failed
(errno 11: Resource temporarily unavailable)

I have never seen this before.
For some reason your kernel fails mmap with EAGAIN. man page only
lists this for EAGAIN:

EAGAIN The file has been locked, or too much memory has been
locked (see setrlimit(2)).

I can only suggest to (1) print what that mmap has returned (-1 or
some other address), (2) add kernel debug output to understand why it
returns EAGAIN from the mmap.

[Mark Rutland]

Hi,

If you're using a recent kernel (e.g. v6.1-rc1), this might be the issue I
debugged on Monday, where the error reported by syz-executor is misleading:

https://lore.kernel.org/lkml/Y07dPJOYqshoX4f+@lakrids/

What was happening there was that mmap() with MAP_FIXED is broken upstream, and
when syz-executor finds the returned pointer is not as expected, it fails (and
reports EAGAIN even though the syscall returned successfully with a cobugs
result).

... and there's a fix from Liam Howlett, which works for me locally, testing on
a ThunderX2 host machine.

https://lore.kernel.org/lkml/20221018191613.413...@oracle.com/

Thanks,
Mark.

[Dmitry Vyukov]

Hey Mark!

Good, the mystery is solved.
I've also sent https://github.com/google/syzkaller/pull/3457 to print
the address on these failures.

[Mark Rutland]

Ah, great! That'll make future debugging much easier. :)

FWIW, Liam's patch is now queued in AKPM's mm-hotfixes-stable branch, so I'm
expecting this to be fixed by v6.1-rc2. In the mean time I'm testing with
Liam's patch applied locally.

Thanks,
Mark.

[Dongliang Mu]

Hi all,

I tried the latest kernel today and compiled it with
aarch64-linux-gnu-gcc(aarch64-linux-gnu-gcc (Ubuntu
12.1.0-2ubuntu1~22.04) 12.1.0).

The crash log still shows "SYZFAIL: mmap of output file failed"

netlink: add addr fe80::44 dev wg2: Operation not supported
netlink: device wg2 up master NULL: No such device
mount of binder at /dev/binderfs failed: 19
[25659ms] exec opts: procid=0 threaded=1 cover=0 comps=0 dedup=1
signal=1 timeouts=500/15000/3 prog=0 filter=0
spawned worker pid 2

SYZFAIL: mmap of output file failed
(errno 11: Resource temporarily unavailable)

SYZFAIL: child failed
(errno 0: Success)
loop exited with status 67
2022/10/17 07:46:53 SYZFATAL: Manager.Check call failed: machine check
failed: program execution failed: executor 0: exit status 67

debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 2
debug1: channel 1: free: 127.0.0.1, nchannels 1
Transferred: sent 13476, received 25688 bytes, in 29.4 seconds
Bytes per second: sent 458.5, received 874.1
debug1: Exit status 1

I doubled check the deployment process and found out the ARM toolchain
different from the syzkaller document.

The document [1] uses ARM64 toolchain from Linaro but I use the
distribution-shipped ARM64 toolchain, I wonder if this is the problem
due to my toolchain change.

[1] https://github.com/google/syzkaller/blob/master/docs/linux/setup_linux-host_qemu-vm_arm64-kernel.md
>
> Thanks,
> Mark.

[Mark Rutland]

Which kernel exactly? Can you say which commit?

Is that mainline, linux-next, something else?

I've been testing v6.1-rc4 and v6.1-rc5 without issue.

I've also just tested with the latest commit as of the time of writing:

cc675d22e422442f ("Merge tag 'for-linus-6.1-rc6-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip")

... which also works.

For config, I'm using the fragments in:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/commit/?h=testing/6.1-rc5&id=25f69830eb20dcfe350dfcddd377a10f154def7d

My Syzkaller binaries are built from commit:

e9faea6a6321e726 ("vm/proxyapp: configure proxyApp log output (#3459)")

The toolchain should not matter for this issue, but I've been using the
kernel.org cross toolchain GCC 12.1.0, which you can find linked from:

https://mirrors.edge.kernel.org/pub/tools/crosstool/

Thanks,
Mark.