syzkaller: testing failed: failed to run ["go" "test" "-short" "./..."]: exit status 1 (15)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:
git tree: https://github.com/google/syzkaller.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=1419a0bc500000
dashboard link: https://syzkaller.appspot.com/bug?extid=450ab34a2179e2a4aa60
compiler: go version go1.15.2 openbsd/amd64

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+450ab3...@syzkaller.appspotmail.com



---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[Anton Lindqvist]

Is there anything that can be done to avoid this timeout?

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-openbsd-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-openbsd...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-openbsd-bugs/000000000000ea57e205b29ad6b3%40google.com.

[Dmitry Vyukov]

On Tue, Oct 27, 2020 at 10:38 AM Anton Lindqvist <an...@basename.se> wrote:
>
> Is there anything that can be done to avoid this timeout?

We could try to reduce the duration of prog package tests a bit. But I
already did several rounds of such optimizations and I am not sure how
much more we could squeeze.
The most promising approaches would be either reduce duplication in
tests (e.g. lots of them generate random programs as first step), or
optimizing the prog package itself so that basic operations like
generation/mutation are faster (help production as well).
But the thing is that on my weak laptop it runs in 8 secs:

$ go test -short ./prog
ok github.com/google/syzkaller/prog 8.663s

So the question is how does it timeout after 600 secs on the OpenBSD
machine?... There is something else happening here. And I think this
only happens on OpenBSD.

We use separate GCE VMs for fuzzing , so they don't conflict with
builds/tests (as compared to using qemu's).
We also serialized kernel builds with running Go tests (as a previous
attempt to fix OpenBSD timeouts, which obviously did not help):
https://github.com/google/syzkaller/blob/master/syz-ci/updater.go#L223-L270

To put down fire, we could add -timeout flag to go test on OpenBSD...
but I am not sure what is the value that will eliminate this.

> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-openbsd-bugs/20201027093851.GA4738%40amd64.basename.se.

[Greg Steuck]

Does the Go test harness limit the amount of "stuff" that's happening? I noted a gazillion compile jobs being launched simultaneously. If a bunch of tests and maybe compilations ran concurrently, we'd promptly run into OpenBSD's kernel limitations (still a lot of BIG lock). So, this particular test may not be to blame, just the general pattern of computation is such that something can get delayed for a very long time.

Thanks

Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-openbsd-bugs/CACT4Y%2BZKxB0dRnL%3DZaGnrRSyxFGE_opzffNUFiYGDg%3Dm0voTrQ%40mail.gmail.com.

--

nest.cx is Gmail hosted, use PGP: https://pgp.key-server.io/0x0B1542BD8DF5A1B0
Fingerprint: 5E2B 2D0E 1E03 2046 BEC3  4D50 0B15 42BD 8DF5 A1B0

[Dmitry Vyukov]

Good question.
go tool itself limits the number of invocations of
compiler/linker/tests to GOMAXPROCS (NumCPU by default). But then we
have lots of parallel tests, which again run with GOMAXPROCS
parallelism. So overall it may oversubscribe.
I've also noticed that go test spawns some 'vet' processes that take
an insane amount of time on our auto-generated huge source files. It
caches the results, so locally it's frequently not so big problems,
but I guess on syz-ci it runs each time.
Maybe we can play with GOMAXPROCS (should control number of
compiler/linker/test) and go test -parallel flag (controls intra-test
parallelism).

Can you reproduce it on the machine?
I think it should be something like:

git clean -fd
make descriptions
go test -short ./...


Another point. Makefile adds some flags to every go invocation, but in
syz-ci we just just 'go test'. This should discard all build caches
(flags are different).
Maybe if we run 'make test' it will help after 'make host target'. But
I don't know how much. Also 'make test' currently runs with coverage
enabled (for CI), which we don't need and which can slowdown execution
on its own...

> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-openbsd-bugs/CANMcDo8UAmw%2Bguk7Jez55HfxuLncAc0vVDUo_NYFkCpjwbHQ7Q%40mail.gmail.com.

[Anton Lindqvist]

Thanks for the pointers. This was the first thing I stumbled upon while
looking into this: go vet manages to run out of memory on my 4GB laptop:

# github.com/google/syzkaller/sys/linux/gen
fatal error: runtime: out of memory

Is the memory requirement of go vet resonable or something worth looking
into? Can go vet be completely disabled for certain directories?

[Dmitry Vyukov]

Hard to say. If there are any low hanging fruits, reducing memory
consumption is always good.

> Can go vet be completely disabled for certain directories?

Perhaps it could skip auto-generated files (there is standard
convention in Go, so it can detect them). But this is something to
discuss with the Go team.
I did not find any other way to ignore files/directories.

[Dmitry Vyukov]

Well, I think this particular one is actually a plain bug in syzkaller
(episodic infinite loop in randGen.flags).

The 2 last commits of https://github.com/google/syzkaller/pull/2249
should fix it.

#syz invalid