assert failed: c->c_magic == CALLOUT_MAGIC

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: af876e934dda Register kUBSan and KCOV in kernel=GENERIC of..
git tree: netbsd
console output: https://syzkaller.appspot.com/x/log.txt?x=154a6a70c00000
dashboard link: https://syzkaller.appspot.com/bug?extid=ed255d35c44b65bc18a1

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+ed255d...@syzkaller.appspotmail.com

[ 132.9508779] panic: kernel diagnostic assertion "c->c_magic ==
CALLOUT_MAGIC" failed:
file "/syzkaller/managers/netbsd/kernel/sys/kern/kern_timeout.c", line 474
[ 132.9508779] cpu0: Begin traceback...
[ 132.9508779] vpanic() at netbsd:vpanic+0x214
[ 132.9508779] _GLOBAL__sub_D_65535_0_cpu_configure() at
netbsd:_GLOBAL__sub_D_65535_0_cpu_configure
[ 132.9508779] callout_halt() at netbsd:callout_halt+0x327
[ 132.9508779] timer_settime() at netbsd:timer_settime+0x41c
[ 132.9508779] dosetitimer() at netbsd:dosetitimer+0x3eb
[ 132.9508779] sys___setitimer50() at netbsd:sys___setitimer50+0x127
[ 132.9508779] sys___syscall() at netbsd:sys___syscall+0xe2
[ 132.9508779] syscall() at netbsd:syscall+0x30e
[ 132.9508779] --- syscall (number 198) ---
[ 132.9508779] 794c62a3f4aa:
[ 132.9508779] cpu0: End traceback...

[ 132.9508779] dumping to dev 4,1 (offset=0, size=0): not possible
[ 132.9508779] rebooting...
SeaBIOS (version 1.8.2-20181029_212248-google)
Total RAM Size = 0x00000001e0000000 = 7680 MiB
CPUs found: 2 Max CPUs supported: 2
found virtio-scsi at 0:3
virtio-scsi vendor='Google' product='PersistentDisk' rev='1' type=0
removable=0
virtio-scsi blksize=512 sectors=4194304 = 2048 MiB
drive 0x000f2a00: PCHS=0/0/0 translation=lba LCHS=520/128/63 s=4194304
Booting from Hard Disk 0...

>> NetBSD/x86 BIOS Boot, Revision 5.10 (Tue Jul 17 14:59:51 UTC 2018) (from
>> NetBSD 8.0)
>> Memory: 639/3144640 k

1. Boot normally
2. Boot single user
3. Disable ACPI
4. Disable ACPI and SMP
5. Drop to boot prompt
| / - \ | / - 36967632\ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
+2878256/ - \ | [1062116/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ +1363008| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ +1044773| / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | ]=0x294fc40
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | WARNING: couldn't
open /var/db/entropy-file
WARNING: 1 module failed to load


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
syzbot.

[Dmitry Vyukov]

Congrats on the first one!

Siddharth, now you will have work :)

Is it possible to make kernel not reboot after crashes? That reboot
output is nasty.
Alternatively, we could strip output on "SeaBIOS" in pkg/report.

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-netbsd-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-netbsd...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-netbsd-bugs/00000000000066549b058293e541%40google.com.
> For more options, visit https://groups.google.com/d/optout.

[Kamil Rytarowski]

Thank you for all your work! Fantastic results!

> Siddharth, now you will have work :)
>
> Is it possible to make kernel not reboot after crashes? That reboot
> output is nasty.

Yes. Actually it would be useful to do in the prepared VM image for fuzzing:

echo ddb.onpanic=1 >> /etc/sysct.conf # and reboot

Alternatively setup it live: sysctl -w ddb.onpanic=1

This way we will enter into a kernel debugger prompt on panic (but not
every kernel crash, e.g. not for triplefault).

In the debugger we can emit "bt" for backtrace as dmesg(8) one can be
mixed with something and not readable.

We can also print panic string from ddb(4) with "show panic".

Later we can just kill VM manually.

I can introduce a dedicated sysctl for Kernel Sanitizers like
kern.panic_on_sanitizer_report (what would be a better name?) with
default value 0 and switchable to 1. It would enter ddb(4) and syzkaller
would emit "bt" and "show panic" to get well-formed messages for scraping.

> Alternatively, we could strip output on "SeaBIOS" in pkg/report.
>

I propose to go for ddb(4) printing of backtrace, as it is more convenient.

[Siddharth Muralee]

Great :)

I will modify pkg/build to add the changes that we would like to make.

I have got KASan and KCov working together.

We still have a small issue in Kcov that we need to fix.

--

Regards, 

  Siddharth M
  Third Year B.Tech (CSE) Student,
  Amrita School of Engineering, Kollam

  Blog 

---------------------------------------

“Most people get ahead during the time that others waste."

[Kamil Rytarowski]

How to get a reproducer for this issue?

We have got a potential fix.

Similar with other reports.

[syzbot]

syzbot has found a reproducer for the following crash on:

HEAD commit: dc893675b200 Hook spi.
git tree: netbsd
console output: https://syzkaller.appspot.com/x/log.txt?x=107a8292c00000
dashboard link: https://syzkaller.appspot.com/bug?extid=ed255d35c44b65bc18a1
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13488b82c00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=134ca30ac00000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+ed255d...@syzkaller.appspotmail.com

[ 36.0260844] panic: kernel diagnostic assertion "c->c_magic ==

CALLOUT_MAGIC" failed:
file "/syzkaller/managers/netbsd/kernel/sys/kern/kern_timeout.c", line 474

[ 36.0260844] cpu0: Begin traceback...
[ 36.0260844] vpanic() at netbsd:vpanic+0x214
[ 36.0260844] _GLOBAL__sub_D_65535_0_cpu_configure() at
netbsd:_GLOBAL__sub_D_65535_0_cpu_configure
[ 36.0260844] callout_halt() at netbsd:callout_halt+0x327
[ 36.0260844] timer_settime() at netbsd:timer_settime+0x41c
[ 36.0260844] dosetitimer() at netbsd:dosetitimer+0x3eb
[ 36.0260844] sys___setitimer50() at netbsd:sys___setitimer50+0x127
[ 36.0260844] sys_syscall() at netbsd:sys_syscall+0xe2
[ 36.0260844] syscall() at netbsd:syscall+0x30e
[ 36.0260844] --- syscall (number 0) ---
[ 36.0260844] 752e1c63f4ca:
[ 36.0260844] cpu0: End traceback...

[ 36.0260844] dumping to dev 4,1 (offset=0, size=0): not possible
[ 36.0260844] rebooting...

[maxv]

Fixed today, but the Reported-by tag was not set properly. Let's see if this works:

#syz fix: The callout is used by any nonvirtual timer including CLOCK_MONOTONIC and needs to be initialized.

[maxv]

Retry with the correct dest:

[Dmitry Vyukov]

syzkaller extracts them automatically, but it takes time and not
always succeeds (e.g. a subtle race, accumulated state, interaction
between several tests, etc).

[Dmitry Vyukov]

FWIW OpenBSD does this on crashes:
https://github.com/google/syzkaller/blob/4d7696cb3118457aa571c33e7cf50e0534c27c59/vm/vmimpl/openbsd.go#L16-L28
Something similar may be useful for NetBSD too.

Siddharth, are you keeping track of all these things (also some
comments on other bugs)? Perhaps file issues at
https://github.com/google/syzkaller/issues so that nothing is lost.

[Siddharth Muralee]

That's a good idea - I will do that. 

I am trying to see what all modifications I can make to sys/netbsd - so I am trying to learn and understand the psuedo grammar thing that you have implemented.

[Dmitry Vyukov]

On Mon, Feb 25, 2019 at 12:37 PM maxv <m...@m00nbsd.net> wrote:
>
> Retry with the correct dest:
>
> #syz fix: The callout is used by any nonvirtual timer including CLOCK_MONOTONIC and needs to be initialized.

This looks like a correct commit title.
But what your email client actually sent is:

#syz fix: The callout is used by any nonvirtual timer including
CLOCK_MONOTONIC and needs to be initialized.

This line split is an unrecoverable corruption that is not possible to
restore to original form.
Yes, I know it's Gmail. Gmail does not allow to send proper text emails.
Now we need to find somebody with email client that allows to send
text emails with lines longer than 80 cols :)

[syzbot]

> On Mon, Feb 25, 2019 at 12:37 PM maxv <m...@m00nbsd.net> wrote:

>> Retry with the correct dest:

>> #syz fix: The callout is used by any nonvirtual timer including
>> CLOCK_MONOTONIC and needs to be initialized.


> This looks like a correct commit title.
> But what your email client actually sent is:

> #syz fix: The callout is used by any nonvirtual timer including

Can't find the corresponding bug.

> CLOCK_MONOTONIC and needs to be initialized.

> This line split is an unrecoverable corruption that is not possible to
> restore to original form.
> Yes, I know it's Gmail. Gmail does not allow to send proper text emails.
> Now we need to find somebody with email client that allows to send
> text emails with lines longer than 80 cols :)

> --
> You received this message because you are subscribed to the Google
> Groups "syzkaller-netbsd-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to syzkaller-netbsd...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/syzkaller-netbsd-bugs/CACT4Y%2BaWhUjuaxh7skaxoNMEWChL%2BVKNmU38N-EAydT%2BcPNe4Q%40mail.gmail.com.

[Maxime Villard]

Retry again, without wrap this time: