general protection fault in process_one_work

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: 38207291 bpf: Prepare btf_ctx_access for non raw_tp use case
git tree: bpf-next
console output: https://syzkaller.appspot.com/x/log.txt?x=14173c0f600000
kernel config: https://syzkaller.appspot.com/x/.config?x=41648156aa09be10
dashboard link: https://syzkaller.appspot.com/bug?extid=9ed8f68ab30761f3678e
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+9ed8f6...@syzkaller.appspotmail.com

kasan: CONFIG_KASAN_INLINE enabled
kasan: GPF could be caused by NULL-ptr deref or user memory access
general protection fault: 0000 [#1] PREEMPT SMP KASAN
CPU: 1 PID: 9149 Comm: kworker/1:3 Not tainted 5.4.0-rc1+ #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Workqueue: events nsim_dev_trap_report_work
RIP: 0010:nsim_dev_trap_report_work+0xc4/0xaf0
drivers/net/netdevsim/dev.c:409
Code: 89 45 d0 0f 84 8b 07 00 00 49 bc 00 00 00 00 00 fc ff df e8 3e ae ef
fc 48 8b 45 d0 48 05 68 01 00 00 48 89 45 90 48 c1 e8 03 <42> 80 3c 20 00
0f 85 b1 09 00 00 48 8b 45 d0 48 8b 98 68 01 00 00
RSP: 0018:ffff88806c98fc90 EFLAGS: 00010a06
RAX: 1bd5a0000000004d RBX: 0000000000000000 RCX: ffffffff84836e22
RDX: 0000000000000000 RSI: ffffffff84836db2 RDI: 0000000000000001
RBP: ffff88806c98fd30 R08: ffff88806c9863c0 R09: ffffed100d75f3d9
R10: ffffed100d75f3d8 R11: ffff88806baf9ec7 R12: dffffc0000000000
R13: ffff88806baf9ec0 R14: ffff8880a9a13900 R15: ffff8880ae934500
FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007efdd0c9e000 CR3: 000000009cc1b000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
process_one_work+0x9af/0x1740 kernel/workqueue.c:2269
worker_thread+0x98/0xe40 kernel/workqueue.c:2415
kthread+0x361/0x430 kernel/kthread.c:255
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
Modules linked in:
---[ end trace ba29cd1c27f63d86 ]---
RIP: 0010:nsim_dev_trap_report_work+0xc4/0xaf0
drivers/net/netdevsim/dev.c:409
Code: 89 45 d0 0f 84 8b 07 00 00 49 bc 00 00 00 00 00 fc ff df e8 3e ae ef
fc 48 8b 45 d0 48 05 68 01 00 00 48 89 45 90 48 c1 e8 03 <42> 80 3c 20 00
0f 85 b1 09 00 00 48 8b 45 d0 48 8b 98 68 01 00 00
RSP: 0018:ffff88806c98fc90 EFLAGS: 00010a06
RAX: 1bd5a0000000004d RBX: 0000000000000000 RCX: ffffffff84836e22
RDX: 0000000000000000 RSI: ffffffff84836db2 RDI: 0000000000000001
RBP: ffff88806c98fd30 R08: ffff88806c9863c0 R09: ffffed100d75f3d9
R10: ffffed100d75f3d8 R11: ffff88806baf9ec7 R12: dffffc0000000000
R13: ffff88806baf9ec0 R14: ffff8880a9a13900 R15: ffff8880ae934500
FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007efdd0c9e000 CR3: 000000009cc1b000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[Dmitry Vyukov]

On Tue, Oct 29, 2019 at 9:38 AM syzbot
<syzbot+9ed8f6...@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following crash on:
>
> HEAD commit: 38207291 bpf: Prepare btf_ctx_access for non raw_tp use case
> git tree: bpf-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=14173c0f600000
> kernel config: https://syzkaller.appspot.com/x/.config?x=41648156aa09be10
> dashboard link: https://syzkaller.appspot.com/bug?extid=9ed8f68ab30761f3678e
> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
>
> Unfortunately, I don't have any reproducer for this crash yet.
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+9ed8f6...@syzkaller.appspotmail.com

+Jiří Pírko, this seems to be related to netdevsim.

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/0000000000001c46d5059608892f%40google.com.

[Ido Schimmel]

On Tue, Oct 29, 2019 at 09:43:27AM +0100, Dmitry Vyukov wrote:
> On Tue, Oct 29, 2019 at 9:38 AM syzbot
> <syzbot+9ed8f6...@syzkaller.appspotmail.com> wrote:
> >
> > Hello,
> >
> > syzbot found the following crash on:
> >
> > HEAD commit: 38207291 bpf: Prepare btf_ctx_access for non raw_tp use case
> > git tree: bpf-next
> > console output: https://syzkaller.appspot.com/x/log.txt?x=14173c0f600000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=41648156aa09be10
> > dashboard link: https://syzkaller.appspot.com/bug?extid=9ed8f68ab30761f3678e
> > compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> >
> > Unfortunately, I don't have any reproducer for this crash yet.
> >
> > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > Reported-by: syzbot+9ed8f6...@syzkaller.appspotmail.com
>
> +Jiří Pírko, this seems to be related to netdevsim.

Will check it.

[Ido Schimmel]

On Tue, Oct 29, 2019 at 10:45:19AM +0200, Ido Schimmel wrote:
> On Tue, Oct 29, 2019 at 09:43:27AM +0100, Dmitry Vyukov wrote:
> > On Tue, Oct 29, 2019 at 9:38 AM syzbot
> > <syzbot+9ed8f6...@syzkaller.appspotmail.com> wrote:
> > >
> > > Hello,
> > >
> > > syzbot found the following crash on:
> > >
> > > HEAD commit: 38207291 bpf: Prepare btf_ctx_access for non raw_tp use case
> > > git tree: bpf-next
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=14173c0f600000
> > > kernel config: https://syzkaller.appspot.com/x/.config?x=41648156aa09be10
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=9ed8f68ab30761f3678e
> > > compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> > >
> > > Unfortunately, I don't have any reproducer for this crash yet.
> > >
> > > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > > Reported-by: syzbot+9ed8f6...@syzkaller.appspotmail.com
> >
> > +Jiří Pírko, this seems to be related to netdevsim.
>
> Will check it.

Didn't have a lot of time today, but I think the issue is a race
condition (note that syzbot only triggered it twice so far).

Upon reload nsim_dev_port_del_all() is called and starts deleting ports
from the ports list without holding the ports list mutex. It is possible
that during this time nsim_dev_trap_report_work() is executing from a
workqueue and accessing freed memory despite holding the ports list
mutex.

I'll try to reproduce and send a fix later this week.

[Ido Schimmel]

Sent a fix:
https://patchwork.ozlabs.org/patch/1187587/

It was quite difficult to reproduce, so I used the below patch to
increase the time window in which the race could occur. Then it became
trivial to trigger :)

diff --git a/drivers/net/netdevsim/dev.c b/drivers/net/netdevsim/dev.c
index 54ca6681ba31..d12abd84c218 100644
--- a/drivers/net/netdevsim/dev.c
+++ b/drivers/net/netdevsim/dev.c
@@ -457,6 +457,7 @@ static void nsim_dev_trap_report_work(struct work_struct *work)
*/
mutex_lock(&nsim_dev->port_list_lock);
list_for_each_entry(nsim_dev_port, &nsim_dev->port_list, list) {
+ msleep(100);
if (!netif_running(nsim_dev_port->ns->netdev))
continue;