kernel panic: corrupted stack end in dput

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: 7b75e49d net: dsa: mv88e6xxx: wait after reset deactivation
git tree: net
console output: https://syzkaller.appspot.com/x/log.txt?x=10f51b13a00000
kernel config: https://syzkaller.appspot.com/x/.config?x=e7c31a94f66cc0aa
dashboard link: https://syzkaller.appspot.com/bug?extid=d88a977731a9888db7ba
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=130f49bda00000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+d88a97...@syzkaller.appspotmail.com

Kernel panic - not syncing: corrupted stack end detected inside scheduler
CPU: 1 PID: 8936 Comm: syz-executor.3 Not tainted 5.2.0-rc6+ #69
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x172/0x1f0 lib/dump_stack.c:113
panic+0x2cb/0x744 kernel/panic.c:219
schedule_debug kernel/sched/core.c:3272 [inline]
__schedule+0x155d/0x1560 kernel/sched/core.c:3381
preempt_schedule_notrace kernel/sched/core.c:3664 [inline]
preempt_schedule_notrace+0xa0/0x130 kernel/sched/core.c:3635
___preempt_schedule_notrace+0x16/0x2f
rcu_is_watching+0x23/0x30 kernel/rcu/tree.c:873
rcu_read_lock include/linux/rcupdate.h:594 [inline]
dput+0x41e/0x690 fs/dcache.c:845
__fput+0x424/0x890 fs/file_table.c:293
____fput+0x16/0x20 fs/file_table.c:313
task_work_run+0x145/0x1c0 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:185 [inline]
exit_to_usermode_loop+0x273/0x2c0 arch/x86/entry/common.c:168
prepare_exit_to_usermode arch/x86/entry/common.c:199 [inline]
syscall_return_slowpath arch/x86/entry/common.c:279 [inline]
do_syscall_64+0x58e/0x680 arch/x86/entry/common.c:304
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x413201
Code: 75 14 b8 03 00 00 00 0f 05 48 3d 01 f0 ff ff 0f 83 04 1b 00 00 c3 48
83 ec 08 e8 0a fc ff ff 48 89 04 24 b8 03 00 00 00 0f 05 <48> 8b 3c 24 48
89 c2 e8 53 fc ff ff 48 89 d0 48 83 c4 08 48 3d 01
RSP: 002b:00007ffcee6f8e20 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
RAX: 0000000000000000 RBX: 0000000000000006 RCX: 0000000000413201
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000005
RBP: 0000000000000001 R08: ffffffffffffffff R09: ffffffffffffffff
R10: 00007ffcee6f8f00 R11: 0000000000000293 R12: 000000000075c9a0
R13: 000000000075c9a0 R14: 00000000000127ed R15: ffffffffffffffff
Shutting down cpus with NMI
Kernel Offset: disabled
Rebooting in 86400 seconds..


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches

[syzbot]

syzbot has bisected this bug to:

commit e9db4ef6bf4ca9894bb324c76e01b8f1a16b2650
Author: John Fastabend <john.fa...@gmail.com>
Date: Sat Jun 30 13:17:47 2018 +0000

bpf: sockhash fix omitted bucket lock in sock_close

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=17f7f4d5a00000
start commit: 7b75e49d net: dsa: mv88e6xxx: wait after reset deactivation
git tree: net
final crash: https://syzkaller.appspot.com/x/report.txt?x=140ff4d5a00000
console output: https://syzkaller.appspot.com/x/log.txt?x=100ff4d5a00000

kernel config: https://syzkaller.appspot.com/x/.config?x=e7c31a94f66cc0aa
dashboard link: https://syzkaller.appspot.com/bug?extid=d88a977731a9888db7ba

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=130f49bda00000

Reported-by: syzbot+d88a97...@syzkaller.appspotmail.com
Fixes: e9db4ef6bf4c ("bpf: sockhash fix omitted bucket lock in sock_close")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[Al Viro]

On Tue, Jul 02, 2019 at 05:21:26PM +0800, Hillf Danton wrote:

> Hello,

> --- a/fs/dcache.c
> +++ b/fs/dcache.c
> @@ -673,14 +673,11 @@ static struct dentry *dentry_kill(struct dentry *dentry)
> if (!IS_ROOT(dentry)) {
> parent = dentry->d_parent;
> if (unlikely(!spin_trylock(&parent->d_lock))) {
> - parent = __lock_parent(dentry);
> - if (likely(inode || !dentry->d_inode))
> - goto got_locks;
> - /* negative that became positive */
> - if (parent)
> - spin_unlock(&parent->d_lock);
> - inode = dentry->d_inode;
> - goto slow_positive;
> + /*
> + * fine if peer is busy either populating or
> + * cleaning up parent
> + */
> + parent = NULL;
> }
> }
> __dentry_kill(dentry);

This is very much *NOT* fine.
1) trylock can fail from any number of reasons, starting
with "somebody is going through the hash chain doing a lookup on
something completely unrelated"
2) whoever had been holding the lock and whatever they'd
been doing might be over right after we get the return value from
spin_trylock().
3) even had that been really somebody adding children in
the same parent *AND* even if they really kept doing that, rather
than unlocking and buggering off, would you care to explain why
dentry_unlist() called by __dentry_kill() and removing the victim
from the list of children would be safe to do in parallel with that?

NAK, in case it's not obvious from the above.

[Hillf Danton]

Hello Al

They are also a red light that we need to bail out of spiraling up
the directory hierarchy imho.

> 2) whoever had been holding the lock and whatever they'd
> been doing might be over right after we get the return value from
> spin_trylock().

Or after we send a mail using git. I don't know.

> 3) even had that been really somebody adding children in
> the same parent *AND* even if they really kept doing that, rather
> than unlocking and buggering off, would you care to explain why
> dentry_unlist() called by __dentry_kill() and removing the victim
> from the list of children would be safe to do in parallel with that?
>

My bad. I have to walk around that unsafety.

> NAK, in case it's not obvious from the above.
>

An update can perhaps turn the three letters into EAGAIN.

Hillf
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -666,6 +666,7 @@ static struct dentry *dentry_kill(struct dentry *dentry)
{
struct inode *inode = dentry->d_inode;
struct dentry *parent = NULL;
+ bool red_light = false;

if (inode && unlikely(!spin_trylock(&inode->i_lock)))
goto slow_positive;
@@ -680,6 +681,7 @@ static struct dentry *dentry_kill(struct dentry *dentry)
if (parent)
spin_unlock(&parent->d_lock);
inode = dentry->d_inode;
+ red_light = true;
goto slow_positive;
}
}
@@ -696,6 +698,8 @@ got_locks:
dentry->d_lockref.count--;
} else if (likely(!retain_dentry(dentry))) {
__dentry_kill(dentry);
+ if (red_light)
+ return NULL;
return parent;
}
/* we are keeping it, after all */
--

[Al Viro]

On Wed, Jul 03, 2019 at 02:43:07PM +0800, Hillf Danton wrote:

> > This is very much *NOT* fine.
> > 1) trylock can fail from any number of reasons, starting
> > with "somebody is going through the hash chain doing a lookup on
> > something completely unrelated"
>
> They are also a red light that we need to bail out of spiraling up
> the directory hierarchy imho.

Translation: "let's leak the reference to parent, shall we?"

> > 2) whoever had been holding the lock and whatever they'd
> > been doing might be over right after we get the return value from
> > spin_trylock().
>
> Or after we send a mail using git. I don't know.
>
> > 3) even had that been really somebody adding children in
> > the same parent *AND* even if they really kept doing that, rather
> > than unlocking and buggering off, would you care to explain why
> > dentry_unlist() called by __dentry_kill() and removing the victim
> > from the list of children would be safe to do in parallel with that?
> >
> My bad. I have to walk around that unsafety.

WHAT unsafety? Can you explain what are you seeing and how to
reproduce it, whatever it is?

[Al Viro]

BTW, what makes you think that it's something inside dput() itself?
All I see is that at some point in the beginning of the loop body
in dput() we observe a buggered stack.

Is that the first iteration through the loop? IOW, is that just
the place where we first notice preexisting corruption, or is
that something the code called from that loop does? If it's
a stack overflow, I would be very surprised to see it here -
dput() is iterative and it's called on a very shallow stack in
those traces.

What happens if you e.g. turn that
dput(dentry);
in __fput() into
rcu_read_lock(); rcu_read_unlock(); // trigger the check
dput(dentry);

and run your reporducer?

[Eric Biggers]

[+bpf and tls maintainers]

Please don't waste your time on this, it looks like just another report from the
massive memory corruption in BPF and/or TLS. Look at reproducer:

bpf$MAP_CREATE(0x0, &(0x7f0000000280)={0xf, 0x4, 0x4, 0x400, 0x0, 0x1}, 0x3c)
socket$rxrpc(0x21, 0x2, 0x800000000a)
r0 = socket$inet6_tcp(0xa, 0x1, 0x0)
setsockopt$inet6_tcp_int(r0, 0x6, 0x13, &(0x7f00000000c0)=0x100000001, 0x1d4)
connect$inet6(r0, &(0x7f0000000140), 0x1c)
bpf$MAP_CREATE(0x0, &(0x7f0000000000)={0x5}, 0xfffffffffffffdcb)
bpf$MAP_CREATE(0x2, &(0x7f0000003000)={0x3, 0x0, 0x77fffb, 0x0, 0x10020000000, 0x0}, 0x2c)
setsockopt$inet6_tcp_TCP_ULP(r0, 0x6, 0x1f, &(0x7f0000000040)='tls\x00', 0x4)

It's the same as like 20 other syzbot reports.

- Eric

[John Fastabend]

There is a missing synchronize_rcu we need to add and we have a race
between map_free and tls close at the moment. The race cuases us to
incorrectly set the sk->prot pointers when tls socket is closed in
this case. I've added a hook to the ULP side now that should let
the map_free reset the saved sk->prot pointers on the TLS side and
am testing this now.

The 20 syzbot reports appear to all be due to these two issues.
This has nothing to do with dput().

Thanks,
John

>
> - Eric

[Hillf Danton]

On Wed, Jul 03, 2019 at 03:40:00PM +0100, Al Viro wrote:
>
> BTW, what makes you think that it's something inside dput() itself?
> All I see is that at some point in the beginning of the loop body
> in dput() we observe a buggered stack.
>

I'd have had some idea about that beginning part.

Thank you very much for the light you are casting, Sir.

Hillf

[Eric Biggers]

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/5d1cd47644b45_8ce2b1bd49125c4ed%40john-XPS-13-9370.notmuch.
> For more options, visit https://groups.google.com/d/optout.

#syz fix: bpf: sockmap/tls, close can race with map free