WARNING: refcount bug in smc_release (2)

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: c4f2cbd3 Merge branch '100GbE' of git://git.kernel.org/pub..
git tree: net-next
console output: https://syzkaller.appspot.com/x/log.txt?x=172bf5cae00000
kernel config: https://syzkaller.appspot.com/x/.config?x=5ac526dc3578c3d3
dashboard link: https://syzkaller.appspot.com/bug?extid=96d3f9ff6a86d37e44c8
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+96d3f9...@syzkaller.appspotmail.com

------------[ cut here ]------------
refcount_t: underflow; use-after-free.
WARNING: CPU: 0 PID: 20712 at lib/refcount.c:190
refcount_sub_and_test_checked lib/refcount.c:190 [inline]
WARNING: CPU: 0 PID: 20712 at lib/refcount.c:190
refcount_sub_and_test_checked+0x1d0/0x200 lib/refcount.c:180
Kernel panic - not syncing: panic_on_warn set ...
CPU: 0 PID: 20712 Comm: syz-executor.0 Not tainted 5.4.0-rc8-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x197/0x210 lib/dump_stack.c:118
panic+0x2e3/0x75c kernel/panic.c:221
__warn.cold+0x2f/0x35 kernel/panic.c:582
report_bug+0x289/0x300 lib/bug.c:195
fixup_bug arch/x86/kernel/traps.c:179 [inline]
fixup_bug arch/x86/kernel/traps.c:174 [inline]
do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:272
do_invalid_op+0x37/0x50 arch/x86/kernel/traps.c:291
invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1028
RIP: 0010:refcount_sub_and_test_checked lib/refcount.c:190 [inline]
RIP: 0010:refcount_sub_and_test_checked+0x1d0/0x200 lib/refcount.c:180
Code: 1d fb 91 7d 06 31 ff 89 de e8 6c f1 2d fe 84 db 75 94 e8 23 f0 2d fe
48 c7 c7 00 d9 e6 87 c6 05 db 91 7d 06 01 e8 b8 22 ff fd <0f> 0b e9 75 ff
ff ff e8 04 f0 2d fe e9 6e ff ff ff 48 89 df e8 17
RSP: 0018:ffff88807446fa80 EFLAGS: 00010286
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 000000000000f9af RSI: ffffffff815d2076 RDI: ffffed100e88df42
RBP: ffff88807446fb18 R08: ffff888070648580 R09: ffffed1015d04101
R10: ffffed1015d04100 R11: ffff8880ae820807 R12: 00000000ffffffff
R13: 0000000000000001 R14: ffff88807446faf0 R15: 0000000000000000
refcount_dec_and_test_checked+0x1b/0x20 lib/refcount.c:220
sock_put include/net/sock.h:1728 [inline]
smc_release+0x236/0x3e0 net/smc/af_smc.c:202
__sock_release+0xce/0x280 net/socket.c:590
sock_close+0x1e/0x30 net/socket.c:1268
__fput+0x2ff/0x890 fs/file_table.c:280
____fput+0x16/0x20 fs/file_table.c:313
task_work_run+0x145/0x1c0 kernel/task_work.c:113
get_signal+0x2078/0x2500 kernel/signal.c:2528
do_signal+0x87/0x1700 arch/x86/kernel/signal.c:815
exit_to_usermode_loop+0x286/0x380 arch/x86/entry/common.c:159
prepare_exit_to_usermode arch/x86/entry/common.c:194 [inline]
syscall_return_slowpath arch/x86/entry/common.c:274 [inline]
do_syscall_64+0x65f/0x760 arch/x86/entry/common.c:300
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x45a639
Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff
ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f1a52923c78 EFLAGS: 00000246 ORIG_RAX: 000000000000002a
RAX: ffffffffffffff8d RBX: 0000000000000003 RCX: 000000000045a639
RDX: 0000000000000010 RSI: 0000000020000000 RDI: 0000000000000003
RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f1a529246d4
R13: 00000000004c0dc8 R14: 00000000004d3a00 R15: 00000000ffffffff
Kernel Offset: disabled
Rebooting in 86400 seconds..


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[syzbot]

syzbot has found a reproducer for the following crash on:

HEAD commit: 81b6b964 Merge branch 'master' of git://git.kernel.org/pub..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=15ce0abce00000
kernel config: https://syzkaller.appspot.com/x/.config?x=333b76551307b2a0

dashboard link: https://syzkaller.appspot.com/bug?extid=96d3f9ff6a86d37e44c8
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=175a767ee00000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+96d3f9...@syzkaller.appspotmail.com

------------[ cut here ]------------
refcount_t: underflow; use-after-free.

WARNING: CPU: 1 PID: 9419 at lib/refcount.c:28
refcount_warn_saturate+0x1dc/0x1f0 lib/refcount.c:28

Kernel panic - not syncing: panic_on_warn set ...

CPU: 1 PID: 9419 Comm: syz-executor.0 Not tainted 5.4.0-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x197/0x210 lib/dump_stack.c:118
panic+0x2e3/0x75c kernel/panic.c:221

__warn.cold+0x2f/0x3e kernel/panic.c:582
report_bug+0x289/0x300 lib/bug.c:195
fixup_bug arch/x86/kernel/traps.c:174 [inline]
fixup_bug arch/x86/kernel/traps.c:169 [inline]
do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:267
do_invalid_op+0x37/0x50 arch/x86/kernel/traps.c:286
invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1027
RIP: 0010:refcount_warn_saturate+0x1dc/0x1f0 lib/refcount.c:28
Code: e9 d8 fe ff ff 48 89 df e8 31 65 25 fe e9 85 fe ff ff e8 07 37 e8 fd
48 c7 c7 60 53 4f 88 c6 05 7d b6 a5 06 01 e8 73 eb b8 fd <0f> 0b e9 ac fe
ff ff 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48
RSP: 0018:ffff88808963fd50 EFLAGS: 00010286

RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000

RDX: 0000000000000000 RSI: ffffffff815e4316 RDI: ffffed10112c7f9c
RBP: ffff88808963fd60 R08: ffff8880977a64c0 R09: ffffed1015d26621
R10: ffffed1015d26620 R11: ffff8880ae933107 R12: 0000000000000003
R13: 0000000000000000 R14: ffff8880965c0100 R15: ffff888094244a98
refcount_sub_and_test include/linux/refcount.h:261 [inline]
refcount_dec_and_test include/linux/refcount.h:281 [inline]
sock_put include/net/sock.h:1728 [inline]
smc_release+0x445/0x520 net/smc/af_smc.c:202
__sock_release+0xce/0x280 net/socket.c:591
sock_close+0x1e/0x30 net/socket.c:1269

__fput+0x2ff/0x890 fs/file_table.c:280
____fput+0x16/0x20 fs/file_table.c:313
task_work_run+0x145/0x1c0 kernel/task_work.c:113

tracehook_notify_resume include/linux/tracehook.h:188 [inline]
exit_to_usermode_loop+0x316/0x380 arch/x86/entry/common.c:164
prepare_exit_to_usermode arch/x86/entry/common.c:195 [inline]
syscall_return_slowpath arch/x86/entry/common.c:278 [inline]
do_syscall_64+0x676/0x790 arch/x86/entry/common.c:304
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x414211
Code: 75 14 b8 03 00 00 00 0f 05 48 3d 01 f0 ff ff 0f 83 04 1b 00 00 c3 48
83 ec 08 e8 0a fc ff ff 48 89 04 24 b8 03 00 00 00 0f 05 <48> 8b 3c 24 48
89 c2 e8 53 fc ff ff 48 89 d0 48 83 c4 08 48 3d 01
RSP: 002b:00007ffc46c3b260 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
RAX: 0000000000000000 RBX: 0000000000000005 RCX: 0000000000414211
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
RBP: 0000000000000000 R08: ffffffffffffffff R09: ffffffffffffffff
R10: 00007ffc46c3b340 R11: 0000000000000293 R12: 000000000075bfc8
R13: 00000000000ee743 R14: 0000000000760458 R15: 000000000075bfd4

[syzbot]

syzbot has found a reproducer for the following crash on:

HEAD commit: 32ef9553 Merge tag 'fsnotify_for_v5.5-rc1' of git://git.ke..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=15f6d82ae00000
kernel config: https://syzkaller.appspot.com/x/.config?x=ff560c3de405258c

dashboard link: https://syzkaller.appspot.com/bug?extid=96d3f9ff6a86d37e44c8
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

userspace arch: i386
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=14b57336e00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=149e357ae00000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+96d3f9...@syzkaller.appspotmail.com

------------[ cut here ]------------
refcount_t: underflow; use-after-free.

WARNING: CPU: 1 PID: 9807 at lib/refcount.c:28
refcount_warn_saturate+0x1dc/0x1f0 lib/refcount.c:28

Kernel panic - not syncing: panic_on_warn set ...

CPU: 1 PID: 9807 Comm: syz-executor293 Not tainted 5.4.0-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x197/0x210 lib/dump_stack.c:118
panic+0x2e3/0x75c kernel/panic.c:221

__warn.cold+0x2f/0x3e kernel/panic.c:582
report_bug+0x289/0x300 lib/bug.c:195

fixup_bug arch/x86/kernel/traps.c:174 [inline]
fixup_bug arch/x86/kernel/traps.c:169 [inline]
do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:267
do_invalid_op+0x37/0x50 arch/x86/kernel/traps.c:286
invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1027
RIP: 0010:refcount_warn_saturate+0x1dc/0x1f0 lib/refcount.c:28

Code: e9 d8 fe ff ff 48 89 df e8 c1 5a 24 fe e9 85 fe ff ff e8 e7 08 e7 fd
48 c7 c7 a0 6f 4f 88 c6 05 60 b8 a4 06 01 e8 53 bd b7 fd <0f> 0b e9 ac fe
ff ff 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48
RSP: 0018:ffff888093c97998 EFLAGS: 00010286

RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000

RDX: 0000000000000000 RSI: ffffffff815e4316 RDI: ffffed1012792f25
RBP: ffff888093c979a8 R08: ffff8880a04d4380 R09: ffffed1015d26621

R10: ffffed1015d26620 R11: ffff8880ae933107 R12: 0000000000000003

R13: 0000000000000000 R14: ffff8880a118d380 R15: ffff88809427e558

refcount_sub_and_test include/linux/refcount.h:261 [inline]
refcount_dec_and_test include/linux/refcount.h:281 [inline]
sock_put include/net/sock.h:1728 [inline]
smc_release+0x445/0x520 net/smc/af_smc.c:202
__sock_release+0xce/0x280 net/socket.c:591
sock_close+0x1e/0x30 net/socket.c:1269

__fput+0x2ff/0x890 fs/file_table.c:280
____fput+0x16/0x20 fs/file_table.c:313
task_work_run+0x145/0x1c0 kernel/task_work.c:113

exit_task_work include/linux/task_work.h:22 [inline]
do_exit+0x8e7/0x2ef0 kernel/exit.c:797
do_group_exit+0x135/0x360 kernel/exit.c:895
get_signal+0x47c/0x24f0 kernel/signal.c:2734
do_signal+0x87/0x1700 arch/x86/kernel/signal.c:815
exit_to_usermode_loop+0x286/0x380 arch/x86/entry/common.c:160

prepare_exit_to_usermode arch/x86/entry/common.c:195 [inline]
syscall_return_slowpath arch/x86/entry/common.c:278 [inline]

do_syscall_32_irqs_on arch/x86/entry/common.c:352 [inline]
do_fast_syscall_32+0xbbd/0xe16 arch/x86/entry/common.c:408
entry_SYSENTER_compat+0x70/0x7f arch/x86/entry/entry_64_compat.S:139
RIP: 0023:0xf7f69a39
Code: Bad RIP value.
RSP: 002b:00000000f7f441ec EFLAGS: 00000296 ORIG_RAX: 00000000000000f0
RAX: fffffffffffffe00 RBX: 00000000080fb018 RCX: 0000000000000080
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000006
RBP: 0000000020000040 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000

[syzbot]

syzbot has bisected this bug to:

commit 50717a37db032ce783f50685a73bb2ac68471a5a
Author: Ursula Braun <ubr...@linux.ibm.com>
Date: Fri Apr 12 10:57:23 2019 +0000

net/smc: nonblocking connect rework

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=10a234a2e00000
start commit: 32ef9553 Merge tag 'fsnotify_for_v5.5-rc1' of git://git.ke..
git tree: upstream
final crash: https://syzkaller.appspot.com/x/report.txt?x=12a234a2e00000
console output: https://syzkaller.appspot.com/x/log.txt?x=14a234a2e00000

kernel config: https://syzkaller.appspot.com/x/.config?x=ff560c3de405258c
dashboard link: https://syzkaller.appspot.com/bug?extid=96d3f9ff6a86d37e44c8

userspace arch: i386
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=14b57336e00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=149e357ae00000

Reported-by: syzbot+96d3f9...@syzkaller.appspotmail.com
Fixes: 50717a37db03 ("net/smc: nonblocking connect rework")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[Hillf Danton]

On Sat, 30 Nov 2019 20:37:09 -0800
>
> syzbot has found a reproducer for the following crash on:
>
> HEAD commit: 32ef9553 Merge tag 'fsnotify_for_v5.5-rc1' of git://git.ke..
> git tree: upstream

> console output: https://syzkaller.appspot.com/x/log.txt?x=15f6d82ae00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=ff560c3de405258c

> dashboard link: https://syzkaller.appspot.com/bug?extid=96d3f9ff6a86d37e44c8
> compiler: gcc (GCC) 9.0.0 20181231 (experimental)

> userspace arch: i386
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=14b57336e00000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=149e357ae00000
>

> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+96d3f9...@syzkaller.appspotmail.com
>
> ------------[ cut here ]------------
> refcount_t: underflow; use-after-free.

> WARNING: CPU: 1 PID: 9807 at lib/refcount.c:28
> refcount_warn_saturate+0x1dc/0x1f0 lib/refcount.c:28

> Kernel panic - not syncing: panic_on_warn set ...

> CPU: 1 PID: 9807 Comm: syz-executor293 Not tainted 5.4.0-syzkaller #0

> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> Google 01/01/2011
> Call Trace:
> __dump_stack lib/dump_stack.c:77 [inline]
> dump_stack+0x197/0x210 lib/dump_stack.c:118
> panic+0x2e3/0x75c kernel/panic.c:221

> __warn.cold+0x2f/0x3e kernel/panic.c:582
> report_bug+0x289/0x300 lib/bug.c:195

> fixup_bug arch/x86/kernel/traps.c:174 [inline]
> fixup_bug arch/x86/kernel/traps.c:169 [inline]
> do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:267
> do_invalid_op+0x37/0x50 arch/x86/kernel/traps.c:286
> invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1027
> RIP: 0010:refcount_warn_saturate+0x1dc/0x1f0 lib/refcount.c:28
> Code: e9 d8 fe ff ff 48 89 df e8 c1 5a 24 fe e9 85 fe ff ff e8 e7 08 e7 fd
> 48 c7 c7 a0 6f 4f 88 c6 05 60 b8 a4 06 01 e8 53 bd b7 fd <0f> 0b e9 ac fe
> ff ff 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48

> RSP: 0018:ffff888093c97998 EFLAGS: 00010286

> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000

> RDX: 0000000000000000 RSI: ffffffff815e4316 RDI: ffffed1012792f25
> RBP: ffff888093c979a8 R08: ffff8880a04d4380 R09: ffffed1015d26621
> R10: ffffed1015d26620 R11: ffff8880ae933107 R12: 0000000000000003
> R13: 0000000000000000 R14: ffff8880a118d380 R15: ffff88809427e558
> refcount_sub_and_test include/linux/refcount.h:261 [inline]
> refcount_dec_and_test include/linux/refcount.h:281 [inline]
> sock_put include/net/sock.h:1728 [inline]
> smc_release+0x445/0x520 net/smc/af_smc.c:202
> __sock_release+0xce/0x280 net/socket.c:591
> sock_close+0x1e/0x30 net/socket.c:1269

> __fput+0x2ff/0x890 fs/file_table.c:280
> ____fput+0x16/0x20 fs/file_table.c:313
> task_work_run+0x145/0x1c0 kernel/task_work.c:113

> exit_task_work include/linux/task_work.h:22 [inline]
> do_exit+0x8e7/0x2ef0 kernel/exit.c:797
> do_group_exit+0x135/0x360 kernel/exit.c:895
> get_signal+0x47c/0x24f0 kernel/signal.c:2734

> do_signal+0x87/0x1700 arch/x86/kernel/signal.c:815

> exit_to_usermode_loop+0x286/0x380 arch/x86/entry/common.c:160
> prepare_exit_to_usermode arch/x86/entry/common.c:195 [inline]
> syscall_return_slowpath arch/x86/entry/common.c:278 [inline]
> do_syscall_32_irqs_on arch/x86/entry/common.c:352 [inline]
> do_fast_syscall_32+0xbbd/0xe16 arch/x86/entry/common.c:408
> entry_SYSENTER_compat+0x70/0x7f arch/x86/entry/entry_64_compat.S:139

Prevent repeated release using cmpxchg and the sock_hold/put pair is
cut off as a bonus cleanup (which would go in another seperate one if
necessary).

--- a/net/smc/af_smc.c
+++ b/net/smc/af_smc.c
@@ -172,10 +172,9 @@ static int smc_release(struct socket *so
struct smc_sock *smc;
int rc = 0;

- if (!sk)
- goto out;
+ if (!sk || sk != cmpxchg(&sock->sk, sk, NULL))
+ return 0;

- sock_hold(sk); /* sock_put below */
smc = smc_sk(sk);

/* cleanup for a dangling non-blocking connect */
@@ -198,9 +197,7 @@ static int smc_release(struct socket *so
sock->sk = NULL;
release_sock(sk);

- sock_put(sk); /* sock_hold above */
sock_put(sk); /* final sock_put */
-out:
return rc;
}

[Ursula Braun]

Thanks, Hilff, for this cmpxchg() idea. We keep it in mind, but analyzing
possible scenarios of the C reproducer I detected an errorneous duplicate
refcount decrease possibility for the combination of non-blocking connect()
and FASTOPEN_KEY setsockopt(). I am working on a fix.

Thus I assume the syzbot problem is not caused by a repeated release.

Kind regards, Ursula