possible deadlock in get_user_pages_unlocked

[syzbot]

Hello,

syzbot hit the following crash on upstream commit
7109a04eae81c41ed529da9f3c48c3655ccea741 (Thu Feb 1 17:37:30 2018 +0000)
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/ide

So far this crash happened 2 times on upstream.
C reproducer is attached.
syzkaller reproducer is attached.
Raw console output is attached.
compiler: gcc (GCC) 7.1.1 20170620
.config is attached.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+bacbe5...@syzkaller.appspotmail.com
It will help syzbot understand when the bug is fixed. See footer for
details.
If you forward the report, please keep this part and the footer.


============================================
WARNING: possible recursive locking detected
4.15.0+ #291 Not tainted
--------------------------------------------
syzkaller746568/4446 is trying to acquire lock:
(&mm->mmap_sem){++++}, at: [<000000006212449e>] __get_user_pages_locked
mm/gup.c:909 [inline]
(&mm->mmap_sem){++++}, at: [<000000006212449e>]
get_user_pages_unlocked+0x218/0x4f0 mm/gup.c:990

but task is already holding lock:
(&mm->mmap_sem){++++}, at: [<000000009bfdb3c4>]
get_user_pages_unlocked+0xff/0x4f0 mm/gup.c:989

other info that might help us debug this:
Possible unsafe locking scenario:

CPU0
----
lock(&mm->mmap_sem);
lock(&mm->mmap_sem);

*** DEADLOCK ***

May be due to missing lock nesting notation

3 locks held by syzkaller746568/4446:
#0: (&vcpu->mutex){+.+.}, at: [<00000000d22fa6cb>] vcpu_load+0x1c/0x70
arch/x86/kvm/../../../virt/kvm/kvm_main.c:158
#1: (&kvm->srcu){....}, at: [<000000007ac6ea14>] vcpu_enter_guest
arch/x86/kvm/x86.c:7064 [inline]
#1: (&kvm->srcu){....}, at: [<000000007ac6ea14>] vcpu_run
arch/x86/kvm/x86.c:7143 [inline]
#1: (&kvm->srcu){....}, at: [<000000007ac6ea14>]
kvm_arch_vcpu_ioctl_run+0x1b11/0x5bf0 arch/x86/kvm/x86.c:7310
#2: (&mm->mmap_sem){++++}, at: [<000000009bfdb3c4>]
get_user_pages_unlocked+0xff/0x4f0 mm/gup.c:989

stack backtrace:
CPU: 0 PID: 4446 Comm: syzkaller746568 Not tainted 4.15.0+ #291
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0x194/0x257 lib/dump_stack.c:53
print_deadlock_bug kernel/locking/lockdep.c:1761 [inline]
check_deadlock kernel/locking/lockdep.c:1805 [inline]
validate_chain kernel/locking/lockdep.c:2401 [inline]
__lock_acquire+0xe8f/0x3e00 kernel/locking/lockdep.c:3431
lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
down_read+0x96/0x150 kernel/locking/rwsem.c:24
__get_user_pages_locked mm/gup.c:909 [inline]
get_user_pages_unlocked+0x218/0x4f0 mm/gup.c:990
hva_to_pfn_slow arch/x86/kvm/../../../virt/kvm/kvm_main.c:1388 [inline]
hva_to_pfn arch/x86/kvm/../../../virt/kvm/kvm_main.c:1493 [inline]
__gfn_to_pfn_memslot+0x386/0xf80
arch/x86/kvm/../../../virt/kvm/kvm_main.c:1549
try_async_pf+0x13b/0xc40 arch/x86/kvm/mmu.c:3802
tdp_page_fault+0x40a/0xa70 arch/x86/kvm/mmu.c:3898
kvm_mmu_page_fault+0x10d/0x2f0 arch/x86/kvm/mmu.c:4928
handle_ept_violation+0x198/0x550 arch/x86/kvm/vmx.c:6550
vmx_handle_exit+0x25d/0x1ce0 arch/x86/kvm/vmx.c:8893
vcpu_enter_guest arch/x86/kvm/x86.c:7081 [inline]
vcpu_run arch/x86/kvm/x86.c:7143 [inline]
kvm_arch_vcpu_ioctl_run+0x1c69/0x5bf0 arch/x86/kvm/x86.c:7310
kvm_vcpu_ioctl+0x64c/0x1010 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2564
vfs_ioctl fs/ioctl.c:46 [inline]
do_vfs_ioctl+0x1b1/0x1520 fs/ioctl.c:686
SYSC_ioctl fs/ioctl.c:701 [inline]
SyS_ioctl+0x8f/0xc0 fs/ioctl.c:692
entry_SYSCALL_64_fastpath+0x29/0xa0
RIP: 0033:0x448fc9
RSP: 002b:00007ff4b260ada8 EFLAGS: 00000297 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007ff4b260b700 RCX: 0000000000448fc9
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000008
RBP: 00007fffca7f9900 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000297 R12: 0000000000000000
R13: 00007fffca7f985f R14: 00007ff4b260b9c0 R15: 000000000000000d
WARNING: CPU: 0 PID: 4446 at mm/gup.c:498 faultin_page mm/gup.c:498 [inline]
WARNING: CPU: 0 PID: 4446 at mm/gup.c:498 __get_user_pages+0x10a7/0x15f0
mm/gup.c:699


---
This bug is generated by a dumb bot. It may contain errors.
See https://goo.gl/tpsmEJ for details.
Direct all questions to syzk...@googlegroups.com.

syzbot will keep track of this bug report.
If you forgot to add the Reported-by tag, once the fix for this bug is
merged
into any tree, please reply to this email with:
#syz fix: exact-commit-title
If you want to test a patch for this bug, please reply with:
#syz test: git://repo/address.git branch
and provide the patch inline or as an attachment.
To mark this as a duplicate of another syzbot report, please reply with:
#syz dup: exact-subject-of-another-report
If it's a one-off invalid bug report, please reply with:
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug
report.
Note: all commands must start from beginning of the line in the email body.

[Al Viro]

On Thu, Feb 01, 2018 at 04:58:00PM -0800, syzbot wrote:
> Hello,
>
> syzbot hit the following crash on upstream commit
> 7109a04eae81c41ed529da9f3c48c3655ccea741 (Thu Feb 1 17:37:30 2018 +0000)
> Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/ide
>
> So far this crash happened 2 times on upstream.
> C reproducer is attached.

Umm... How reproducible that is?

> syzkaller reproducer is attached.
> Raw console output is attached.
> compiler: gcc (GCC) 7.1.1 20170620
> .config is attached.

Can't reproduce with gcc 5.4.1 (same .config, same C reproducer).

It looks like __get_user_pages_locked() returning with *locked zeroed,
but ->mmap_sem not dropped. I don't see what could've lead to it and
attempts to reproduce had not succeeded so far...

How long does it normally take for lockdep splat to trigger?

[Eric Biggers]

Try starting up multiple instances of the program; that sometimes helps with
these races that are hard to hit (since you may e.g. have a different number of
CPUs than syzbot used). If I start up 4 instances I see the lockdep splat after
around 2-5 seconds. This is on latest Linus tree (4bf772b1467). Also note the
reproducer uses KVM, so if you're running it in a VM it will only work if you've
enabled nested virtualization on the host (kvm_intel.nested=1).

Also it appears to go away if I revert ce53053ce378c21 ("kvm: switch
get_user_page_nowait() to get_user_pages_unlocked()").

- Eric

[Al Viro]

On Thu, Feb 01, 2018 at 09:35:02PM -0800, Eric Biggers wrote:

> Try starting up multiple instances of the program; that sometimes helps with
> these races that are hard to hit (since you may e.g. have a different number of
> CPUs than syzbot used). If I start up 4 instances I see the lockdep splat after
> around 2-5 seconds.

5 instances in parallel, 10 minutes into the run...

> This is on latest Linus tree (4bf772b1467). Also note the
> reproducer uses KVM, so if you're running it in a VM it will only work if you've
> enabled nested virtualization on the host (kvm_intel.nested=1).

cat /sys/module/kvm_amd/parameters/nested
1

on host

> Also it appears to go away if I revert ce53053ce378c21 ("kvm: switch
> get_user_page_nowait() to get_user_pages_unlocked()").

That simply prevents this reproducer hitting get_user_pages_unlocked()
instead of grab mmap_sem/get_user_pages/drop mmap_sem. I.e. does not
allow __get_user_pages_locked() to drop/regain ->mmap_sem.

The bug may be in the way we call get_user_pages_unlocked() in that
commit, but it might easily be a bug in __get_user_pages_locked()
exposed by that reproducer somehow.

[Al Viro]

I think I understand what's going on. FOLL_NOWAIT handling is a serious
mess ;-/ I'll probably have something to test tomorrow - I still can't
reproduce it here, unfortunately.

[Dmitry Vyukov]

Hi Al,

syzbot tests for up to 5 minutes. However, if there is a race involved
then you may need more time because the crash is probabilistic.
But from what I see most of the time, if one can't reproduce it
easily, it's usually due to some differences in setup that just don't
allow the crash to happen at all.
FWIW syzbot re-runs each reproducer on a freshly booted dedicated VM
and what it provided is the kernel output it got during run of the
provided program. So we have reasonably high assurance that this
reproducer worked in at least one setup.

Even if you can't reproduce it locally, you can use syzbot testing
service, see "syz test" here:
https://github.com/google/syzkaller/blob/master/docs/syzbot.md#communication-with-syzbot

We also try to collect known causes of non-working reproducers, so if
you get any hints as to why it does not reproduce for you, we can add
it here:
https://github.com/google/syzkaller/blob/master/docs/syzbot.md#crash-does-not-reproduce
Since kvm/ept are present in the stacks, I suspect that it may be due
to a different host CPU unfortunately.

[Al Viro]

On Fri, Feb 02, 2018 at 09:57:27AM +0100, Dmitry Vyukov wrote:

> syzbot tests for up to 5 minutes. However, if there is a race involved
> then you may need more time because the crash is probabilistic.
> But from what I see most of the time, if one can't reproduce it
> easily, it's usually due to some differences in setup that just don't
> allow the crash to happen at all.
> FWIW syzbot re-runs each reproducer on a freshly booted dedicated VM
> and what it provided is the kernel output it got during run of the
> provided program. So we have reasonably high assurance that this
> reproducer worked in at least one setup.

Could you guys check if the following fixes the reproducer?

diff --git a/mm/gup.c b/mm/gup.c
index 61015793f952..058a9a8e4e2e 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -861,6 +861,9 @@ static __always_inline long __get_user_pages_locked(struct task_struct *tsk,
BUG_ON(*locked != 1);
}

+ if (flags & FOLL_NOWAIT)
+ locked = NULL;
+
if (pages)
flags |= FOLL_GET;

[Eric Biggers]

Hi Al,

Yes that fixes the reproducer for me.

- Eric

[Eric Biggers]

Just to follow up on this: it seems that Al's suggested fix didn't go anywhere,
but someone else eventually ran into this bug (which was a real deadlock) and a
slightly different fix was merged, commit 96312e61282ae. It fixes the
reproducer for me too. Telling syzbot so that it can close the bug:

#syz fix: mm/gup.c: teach get_user_pages_unlocked to handle FOLL_NOWAIT

- Eric