Nullfs - Panic when umounting and lower fs has been removed

[Antoine Brodin]

Hi there,

I encountered a panic on head today with nullfs using poudriere.

It is quite easy to reproduce, the script is probably doing something
wrong but panicking is not nice

%%%
mkdir /tmp/.new_packages /mnt/new_packages
mount -t nullfs /tmp/.new_packages /mnt/new_packages
dd if=/dev/zero of=/mnt/new_packages/bar count=20000
mv /tmp/.new_packages/bar /tmp/
rm -rf /tmp/.new_packages
umount -f /mnt/new_packages
%%%

The panic:

userret: returning with the following locks held:
exclusive lockmgr ufs (ufs) r = 0 (0xfffffe0010a77548) locked @
/usr/src/sys/modules/nullfs/../../fs/nullfs/null_vnops.c:615
panic: witness_warn
cpuid = 0
KDB: enter: panic

db:1:lockinfo> show locks
exclusive lockmgr ufs (ufs) r = 0 (0xfffffe0010a77548) locked @
/usr/src/sys/modules/nullfs/../../fs/nullfs/null_vnops.c:615

db:1:locks> show alllocks
Process 1032 (umount) thread 0xfffffe0010623920 (100116)
exclusive lockmgr ufs (ufs) r = 0 (0xfffffe0010a77548) locked @
/usr/src/sys/modules/nullfs/../../fs/nullfs/null_vnops.c:615

db:1:alllocks> show lockedvnods
Locked vnodes
0xfffffe0010a774e0: tag ufs, type VDIR
usecount 0, writecount 0, refcount 0 mountedhere 0
flags (VI_FREE)
v_object 0xfffffe0010af0960 ref 0 pages 0
#0 0xffffffff8088cd59 at __lockmgr_args+0x979
#1 0xffffffff80af4df2 at ffs_lock+0x92
#2 0xffffffff80ce96f9 at VOP_LOCK1_APV+0xf9
#3 0xffffffff81a5f8df at null_lock+0xdf
#4 0xffffffff80ce96f9 at VOP_LOCK1_APV+0xf9
#5 0xffffffff8095a836 at _vn_lock+0xc6
#6 0xffffffff8094b681 at vflush+0x3a1
#7 0xffffffff81a5eace at nullfs_unmount+0x2e
#8 0xffffffff809436cb at dounmount+0x39b
#9 0xffffffff8094330b at sys_unmount+0x37b
#10 0xffffffff80c41322 at amd64_syscall+0x282
#11 0xffffffff80c2981b at Xfast_syscall+0xfb

db:0:kdb.enter.panic> bt
Tracing pid 1032 tid 100116 td 0xfffffe0010623920
kdb_enter() at kdb_enter+0x3e/frame 0xffffff8122567940
vpanic() at vpanic+0x146/frame 0xffffff8122567980
kassert_panic() at kassert_panic+0x136/frame 0xffffff81225679f0
witness_warn() at witness_warn+0x4e4/frame 0xffffff8122567ab0
userret() at userret+0x89/frame 0xffffff8122567ae0
amd64_syscall() at amd64_syscall+0x397/frame 0xffffff8122567bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8122567bf0
--- syscall (0, FreeBSD ELF64, nosys), rip = 0x80087abcc, rsp =
0x7fffffffd398, rbp = 0x800c0a9d0 ---


Cheers,

Antoine
_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"

[Antoine Brodin]

Reverting r250505 fixes this issue.

[Konstantin Belousov]

On Sun, May 19, 2013 at 08:41:03PM +0000, Antoine Brodin wrote:
> Hi there,
>
> I encountered a panic on head today with nullfs using poudriere.
>
> It is quite easy to reproduce, the script is probably doing something
> wrong but panicking is not nice
>
> %%%
> mkdir /tmp/.new_packages /mnt/new_packages
> mount -t nullfs /tmp/.new_packages /mnt/new_packages
> dd if=/dev/zero of=/mnt/new_packages/bar count=20000
> mv /tmp/.new_packages/bar /tmp/
> rm -rf /tmp/.new_packages
> umount -f /mnt/new_packages
> %%%
>
> The panic:
>
> userret: returning with the following locks held:
> exclusive lockmgr ufs (ufs) r = 0 (0xfffffe0010a77548) locked @
> /usr/src/sys/modules/nullfs/../../fs/nullfs/null_vnops.c:615
> panic: witness_warn
> cpuid = 0
> KDB: enter: panic

This is strange, do you have witness enabled, but invariants not ?

Please try the following change.

diff --git a/sys/fs/nullfs/null_vfsops.c b/sys/fs/nullfs/null_vfsops.c
index ad02236..bece8c8 100644
--- a/sys/fs/nullfs/null_vfsops.c
+++ b/sys/fs/nullfs/null_vfsops.c
@@ -417,8 +417,13 @@ nullfs_unlink_lowervp(struct mount *mp, struct vnode *lowervp)
*/
if (vp->v_usecount == 0) {
KASSERT((vp->v_iflag & VI_DOOMED) != 0,
- ("not reclaimed %p", vp));
+ ("not reclaimed nullfs vnode %p", vp));
VOP_UNLOCK(vp, 0);
+ } else {
+ ASSERT_VOP_ELOCKED(vp, "unlink_lowervp");
+ KASSERT((vp->v_iflag & VI_DOOMED) == 0,
+ ("reclaimed nullfs vnode %p", vp));
+ xp->null_flags &= ~NULLV_NOUNLOCK;
}
vdrop(vp);
}

[Peter Holm]

I can confirm the panic
http://people.freebsd.org/~pho/stress/log/kostik567.txt
and that the patch fixes that problem.

- Peter

[Antoine Brodin]

On Mon, May 20, 2013 at 7:34 AM, Konstantin Belousov
<kost...@gmail.com> wrote:
> This is strange, do you have witness enabled, but invariants not ?
>
> Please try the following change.
>
> diff --git a/sys/fs/nullfs/null_vfsops.c b/sys/fs/nullfs/null_vfsops.c
> index ad02236..bece8c8 100644
> --- a/sys/fs/nullfs/null_vfsops.c

> +++ b/sys/fs/nullfs/null_vfsops.c
> @@ -417,8 +417,13 @@ nullfs_unlink_lowervp(struct mount *mp, struct vnode *lowervp)
> */
> if (vp->v_usecount == 0) {
> KASSERT((vp->v_iflag & VI_DOOMED) != 0,
> - ("not reclaimed %p", vp));
> + ("not reclaimed nullfs vnode %p", vp));
> VOP_UNLOCK(vp, 0);
> + } else {
> + ASSERT_VOP_ELOCKED(vp, "unlink_lowervp");
> + KASSERT((vp->v_iflag & VI_DOOMED) == 0,
> + ("reclaimed nullfs vnode %p", vp));
> + xp->null_flags &= ~NULLV_NOUNLOCK;
> }
> vdrop(vp);
> }

I have both witness and invariants.
And this patch fixes the panic.

Thanks!

Antoine