Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

[ANN] unionfs patchset-17 release, lock mechanism changed for robust

7 views
Skip to first unread message

Daichi GOTO

unread,
Dec 1, 2006, 8:38:34 AM12/1/06
to
Hi Guys!

It is my pleasure and honor to announce the availability of
the unionfs patchset-17. p17 have some significant improvements
around the lock mechanism for robust and stable working.

Patchset-17:
For 7-current
http://people.freebsd.org/~daichi/unionfs/unionfs-p17.diff

For 6.x
sorry, it is for current only.

Changes in unionfs-p17.diff
- Fs takes illegal access without lock of lower layer
vnode if the both upper/lower layers have both vnode.
To fix this problem, we change the lock mechanism to
get locks for both upper/lower layer always.
- Kernel gets a dead-lock easily within above
upper/lower-layer-always-lock-mechanism. To avoide
above dead-lock, we changed vfs_lookup.c. By that change,
it always locks vnodes parent first and children second.
You could see the same lock-order-control implementation
around cache_lookup.
- It takes the both open/close operations per kernel thread.
- It takes readdir-treat-status-management per kernel thread.
- It reopens vnode if needed when coping to upper layer on
advlock.
- mount_unionfs(8) changes option style fitting for fstab(5)
style. (by rodrigc)
- manual of mount_unionfs(8) was changed. (by rodrigc)

The documents of those unionfs patches:

http://people.freebsd.org/~daichi/unionfs/ (English)
http://people.freebsd.org/~daichi/unionfs/index-ja.html (Japanese)


After release of p16, some folks gave us some panic reports that
indicate our implementations has a critical problem around the lock
mechanism.

After our long researches and discussions, we have tried to
re-implement our unionfs lock mechanism. And it is done :)

For unionfs lovers (including FreeSBIE developers, ports cluster
managers, heavy memory-fs users, or folks use unionfs), could you
try p17 please? If p17 solves that panics, we guess it is unionfs
merge time for current branch.

Thanks


P.S.

Current English document of web has some Japanese contents. We
need a translator ;-)

--
Daichi GOTO, http://people.freebsd.org/~daichi
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

Kris Kennaway

unread,
Dec 2, 2006, 5:24:23 PM12/2/06
to
On Fri, Dec 01, 2006 at 10:38:34PM +0900, Daichi GOTO wrote:
> Hi Guys!
>
> It is my pleasure and honor to announce the availability of
> the unionfs patchset-17. p17 have some significant improvements
> around the lock mechanism for robust and stable working.

I got the following locking assertion as soon as I tried to start a
package build in the unionfs -b mountpoint.

KDB: stack backtrace:
db_trace_self_wrapper(c4fd6700,eca4fb00,c073871a,eca4fb10,ca025c18,...) at db_trace_self_wrapper+0x37
vfs_badlock(ca025c18,eca4fb10,c07d4d20,ca025c18,c4fd6700,eca4fb60) at vfs_badlock+0x76
assert_vop_elocked(ca025c18,c078ee83,eca4fbf0,eca4fbf0,1,...) at assert_vop_elocked+0x63
VOP_MKDIR_APV(c07a4e00,eca4fb60,c0771a8c,d6e,7070000,...) at VOP_MKDIR_APV+0xea
kern_mkdir(c4fd6700,bfbfeb15,0,1ff,eca4fd30,...) at kern_mkdir+0x327
mkdir(c4fd6700,eca4fd04,8,eca4fd38,2,...) at mkdir+0x29
syscall(3b,3b,3b,bfbfeb15,1,...) at syscall+0x152
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (136, FreeBSD ELF32, mkdir), eip = 0x281479e7, esp = 0xbfbfe89c, ebp = 0xbfbfe978 ---
VOP_MKDIR: 0xca025c18 is not exclusive locked but should be

db> show lockedvnods
Locked vnodes

0xc90b4c18: tag ufs, type VDIR
usecount 1, writecount 0, refcount 2 mountedhere 0
flags ()
v_object 0xcbe4b690 ref 0 pages 0
lock type ufs: EXCL (count 1) by thread 0xc4fd6700 (pid 98678)#0 0xc0545c13 at _lockmgr+0x538
#1 0xc06a766f at ffs_lock+0x6a
#2 0xc0738675 at _VOP_LOCK_APV+0x69
#3 0xc051147f at unionfs_lock+0x186
#4 0xc0738675 at _VOP_LOCK_APV+0x69
#5 0xc05d7f4a at _vn_lock+0x73
#6 0xc05c06ee at lookup+0xe9
#7 0xc05c1703 at namei+0x358
#8 0xc05d47c0 at kern_mkdir+0x72
#9 0xc05d4b82 at mkdir+0x29
#10 0xc072156b at syscall+0x152
#11 0xc07093af at Xint0x80_syscall+0x1f

ino 2120020, on dev da0s1e

0xcaa1d560: tag ufs, type VDIR
usecount 2, writecount 0, refcount 4 mountedhere 0
flags ()
v_object 0xcbe0e708 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc4dc2700 (pid 98188)#0 0xc0545c13 at _lockmgr+0x538
#1 0xc06a766f at ffs_lock+0x6a
#2 0xc0738675 at _VOP_LOCK_APV+0x69
#3 0xc05d7f4a at _vn_lock+0x73
#4 0xc05c06ee at lookup+0xe9
#5 0xc05c1703 at namei+0x358
#6 0xc05d17ed at kern_unlink+0x60
#7 0xc05d19f9 at unlink+0x22
#8 0xc072156b at syscall+0x152
#9 0xc07093af at Xint0x80_syscall+0x1f

ino 2192674, on dev da0s1e

0xcbe47408: tag ufs, type VDIR
usecount 2, writecount 0, refcount 4 mountedhere 0
flags ()
v_object 0xcbe4b870 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc5b35e00 (pid 98106)#0 0xc0545c13 at _lockmgr+0x538
#1 0xc06a766f at ffs_lock+0x6a
#2 0xc0738675 at _VOP_LOCK_APV+0x69
#3 0xc05d7f4a at _vn_lock+0x73
#4 0xc05d54e0 at getdirentries+0xf7
#5 0xc072156b at syscall+0x152
#6 0xc07093af at Xint0x80_syscall+0x1f

ino 2025752, on dev da0s1e

0xcaa30968: tag ufs, type VDIR
usecount 1, writecount 0, refcount 2 mountedhere 0
flags ()
v_object 0xcbe4b708 ref 0 pages 0
lock type ufs: EXCL (count 1) by thread 0xc4fd6700 (pid 98678)#0 0xc0545c13 at _lockmgr+0x538
#1 0xc06a766f at ffs_lock+0x6a
#2 0xc0738675 at _VOP_LOCK_APV+0x69
#3 0xc0511569 at unionfs_lock+0x270
#4 0xc0738675 at _VOP_LOCK_APV+0x69
#5 0xc05d7f4a at _vn_lock+0x73
#6 0xc05c06ee at lookup+0xe9
#7 0xc05c1703 at namei+0x358
#8 0xc05d47c0 at kern_mkdir+0x72
#9 0xc05d4b82 at mkdir+0x29
#10 0xc072156b at syscall+0x152
#11 0xc07093af at Xint0x80_syscall+0x1f

ino 2169764, on dev da0s1e

0xca00c560: tag ufs, type VDIR
usecount 1, writecount 0, refcount 2 mountedhere 0
flags ()
lock type ufs: SHARED (count 1)#0 0xc0545c13 at _lockmgr+0x538
#1 0xc06a766f at ffs_lock+0x6a
#2 0xc0738675 at _VOP_LOCK_APV+0x69
#3 0xc0511569 at unionfs_lock+0x270
#4 0xc0738675 at _VOP_LOCK_APV+0x69
#5 0xc05d7f4a at _vn_lock+0x73
#6 0xc050ba83 at unionfs_nodeget+0x649
#7 0xc05109ca at unionfs_mkdir+0x116
#8 0xc073b771 at VOP_MKDIR_APV+0x94
#9 0xc05d4a75 at kern_mkdir+0x327
#10 0xc05d4b82 at mkdir+0x29
#11 0xc072156b at syscall+0x152
#12 0xc07093af at Xint0x80_syscall+0x1f

ino 2169777, on dev da0s1e

0xc9fe5810: tag ufs, type VREG
usecount 1, writecount 0, refcount 1 mountedhere 0
flags ()
lock type ufs: EXCL (count 1) by thread 0xc4dc2700 (pid 98188)#0 0xc0545c13 at _lockmgr+0x538
#1 0xc06a766f at ffs_lock+0x6a
#2 0xc0738675 at _VOP_LOCK_APV+0x69
#3 0xc05d7f4a at _vn_lock+0x73
#4 0xc05cb994 at vget+0x8b
#5 0xc05bfd42 at vfs_hash_get+0x105
#6 0xc06a3003 at ffs_vget+0x49
#7 0xc06add11 at ufs_lookup+0x967
#8 0xc0739a97 at VOP_CACHEDLOOKUP_APV+0x94
#9 0xc05bc44d at vfs_cache_lookup+0xec
#10 0xc0739bd1 at VOP_LOOKUP_APV+0x9c
#11 0xc05c0971 at lookup+0x36c
#12 0xc05c1703 at namei+0x358
#13 0xc05d17ed at kern_unlink+0x60
#14 0xc05d19f9 at unlink+0x22
#15 0xc072156b at syscall+0x152
#16 0xc07093af at Xint0x80_syscall+0x1f

ino 2194373, on dev da0s1e

0xcbe3e560: tag unionfs, type VDIR
usecount 7, writecount 0, refcount 7 mountedhere 0
flags ()
v_object 0xcbe4b708 ref 0 pages 0
lock type ufs: EXCL (count 1) by thread 0xc4fd6700 (pid 98678)#0 0xc0545c13 at _lockmgr+0x538
#1 0xc06a766f at ffs_lock+0x6a
#2 0xc0738675 at _VOP_LOCK_APV+0x69
#3 0xc0511569 at unionfs_lock+0x270
#4 0xc0738675 at _VOP_LOCK_APV+0x69
#5 0xc05d7f4a at _vn_lock+0x73
#6 0xc05c06ee at lookup+0xe9
#7 0xc05c1703 at namei+0x358
#8 0xc05d47c0 at kern_mkdir+0x72
#9 0xc05d4b82 at mkdir+0x29
#10 0xc072156b at syscall+0x152
#11 0xc07093af at Xint0x80_syscall+0x1f

unionfs_vp=0xcbe3e560, uppervp=0xcaa30968, lowervp=0xc90b4c18
unionfs: upper
0xcaa30968: tag ufs, type VDIR
usecount 1, writecount 0, refcount 2 mountedhere 0
flags ()
v_object 0xcbe4b708 ref 0 pages 0
lock type ufs: EXCL (count 1) by thread 0xc4fd6700 (pid 98678)#0 0xc0545c13 at _lockmgr+0x538
#1 0xc06a766f at ffs_lock+0x6a
#2 0xc0738675 at _VOP_LOCK_APV+0x69
#3 0xc0511569 at unionfs_lock+0x270
#4 0xc0738675 at _VOP_LOCK_APV+0x69
#5 0xc05d7f4a at _vn_lock+0x73
#6 0xc05c06ee at lookup+0xe9
#7 0xc05c1703 at namei+0x358
#8 0xc05d47c0 at kern_mkdir+0x72
#9 0xc05d4b82 at mkdir+0x29
#10 0xc072156b at syscall+0x152
#11 0xc07093af at Xint0x80_syscall+0x1f

ino 2169764, on dev da0s1e
unionfs: lower
0xc90b4c18: tag ufs, type VDIR
usecount 1, writecount 0, refcount 2 mountedhere 0
flags ()
v_object 0xcbe4b690 ref 0 pages 0
lock type ufs: EXCL (count 1) by thread 0xc4fd6700 (pid 98678)#0 0xc0545c13 at _lockmgr+0x538
#1 0xc06a766f at ffs_lock+0x6a
#2 0xc0738675 at _VOP_LOCK_APV+0x69
#3 0xc051147f at unionfs_lock+0x186
#4 0xc0738675 at _VOP_LOCK_APV+0x69
#5 0xc05d7f4a at _vn_lock+0x73
#6 0xc05c06ee at lookup+0xe9
#7 0xc05c1703 at namei+0x358
#8 0xc05d47c0 at kern_mkdir+0x72
#9 0xc05d4b82 at mkdir+0x29
#10 0xc072156b at syscall+0x152
#11 0xc07093af at Xint0x80_syscall+0x1f

ino 2120020, on dev da0s1e

0xca025c18: tag unionfs, type VDIR
usecount 1, writecount 0, refcount 1 mountedhere 0
flags ()
lock type ufs: SHARED (count 1)#0 0xc0545c13 at _lockmgr+0x538
#1 0xc06a766f at ffs_lock+0x6a
#2 0xc0738675 at _VOP_LOCK_APV+0x69
#3 0xc0511569 at unionfs_lock+0x270
#4 0xc0738675 at _VOP_LOCK_APV+0x69
#5 0xc05d7f4a at _vn_lock+0x73
#6 0xc050ba83 at unionfs_nodeget+0x649
#7 0xc05109ca at unionfs_mkdir+0x116
#8 0xc073b771 at VOP_MKDIR_APV+0x94
#9 0xc05d4a75 at kern_mkdir+0x327
#10 0xc05d4b82 at mkdir+0x29
#11 0xc072156b at syscall+0x152
#12 0xc07093af at Xint0x80_syscall+0x1f

unionfs_vp=0xca025c18, uppervp=0xca00c560, lowervp=0
unionfs: upper
0xca00c560: tag ufs, type VDIR
usecount 1, writecount 0, refcount 2 mountedhere 0
flags ()
lock type ufs: SHARED (count 1)#0 0xc0545c13 at _lockmgr+0x538
#1 0xc06a766f at ffs_lock+0x6a
#2 0xc0738675 at _VOP_LOCK_APV+0x69
#3 0xc0511569 at unionfs_lock+0x270
#4 0xc0738675 at _VOP_LOCK_APV+0x69
#5 0xc05d7f4a at _vn_lock+0x73
#6 0xc050ba83 at unionfs_nodeget+0x649
#7 0xc05109ca at unionfs_mkdir+0x116
#8 0xc073b771 at VOP_MKDIR_APV+0x94
#9 0xc05d4a75 at kern_mkdir+0x327
#10 0xc05d4b82 at mkdir+0x29
#11 0xc072156b at syscall+0x152
#12 0xc07093af at Xint0x80_syscall+0x1f

ino 2169777, on dev da0s1e
db>

Core available.

Kris

Kris Kennaway

unread,
Dec 2, 2006, 5:36:21 PM12/2/06
to
On Sat, Dec 02, 2006 at 05:24:23PM -0500, Kris Kennaway wrote:
> On Fri, Dec 01, 2006 at 10:38:34PM +0900, Daichi GOTO wrote:
> > Hi Guys!
> >
> > It is my pleasure and honor to announce the availability of
> > the unionfs patchset-17. p17 have some significant improvements
> > around the lock mechanism for robust and stable working.
>
> I got the following locking assertion as soon as I tried to start a
> package build in the unionfs -b mountpoint.

Even simpler test:

mount_unionfs /usr/src /c/test/src
cd /c/test/src
make buildworld -j4

panics immediately with:

db_trace_self_wrapper(c5685a80,ecd01ab4,c073871a,ecd01ac4,c57af560,...) at db_trace_self_wrapper+0x37
vfs_badlock(c57af560,ecd01ac4,c07d4d20,c57af560,c5685a80,c07ea370) at vfs_badlock+0x76
assert_vop_elocked(c57af560,c0760340,ecd01b78,c4fb8a00,ecd01b78,...) at assert_vop_elocked+0x63
unionfs_get_node_status(c4fb8a00,c5685a80,ecd01b0c,c054a2cc,c5683d88,...) at unionfs_get_node_status+0x29
unionfs_ioctl(ecd01b78,c078ed87,ecd01c0c,c57af560,0,...) at unionfs_ioctl+0x2e
VOP_IOCTL_APV(c07a4e00,ecd01b78,c0771b2f,2e7,0,...) at VOP_IOCTL_APV+0x94
vn_ioctl(c53aaa20,402c7413,c5786700,c55a9000,c5685a80,...) at vn_ioctl+0x135
kern_ioctl(c5685a80,3,402c7413,c5786700,c5786700,...) at kern_ioctl+0x8a
ioctl(c5685a80,ecd01d04,c,ecd01d38,3,...) at ioctl+0xac
syscall(3b,3b,3b,0,8220160,...) at syscall+0x152
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x2817fb6f, esp = 0xbfbfdbdc, ebp = 0xbfbfdbf8 ---
unionfs_get_node_status: 0xc57af560 is not exclusive locked but should be

db> show lockedvnods
Locked vnodes

0xc57a0000: tag ufs, type VREG
usecount 0, writecount 0, refcount 3 mountedhere 0
flags ()
v_object 0xc575d3c0 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc5399540 (pid 1512)#0 0xc0545c13 at _lockmgr+0x538


#1 0xc06a766f at ffs_lock+0x6a
#2 0xc0738675 at _VOP_LOCK_APV+0x69
#3 0xc05d7f4a at _vn_lock+0x73
#4 0xc05cb994 at vget+0x8b
#5 0xc05bfd42 at vfs_hash_get+0x105
#6 0xc06a3003 at ffs_vget+0x49
#7 0xc06add11 at ufs_lookup+0x967
#8 0xc0739a97 at VOP_CACHEDLOOKUP_APV+0x94
#9 0xc05bc44d at vfs_cache_lookup+0xec
#10 0xc0739bd1 at VOP_LOOKUP_APV+0x9c
#11 0xc05c0971 at lookup+0x36c
#12 0xc05c1703 at namei+0x358
#13 0xc05d17ed at kern_unlink+0x60
#14 0xc05d19f9 at unlink+0x22
#15 0xc072156b at syscall+0x152
#16 0xc07093af at Xint0x80_syscall+0x1f

ino 10729, on dev da0s1a
db>

Please don't take this the wrong way, but I wonder if your internal
tests can be improved so we don't have to go through this kind of
extended debugging cycle.

Kris

Scott Long

unread,
Dec 2, 2006, 8:11:11 PM12/2/06
to
Kris Kennaway wrote:
> On Fri, Dec 01, 2006 at 10:38:34PM +0900, Daichi GOTO wrote:
>> Hi Guys!
>>
>> It is my pleasure and honor to announce the availability of
>> the unionfs patchset-17. p17 have some significant improvements
>> around the lock mechanism for robust and stable working.
>
> I got the following locking assertion as soon as I tried to start a
> package build in the unionfs -b mountpoint.
>

Was this with the full p17 patchset applied, or just the partial set
that was committed to CVS (which is about to be yanked if it doesn't
get fixed)?

Scott

Kris Kennaway

unread,
Dec 2, 2006, 8:25:38 PM12/2/06
to
On Sat, Dec 02, 2006 at 06:11:11PM -0700, Scott Long wrote:
> Kris Kennaway wrote:
> >On Fri, Dec 01, 2006 at 10:38:34PM +0900, Daichi GOTO wrote:
> >>Hi Guys!
> >>
> >>It is my pleasure and honor to announce the availability of
> >>the unionfs patchset-17. p17 have some significant improvements
> >>around the lock mechanism for robust and stable working.
> >
> >I got the following locking assertion as soon as I tried to start a
> >package build in the unionfs -b mountpoint.
> >
>
> Was this with the full p17 patchset applied, or just the partial set
> that was committed to CVS (which is about to be yanked if it doesn't
> get fixed)?

The full set.

Kris

Daichi GOTO

unread,
Dec 3, 2006, 8:12:18 AM12/3/06
to
Hi guys, it's congratulations!

It is a red-letter day for new FreeBSD unionfs. The unionfs-17.diff
(without patch for sys/kern/vfs_lookup.c) was committed to FreeBSD
7-current of 2006-12-02 19:35:56 UTC by rodrigc, my src mentor.


> Current English document of web has some Japanese contents. We
> need a translator ;-)

Hiroharu TAMARU-san gave us good English text. Thanks.
And we added some text around committed of current.

http://people.freebsd.org/~daichi/unionfs/ (English)
http://people.freebsd.org/~daichi/unionfs/index-ja.html (Japanese)


Hopefully we can get more feedback from testers :)

Daichi GOTO

unread,
Dec 3, 2006, 8:21:59 AM12/3/06
to
Kris Kennaway wrote:
> I got the following locking assertion as soon as I tried to start a
> package build in the unionfs -b mountpoint.

oops X(

> Core available.
>
> Kris

Alright. We are trying to get the same result. Please wait :)

0 new messages