BeeGFS re export NFS performance issue

[Chen Bill]

Hi ,

We export the BeeGFS as NFS share to normal NFS client through 10Gb network, but the NFS performance is very low around 20-30MB/s, 

And normal NFS  share (not export BeeGFS) can reach 500-600MB/s.

Is anybody know how to improve NFS performance over BeeGFS?

Best Regards & many thanks,

Bill

[Sven Breuner]

Hi Bill,

Chen Bill wrote on 01/17/2015 04:28 AM:
> We export the BeeGFS as NFS share to normal NFS client through 10Gb
> network, but the NFS performance is very low around 20-30MB/s,
> And normal NFS share (not export BeeGFS) can reach 500-600MB/s.
>
> Is anybody know how to improve NFS performance over BeeGFS?

which nfs server are you using? Did you try unfs3 already?

Best regards,
Sven

[Chen Bill]

Thank you Sven,

I didn't use unfs3, I just use normal NFS Server on CentOS6.4. I'll try the unfs3 later.

Regards,

Bill

[karim.l...@eurocfd.com]

Hi Chen,

We have the same problem. Did you solve the matter ?

Thanks for your answer.

[Jzx]

I'm experiencing the same issue,a client machine without other roles installed with ver 7.1.3 client, kernel nfs3/4 export get a very low speed around 50MB/s,either mount from local server or via FDR IB gets the same result, however local beegfs mount can achieve 2GB/s speed, that's really weird.

Here are  configurations and logs

client:

tuneFileCacheType             = buffered (native/none is tried)

tunePreferredMetaFile         =

tunePreferredStorageFile      =

tuneRemoteFSync               = true (false is tried)

tuneUseGlobalAppendLocks      = false

tuneUseGlobalFileLocks        = false

tuneRefreshOnGetAttr          = true

export:

/mnt/beegfs 127.0.0.0/20(rw,no_subtree_check,crossmnt,insecure,root_squash,fsid=0,anonuid=1001,anongid=100) 

mount:

mount 127.0.0.1:/ /mnt/beenfs -o noatime,nodiratime,async,vers=4.2 (mount 127.0.0.1:/mnt/beegfs won't work in nfs4)
part of nfs debug msg:

[1384614.756340] NFS: permission(0:42/11686834649927028163), mask=0x81, res=0

[1384614.756346] NFS: nfs_lookup_revalidate(ceshi/test) is valid

[1384614.756360] NFS: permission(0:42/11686834649927028163), mask=0x81, res=0

[1384614.756361] NFS: nfs_lookup_revalidate(ceshi/test.0.0) is valid

[1384614.756364] NFS: revalidating (0:42/13936344780148527098)

[1384614.756371] --> nfs41_call_sync_prepare data->seq_server ffff8ef3c05dc800

[1384614.756372] --> nfs41_setup_sequence

[1384614.756374] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31

[1384614.756375] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0

[1384614.756376] <-- nfs41_setup_sequence slotid=0 seqid=12949

[1384614.756385] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=12949 slotid=0 max_slotid=0 cache_this=0

[1384614.756604] decode_attr_type: type=0100000

[1384614.756605] decode_attr_change: change attribute=6770596920839634944

[1384614.756606] decode_attr_size: file size=1073741824

[1384614.756606] decode_attr_fsid: fsid=(0x0/0x0)

[1384614.756607] decode_attr_fileid: fileid=13936344780148527098

[1384614.756608] decode_attr_fs_locations: fs_locations done, error = 0

[1384614.756609] decode_attr_mode: file mode=0777

[1384614.756609] decode_attr_nlink: nlink=1

[1384614.756611] decode_attr_owner: uid=1001

[1384614.756612] decode_attr_group: gid=100

[1384614.756612] decode_attr_rdev: rdev=(0x0:0x0)

[1384614.756613] decode_attr_space_used: space used=1073741824

[1384614.756614] decode_attr_time_access: atime=1576402547

[1384614.756614] decode_attr_time_metadata: ctime=1576402439

[1384614.756615] decode_attr_time_modify: mtime=1576402439

[1384614.756615] decode_attr_mounted_on_fileid: fileid=13936344780148527098

[1384614.756616] decode_getfattr_attrs: xdr returned 0

[1384614.756617] decode_getfattr_generic: xdr returned 0

[1384614.756619] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31

[1384614.756620] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1

[1384614.756621] nfs4_free_slot: slotid 1 highest_used_slotid 0

[1384614.756621] nfs41_sequence_process: Error 0 free the slot

[1384614.756622] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295

[1384614.756629] NFS: nfs_update_inode(0:42/13936344780148527098 fh_crc=0x079bd548 ct=1 info=0x427e7f)

[1384614.756630] NFS: (0:42/13936344780148527098) revalidation complete

[1384614.756632] NFS: dentry_delete(ceshi/test.0.0, 4808cc)

[1384614.971621] NFS: permission(0:42/11686834649927028163), mask=0x81, res=0

[1384614.971624] NFS: nfs_lookup_revalidate(ceshi/test.0.0) is valid

[1384614.971625] NFS: dentry_delete(ceshi/test.0.0, 4808cc)

[1384614.971629] NFS: permission(0:42/11686834649927028163), mask=0x81, res=0

[1384614.971630] NFS: nfs_lookup_revalidate(ceshi/test.0.0) is valid

[1384614.971630] NFS: dentry_delete(ceshi/test.0.0, 4808cc)

[1384615.114864] NFS: permission(0:42/11686834649927028163), mask=0x81, res=0

[1384615.114870] NFS: revalidating (0:42/13936344780148527098)

[1384615.114877] --> nfs41_call_sync_prepare data->seq_server ffff8ef3c05dc800

[1384615.114878] --> nfs41_setup_sequence

[1384615.114879] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31

[1384615.114881] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0

[1384615.114881] <-- nfs41_setup_sequence slotid=0 seqid=12950

[1384615.114891] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=12950 slotid=0 max_slotid=0 cache_this=0

[1384615.115283] decode_attr_type: type=0100000

[1384615.115284] decode_attr_change: change attribute=6770596920839634944

[1384615.115285] decode_attr_size: file size=1073741824

[1384615.115286] decode_attr_fsid: fsid=(0x0/0x0)

[1384615.115288] decode_attr_fileid: fileid=13936344780148527098

[1384615.115289] decode_attr_fs_locations: fs_locations done, error = 0

[1384615.115290] decode_attr_mode: file mode=0777

[1384615.115291] decode_attr_nlink: nlink=1

[1384615.115294] decode_attr_owner: uid=1001

[1384615.115296] decode_attr_group: gid=100

[1384615.115297] decode_attr_rdev: rdev=(0x0:0x0)

[1384615.115298] decode_attr_space_used: space used=1073741824

[1384615.115300] decode_attr_time_access: atime=1576402547

[1384615.115301] decode_attr_time_metadata: ctime=1576402439

[1384615.115302] decode_attr_time_modify: mtime=1576402439

[1384615.115303] decode_attr_mounted_on_fileid: fileid=13936344780148527098

[1384615.115304] decode_getfattr_attrs: xdr returned 0

[1384615.115305] decode_getfattr_generic: xdr returned 0

[1384615.115308] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31

[1384615.115310] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1

[1384615.115311] nfs4_free_slot: slotid 1 highest_used_slotid 0

[1384615.115313] nfs41_sequence_process: Error 0 free the slot

[1384615.115314] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295

[1384615.115324] NFS: nfs_update_inode(0:42/13936344780148527098 fh_crc=0x079bd548 ct=1 info=0x427e7f)

[1384615.115326] NFS: (0:42/13936344780148527098) revalidation complete

[1384615.115328] NFS: nfs_lookup_revalidate(ceshi/test.0.0) is valid

[1384615.115333] NFS: open file(ceshi/test.0.0)

[1384615.115338] --> nfs_put_client({2})

[1384615.115352] --> nfs4_setup_sequence clp ffff8ef87a8c9c00 session ffff8ef3a3324000 sr_slot 4294967295

[1384615.115353] --> nfs41_setup_sequence

[1384615.115354] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31

[1384615.115355] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0

[1384615.115356] <-- nfs41_setup_sequence slotid=0 seqid=12951

[1384615.115357] <-- nfs4_setup_sequence status=0

[1384615.115364] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=12951 slotid=0 max_slotid=0 cache_this=1

[1384615.115954] decode_attr_type: type=0100000

[1384615.115955] decode_attr_change: change attribute=6770596920839634944

[1384615.115956] decode_attr_size: file size=1073741824

[1384615.115957] decode_attr_fsid: fsid=(0x0/0x0)

[1384615.115957] decode_attr_fileid: fileid=13936344780148527098

[1384615.115958] decode_attr_fs_locations: fs_locations done, error = 0

[1384615.115959] decode_attr_mode: file mode=0777

[1384615.115959] decode_attr_nlink: nlink=1

[1384615.115960] decode_attr_owner: uid=-2

[1384615.115961] decode_attr_group: gid=-2

[1384615.115962] decode_attr_rdev: rdev=(0x0:0x0)

[1384615.115962] decode_attr_space_used: space used=1073741824

[1384615.115963] decode_attr_time_access: atime=1576402547

[1384615.115963] decode_attr_time_metadata: ctime=1576402439

[1384615.115964] decode_attr_time_modify: mtime=1576402439

[1384615.115964] decode_attr_mounted_on_fileid: fileid=13936344780148527098

[1384615.115964] decode_getfattr_attrs: xdr returned 0

[1384615.115965] decode_getfattr_generic: xdr returned 0

[1384615.115966] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31

[1384615.115967] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1

[1384615.115968] nfs4_free_slot: slotid 1 highest_used_slotid 0

[1384615.115969] nfs41_sequence_process: Error 0 free the slot

[1384615.115977] NFS: nfs_update_inode(0:42/13936344780148527098 fh_crc=0x079bd548 ct=3 info=0x427e7f)

[1384615.115978] NFS: nfs_fhget(0:42/13936344780148527098 fh_crc=0x079bd548 ct=3)

[1384615.115980] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295

[1384615.115984] NFS: nfsi 0xffff8ef801b98000 disabling cache

[1384615.116298] NFS: direct write(ceshi/test.0.0, 4194304@0)

[1384615.116431] NFS: initiated pgio call (req 0:42/13936344780148527098, 1048576 bytes @ offset 0)

[1384615.116438] --> nfs4_setup_sequence clp ffff8ef87a8c9c00 session ffff8ef3a3324000 sr_slot 4294967295

[1384615.116438] --> nfs41_setup_sequence

[1384615.116439] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31

[1384615.116441] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0

[1384615.116441] <-- nfs41_setup_sequence slotid=0 seqid=12952

[1384615.116442] <-- nfs4_setup_sequence status=0

[1384615.116447] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=12952 slotid=0 max_slotid=0 cache_this=1

[1384615.116667] NFS: initiated pgio call (req 0:42/13936344780148527098, 1048576 bytes @ offset 1048576)

[1384615.116678] --> nfs4_setup_sequence clp ffff8ef87a8c9c00 session ffff8ef3a3324000 sr_slot 4294967295

[1384615.116678] --> nfs41_setup_sequence

[1384615.116679] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31

[1384615.116680] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1

[1384615.116681] <-- nfs41_setup_sequence slotid=1 seqid=418

[1384615.116682] <-- nfs4_setup_sequence status=0

[1384615.116783] NFS: initiated pgio call (req 0:42/13936344780148527098, 1048576 bytes @ offset 2097152)

[1384615.116787] --> nfs4_setup_sequence clp ffff8ef87a8c9c00 session ffff8ef3a3324000 sr_slot 4294967295

[1384615.116788] --> nfs41_setup_sequence

[1384615.116789] --> nfs4_alloc_slot used_slots=0003 highest_used=1 max_slots=31

[1384615.116790] <-- nfs4_alloc_slot used_slots=0007 highest_used=2 slotid=2

[1384615.116791] <-- nfs41_setup_sequence slotid=2 seqid=398

[1384615.116792] <-- nfs4_setup_sequence status=0

[1384615.116808] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=418 slotid=1 max_slotid=2 cache_this=1

[1384615.116886] NFS: initiated pgio call (req 0:42/13936344780148527098, 1048576 bytes @ offset 3145728)

[1384615.116890] --> nfs4_setup_sequence clp ffff8ef87a8c9c00 session ffff8ef3a3324000 sr_slot 4294967295

[1384615.116891] --> nfs41_setup_sequence

[1384615.116892] --> nfs4_alloc_slot used_slots=0007 highest_used=2 max_slots=31

[1384615.116893] <-- nfs4_alloc_slot used_slots=000f highest_used=3 slotid=3

[1384615.116894] <-- nfs41_setup_sequence slotid=3 seqid=383

[1384615.116895] <-- nfs4_setup_sequence status=0

[1384615.117516] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=398 slotid=2 max_slotid=3 cache_this=1

[1384615.117775] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=383 slotid=3 max_slotid=3 cache_this=1

[1384615.117943] NFS: nfs_pgio_result: 63161, (status 1048576)

[1384615.117945] --> nfs4_alloc_slot used_slots=000f highest_used=3 max_slots=31

[1384615.117947] <-- nfs4_alloc_slot used_slots=001f highest_used=4 slotid=4

[1384615.117947] nfs4_free_slot: slotid 4 highest_used_slotid 3

[1384615.117948] nfs41_sequence_process: Error 0 free the slot

[1384615.117949] nfs4_free_slot: slotid 0 highest_used_slotid 3

[1384615.121615] NFS: nfs_pgio_result: 63163, (status 1048576)

[1384615.121617] --> nfs4_alloc_slot used_slots=000e highest_used=3 max_slots=31

[1384615.121617] <-- nfs4_alloc_slot used_slots=000f highest_used=3 slotid=0

[1384615.121618] nfs4_free_slot: slotid 0 highest_used_slotid 3

[1384615.121619] nfs41_sequence_process: Error 0 free the slot

[1384615.121619] nfs4_free_slot: slotid 2 highest_used_slotid 3

[1384615.121945] NFS: nfs_pgio_result: 63162, (status 1048576)

[1384615.121947] --> nfs4_alloc_slot used_slots=000a highest_used=3 max_slots=31

[1384615.121948] <-- nfs4_alloc_slot used_slots=000b highest_used=3 slotid=0

[1384615.121948] nfs4_free_slot: slotid 0 highest_used_slotid 3

[1384615.121949] nfs41_sequence_process: Error 0 free the slot

[1384615.121949] nfs4_free_slot: slotid 1 highest_used_slotid 3

[1384615.122763] NFS: nfs_pgio_result: 63164, (status 1048576)

[1384615.122765] --> nfs4_alloc_slot used_slots=0008 highest_used=3 max_slots=31

[1384615.122765] <-- nfs4_alloc_slot used_slots=0009 highest_used=3 slotid=0

[1384615.122766] nfs4_free_slot: slotid 0 highest_used_slotid 3

[1384615.122766] nfs41_sequence_process: Error 0 free the slot

[1384615.122767] nfs4_free_slot: slotid 3 highest_used_slotid 4294967295

[1384615.122836] NFS: initiated commit call

[1384615.122838] --> nfs4_setup_sequence clp ffff8ef87a8c9c00 session ffff8ef3a3324000 sr_slot 4294967295

[1384615.122838] --> nfs41_setup_sequence

[1384615.122839] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31

[1384615.122839] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0

[1384615.122840] <-- nfs41_setup_sequence slotid=0 seqid=12953

[1384615.122840] <-- nfs4_setup_sequence status=0

[1384615.122843] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=12953 slotid=0 max_slotid=0 cache_this=1

[1384615.123080] NFS: 63165 nfs_commit_done (status 0)

[1384615.123081] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31

[1384615.123082] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1

[1384615.123083] nfs4_free_slot: slotid 1 highest_used_slotid 0

[1384615.123083] nfs41_sequence_process: Error 0 free the slot

[1384615.123084] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295

[1384615.123086] NFS: 63165 commit returned 0

[1384615.123369] NFS: direct write(ceshi/test.0.0, 4194304@4194304)

[1384615.123435] NFS: initiated pgio call (req 0:42/13936344780148527098, 1048576 bytes @ offset 4194304)

[1384615.123444] --> nfs4_setup_sequence clp ffff8ef87a8c9c00 session ffff8ef3a3324000 sr_slot 4294967295

[1384615.123446] --> nfs41_setup_sequence

[1384615.123449] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31

[1384615.123453] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0

[1384615.123456] <-- nfs41_setup_sequence slotid=0 seqid=12954

[1384615.123458] <-- nfs4_setup_sequence status=0

[1384615.123464] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=12954 slotid=0 max_slotid=0 cache_this=1

[1384615.123952] NFS: initiated pgio call (req 0:42/13936344780148527098, 1048576 bytes @ offset 5242880)

[1384615.123958] --> nfs4_setup_sequence clp ffff8ef87a8c9c00 session ffff8ef3a3324000 sr_slot 4294967295

[1384615.123959] --> nfs41_setup_sequence

[1384615.123960] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31

[1384615.123961] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1

[1384615.123961] <-- nfs41_setup_sequence slotid=1 seqid=419

[1384615.123962] <-- nfs4_setup_sequence status=0

[1384615.124040] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=419 slotid=1 max_slotid=1 cache_this=1

[1384615.124042] NFS: initiated pgio call (req 0:42/13936344780148527098, 1048576 bytes @ offset 6291456)

[1384615.124046] --> nfs4_setup_sequence clp ffff8ef87a8c9c00 session ffff8ef3a3324000 sr_slot 4294967295

[1384615.124047] --> nfs41_setup_sequence

[1384615.124048] --> nfs4_alloc_slot used_slots=0003 highest_used=1 max_slots=31

[1384615.124049] <-- nfs4_alloc_slot used_slots=0007 highest_used=2 slotid=2

[1384615.124049] <-- nfs41_setup_sequence slotid=2 seqid=399

[1384615.124050] <-- nfs4_setup_sequence status=0

[1384615.124127] NFS: initiated pgio call (req 0:42/13936344780148527098, 1048576 bytes @ offset 7340032)

[1384615.124131] --> nfs4_setup_sequence clp ffff8ef87a8c9c00 session ffff8ef3a3324000 sr_slot 4294967295

[1384615.124132] --> nfs41_setup_sequence

[1384615.124132] --> nfs4_alloc_slot used_slots=0007 highest_used=2 max_slots=31

[1384615.124133] <-- nfs4_alloc_slot used_slots=000f highest_used=3 slotid=3

[1384615.124134] <-- nfs41_setup_sequence slotid=3 seqid=384

[1384615.124135] <-- nfs4_setup_sequence status=0

[1384615.124604] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=399 slotid=2 max_slotid=3 cache_this=1

[1384615.124718] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=384 slotid=3 max_slotid=3 cache_this=1

[1384615.124910] NFS: nfs_pgio_result: 63166, (status 1048576)

[1384615.124911] --> nfs4_alloc_slot used_slots=000f highest_used=3 max_slots=31

[1384615.124912] <-- nfs4_alloc_slot used_slots=001f highest_used=4 slotid=4

[1384615.124913] nfs4_free_slot: slotid 4 highest_used_slotid 3

[1384615.124914] nfs41_sequence_process: Error 0 free the slot

[1384615.124914] nfs4_free_slot: slotid 0 highest_used_slotid 3

[1384615.129401] NFS: nfs_pgio_result: 63168, (status 1048576)

[1384615.129402] --> nfs4_alloc_slot used_slots=000e highest_used=3 max_slots=31

[1384615.129403] <-- nfs4_alloc_slot used_slots=000f highest_used=3 slotid=0

[1384615.129403] nfs4_free_slot: slotid 0 highest_used_slotid 3

[1384615.129404] nfs41_sequence_process: Error 0 free the slot

[1384615.129405] nfs4_free_slot: slotid 2 highest_used_slotid 3

[1384615.130801] NFS: nfs_pgio_result: 63169, (status 1048576)

[1384615.130806] --> nfs4_alloc_slot used_slots=000a highest_used=3 max_slots=31

[1384615.130808] <-- nfs4_alloc_slot used_slots=000b highest_used=3 slotid=0

[1384615.130809] nfs4_free_slot: slotid 0 highest_used_slotid 3

[1384615.130810] nfs41_sequence_process: Error 0 free the slot

[1384615.130811] nfs4_free_slot: slotid 3 highest_used_slotid 1

[1384615.130840] NFS: nfs_pgio_result: 63167, (status 1048576)

[1384615.130843] --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=31

[1384615.130844] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0

[1384615.130846] nfs4_free_slot: slotid 0 highest_used_slotid 1

[1384615.130847] nfs41_sequence_process: Error 0 free the slot

[1384615.130848] nfs4_free_slot: slotid 1 highest_used_slotid 4294967295

[1384615.130977] NFS: initiated commit call

[1384615.130981] --> nfs4_setup_sequence clp ffff8ef87a8c9c00 session ffff8ef3a3324000 sr_slot 4294967295

[1384615.130982] --> nfs41_setup_sequence

[1384615.130983] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31

[1384615.130984] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0

[1384615.130986] <-- nfs41_setup_sequence slotid=0 seqid=12955

[1384615.130987] <-- nfs4_setup_sequence status=0

[1384615.130993] encode_sequence: sessionid=1575630241:2734339359:6:0 seqid=12955 slotid=0 max_slotid=0 cache_this=1

[1384615.131197] NFS: 63170 nfs_commit_done (status 0)

[1384615.131199] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31

[1384615.131200] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1

[1384615.131201] nfs4_free_slot: slotid 1 highest_used_slotid 0

[1384615.131202] nfs41_sequence_process: Error 0 free the slot

[1384615.131204] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295

[1384615.131208] NFS: 63170 commit returned 0

part of nfsd debug msg:

[1452705.130491] nfsd4_sequence: slotid 6

[1452705.130492] check_slot_seqid enter. seqid 80 slot_seqid 79

[1452705.130493] nfsv4 compound op ffff8f08b6d12080 opcnt 3 #1: 53: status 0

[1452705.130494] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.130495] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000003 5de47b5c)

[1452705.130500] nfsv4 compound op ffff8f08b6d12080 opcnt 3 #2: 22: status 0

[1452705.130500] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.161182] nfsv4 compound op ffff8f08b77b0080 opcnt 3 #3: 25: status 0

[1452705.161182] nfsv4 compound returned 0

[1452705.161183] --> nfsd4_store_cache_entry slot ffff8ef8632a5000

[1452705.162038] nfsd_dispatch: vers 4 proc 1

[1452705.162040] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.162041] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.162042] nfsd4_sequence: slotid 4

[1452705.162042] check_slot_seqid enter. seqid 77 slot_seqid 76

[1452705.162044] nfsv4 compound op ffff8f08b77b0080 opcnt 3 #1: 53: status 0

[1452705.162044] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.162046] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000003 5de47b5c)

[1452705.162050] nfsv4 compound op ffff8f08b77b0080 opcnt 3 #2: 22: status 0

[1452705.162051] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.162575] nfsv4 compound op ffff8f08737ce080 opcnt 3 #3: 25: status 0

[1452705.162576] nfsv4 compound returned 0

[1452705.162577] --> nfsd4_store_cache_entry slot ffff8ef881457000

[1452705.163174] nfsd_dispatch: vers 4 proc 1

[1452705.163176] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.163177] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.163178] nfsd4_sequence: slotid 24

[1452705.163178] check_slot_seqid enter. seqid 74 slot_seqid 73

[1452705.163180] nfsv4 compound op ffff8f08737ce080 opcnt 3 #1: 53: status 0

[1452705.163180] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.163182] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000003 5de47b5c)

[1452705.163187] nfsv4 compound op ffff8f08737ce080 opcnt 3 #2: 22: status 0

[1452705.163187] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.201876] nfsv4 compound op ffff8f08abfae080 opcnt 3 #3: 25: status 0

[1452705.201877] nfsv4 compound returned 0

[1452705.201878] --> nfsd4_store_cache_entry slot ffff8ef41bf84000

[1452705.202206] nfsv4 compound op ffff8f0873513080 opcnt 3 #3: 25: status 0

[1452705.202207] nfsv4 compound returned 0

[1452705.202208] --> nfsd4_store_cache_entry slot ffff8ef8b7a2d000

[1452705.202571] nfsv4 compound op ffff8f08b6884080 opcnt 3 #3: 25: status 0

[1452705.202573] nfsv4 compound returned 0

[1452705.202574] --> nfsd4_store_cache_entry slot ffff8ef88130e000

[1452705.202772] nfsv4 compound op ffff8f08b5f9c080 opcnt 3 #3: 25: status 0

[1452705.202773] nfsv4 compound returned 0

[1452705.202773] --> nfsd4_store_cache_entry slot ffff8ef305e53000

[1452705.202793] nfsd_dispatch: vers 4 proc 1

[1452705.202795] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.202796] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.202797] nfsd4_sequence: slotid 3

[1452705.202798] check_slot_seqid enter. seqid 77 slot_seqid 76

[1452705.202799] nfsv4 compound op ffff8f08abfae080 opcnt 3 #1: 53: status 0

[1452705.202800] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.202801] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000003 5de47b5c)

[1452705.202805] nfsv4 compound op ffff8f08abfae080 opcnt 3 #2: 22: status 0

[1452705.202806] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.203487] nfsd_dispatch: vers 4 proc 1

[1452705.203488] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.203489] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.203490] nfsd4_sequence: slotid 9

[1452705.203491] check_slot_seqid enter. seqid 78 slot_seqid 77

[1452705.203492] nfsv4 compound op ffff8f0873513080 opcnt 3 #1: 53: status 0

[1452705.203493] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.203494] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000003 5de47b5c)

[1452705.203499] nfsv4 compound op ffff8f0873513080 opcnt 3 #2: 22: status 0

[1452705.203500] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.204004] nfsd_dispatch: vers 4 proc 1

[1452705.204005] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.204006] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.204007] nfsd4_sequence: slotid 30

[1452705.204008] check_slot_seqid enter. seqid 78 slot_seqid 77

[1452705.204009] nfsv4 compound op ffff8f08b6884080 opcnt 3 #1: 53: status 0

[1452705.204009] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.204011] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000001 5de47a22)

[1452705.204017] nfsv4 compound op ffff8f08b6884080 opcnt 3 #2: 22: status 0

[1452705.204017] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.204362] nfsd_dispatch: vers 4 proc 1

[1452705.204363] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.204364] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.204365] nfsd4_sequence: slotid 12

[1452705.204366] check_slot_seqid enter. seqid 76 slot_seqid 75

[1452705.204367] nfsv4 compound op ffff8f08b5f9c080 opcnt 3 #1: 53: status 0

[1452705.204367] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.204369] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000001 5de47a22)

[1452705.204374] nfsv4 compound op ffff8f08b5f9c080 opcnt 3 #2: 22: status 0

[1452705.204375] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.225062] nfsv4 compound op ffff8f07bab83080 opcnt 3 #3: 25: status 0

[1452705.225063] nfsv4 compound returned 0

[1452705.225064] --> nfsd4_store_cache_entry slot ffff8ef39fbfc000

[1452705.225134] nfsv4 compound op ffff8f08b6884080 opcnt 3 #3: 25: status 0

[1452705.225135] nfsv4 compound returned 0

[1452705.225136] --> nfsd4_store_cache_entry slot ffff8ef881488000

[1452705.225713] nfsv4 compound op ffff8f08b5f9c080 opcnt 3 #3: 25: status 0

[1452705.225715] nfsv4 compound returned 0

[1452705.225716] --> nfsd4_store_cache_entry slot ffff8ef869d33000

[1452705.225824] nfsd_dispatch: vers 4 proc 1

[1452705.225825] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.225826] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.225827] nfsd4_sequence: slotid 29

[1452705.225828] check_slot_seqid enter. seqid 77 slot_seqid 76

[1452705.225829] nfsv4 compound op ffff8f07bab83080 opcnt 3 #1: 53: status 0

[1452705.225830] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.225831] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000001 5de47a22)

[1452705.225836] nfsv4 compound op ffff8f07bab83080 opcnt 3 #2: 22: status 0

[1452705.225836] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.226527] nfsd_dispatch: vers 4 proc 1

[1452705.226528] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.226529] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.226530] nfsd4_sequence: slotid 15

[1452705.226531] check_slot_seqid enter. seqid 76 slot_seqid 75

[1452705.226532] nfsv4 compound op ffff8f08b6884080 opcnt 3 #1: 53: status 0

[1452705.226533] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.226534] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000001 5de47a22)

[1452705.226538] nfsv4 compound op ffff8f08b6884080 opcnt 3 #2: 22: status 0

[1452705.226539] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.226988] nfsd_dispatch: vers 4 proc 1

[1452705.226990] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.226991] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.226991] nfsd4_sequence: slotid 14

[1452705.226992] check_slot_seqid enter. seqid 77 slot_seqid 76

[1452705.226993] nfsv4 compound op ffff8f08b5f9c080 opcnt 3 #1: 53: status 0

[1452705.226994] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.226995] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000001 5de47a22)

[1452705.226999] nfsv4 compound op ffff8f08b5f9c080 opcnt 3 #2: 22: status 0

[1452705.227000] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.233940] nfsv4 compound op ffff8f08b6d12080 opcnt 3 #3: 25: status 0

[1452705.233941] nfsv4 compound returned 0

[1452705.233941] --> nfsd4_store_cache_entry slot ffff8ef3902a0000

[1452705.234719] nfsd_dispatch: vers 4 proc 1

[1452705.234721] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.234722] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.234723] nfsd4_sequence: slotid 5

[1452705.234723] check_slot_seqid enter. seqid 79 slot_seqid 78

[1452705.234724] nfsv4 compound op ffff8f08b6d12080 opcnt 3 #1: 53: status 0

[1452705.234725] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.234726] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000001 5de47a22)

[1452705.234731] nfsv4 compound op ffff8f08b6d12080 opcnt 3 #2: 22: status 0

[1452705.234731] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.251907] nfsv4 compound op ffff8f08737ce080 opcnt 3 #3: 25: status 0

[1452705.251908] nfsv4 compound returned 0

[1452705.251909] --> nfsd4_store_cache_entry slot ffff8ef37d59b000

[1452705.252482] nfsd_dispatch: vers 4 proc 1

[1452705.252483] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.252484] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.252485] nfsd4_sequence: slotid 17

[1452705.252486] check_slot_seqid enter. seqid 83 slot_seqid 82

[1452705.252487] nfsv4 compound op ffff8f08737ce080 opcnt 3 #1: 53: status 0

[1452705.252487] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.252489] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000001 5de47a22)

[1452705.252494] nfsv4 compound op ffff8f08737ce080 opcnt 3 #2: 22: status 0

[1452705.252495] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.257202] nfsv4 compound op ffff8f08b77b0080 opcnt 3 #3: 25: status 0

[1452705.257203] nfsv4 compound returned 0

[1452705.257204] --> nfsd4_store_cache_entry slot ffff8ef86315d000

[1452705.257369] nfsv4 compound op ffff8f08abfae080 opcnt 3 #3: 25: status 0

[1452705.257370] nfsv4 compound returned 0

[1452705.257371] --> nfsd4_store_cache_entry slot ffff8ef394617000

[1452705.257486] nfsv4 compound op ffff8f0873513080 opcnt 3 #3: 25: status 0

[1452705.257487] nfsv4 compound returned 0

[1452705.257488] --> nfsd4_store_cache_entry slot ffff8ef8b44fc000

[1452705.257987] nfsd_dispatch: vers 4 proc 1

[1452705.257988] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.257989] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.257990] nfsd4_sequence: slotid 8

[1452705.257991] check_slot_seqid enter. seqid 83 slot_seqid 82

[1452705.257992] nfsv4 compound op ffff8f08b77b0080 opcnt 3 #1: 53: status 0

[1452705.257993] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.257994] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000001 5de47a22)

[1452705.257999] nfsv4 compound op ffff8f08b77b0080 opcnt 3 #2: 22: status 0

[1452705.258000] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.258513] nfsd_dispatch: vers 4 proc 1

[1452705.258515] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.258516] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.258517] nfsd4_sequence: slotid 10

[1452705.258517] check_slot_seqid enter. seqid 77 slot_seqid 76

[1452705.258519] nfsv4 compound op ffff8f08abfae080 opcnt 3 #1: 53: status 0

[1452705.258519] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.258521] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000001 5de47a22)

[1452705.258525] nfsv4 compound op ffff8f08abfae080 opcnt 3 #2: 22: status 0

[1452705.258526] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.258975] nfsd_dispatch: vers 4 proc 1

[1452705.258977] nfsv4 compound op #1/3: 53 (OP_SEQUENCE)

[1452705.258978] __find_in_sessionid_hashtbl: 1575630241:2734339361:8:0

[1452705.258979] nfsd4_sequence: slotid 22

[1452705.258980] check_slot_seqid enter. seqid 80 slot_seqid 79

[1452705.258981] nfsv4 compound op ffff8f0873513080 opcnt 3 #1: 53: status 0

[1452705.258982] nfsv4 compound op #2/3: 22 (OP_PUTFH)

[1452705.258984] nfsd: fh_verify(44: f3010001 00000000 00000019 5be19baf 00000001 5de47a22)

[1452705.258989] nfsv4 compound op ffff8f0873513080 opcnt 3 #2: 22: status 0

[1452705.258990] nfsv4 compound op #3/3: 25 (OP_READ)

[1452705.267050] nfsv4 compound op ffff8f07bab83080 opcnt 3 #3: 25: status 0

[1452705.267051] nfsv4 compound returned 0

[1452705.267052] --> nfsd4_store_cache_entry slot ffff8ef42d7bf000

Can anybody help me deal with this problem?

Thanks for your concern & Best Regards,

Jia

在 2015年1月17日星期六 UTC+8上午11:28:11，Chen Bill写道：

[do...@physbio-tech.net]

I am assuming BeeGFS uses RDMA for its mounts while NFS isn't by default.

Can you try NFS over RDMA?  Also did you try mounting with "async" as well as increasing "rsize" and "wsize?"

[Schweiss, Chip]

You are not alone.   I'm seeing almost exactly the same thing here.

I'm using CentOS 7 as the NFS gateway.   

I cannot use RDMA for these NFS mounts.  This is kind of the point of the gateway.   

Hopefully, someone with better success can chime in on this issue.

-Chip

--
You received this message because you are subscribed to the Google Groups "beegfs-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/69e80266-382b-4547-bc23-9e77c5b50871%40googlegroups.com.

[Douglas Duckworth]

Even with TCP as the transport you should be able to max out at nearly 1GB/s.  RDMA will allow throughput above that assuming the disks can handle.

It sounds like BeeGFS isn't a good replacement for Lustre or GPFS.

Thanks,

Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit

Weill Cornell Medicine

1300 York Avenue

New York, NY 10065

E: do...@med.cornell.edu
O: 212-746-6305
F: 212-746-8690

You received this message because you are subscribed to a topic in the Google Groups "beegfs-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fhgfs-user/9UJ7Bbnjszc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fhgfs-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/CALeZrrR6oi00-A1i3Ro-3TsBLhsx5R-OYLONWPkp6tqN%3DyMNPg%40mail.gmail.com.

[Bernd Schubert]

Hmm, so back in 2014 I reached around 2GB/s, I think. NFS-over-RDMA
didn't work sufficiently stable, so I used IPoIB. Now that was with the
'paged' mode and an additional tuning setting to avoid cache or
attribute invalidation all the time. I think we had also internally
discussed that the default 'buffered' mode reached the same numbers,
with that specific tuning (I hadn't tested that myself, as NFS using the
page anyway, another BeeGFS cache wouldn't have mattered, imho). Now, I
simply do not remember that specific tuning setting anymore, but I'm
sure it was documented in the wiki made for paying customers.

Grepping a bit through the sources, I *think* it was
tuneFileSubentryCacheValidityMS. Maybe try to set this to something like
2000ms.
Unfortunately, I really don't have the time to try this out myself any
time soon.

Hope it helps,
Bernd

On 2/13/20 4:43 PM, Douglas Duckworth wrote:
> Even with TCP as the transport you should be able to max out at nearly
> 1GB/s.  RDMA will allow throughput above that assuming the disks can handle.
>
> It sounds like BeeGFS isn't a good replacement for Lustre or GPFS.
>
>
> Thanks,
>
> Douglas Duckworth, MSc, LFCS
> HPC System Administrator

> Scientific Computing Unit <https://scu.med.cornell.edu/>

> Weill Cornell Medicine
> 1300 York Avenue
> New York, NY 10065

> E: do...@med.cornell.edu <mailto:do...@med.cornell.edu>

> <http://127.0.0.0/20(rw,no_subtree_check,crossmnt,insecure,root_squash,fsid=0,anonuid=1001,anongid=100)> 

> <mailto:fhgfs-user+...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/fhgfs-user/69e80266-382b-4547-bc23-9e77c5b50871%40googlegroups.com

> <https://groups.google.com/d/msgid/fhgfs-user/69e80266-382b-4547-bc23-9e77c5b50871%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "beegfs-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/fhgfs-user/9UJ7Bbnjszc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> fhgfs-user+...@googlegroups.com

> <mailto:fhgfs-user+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/fhgfs-user/CALeZrrR6oi00-A1i3Ro-3TsBLhsx5R-OYLONWPkp6tqN%3DyMNPg%40mail.gmail.com
> <https://groups.google.com/d/msgid/fhgfs-user/CALeZrrR6oi00-A1i3Ro-3TsBLhsx5R-OYLONWPkp6tqN%3DyMNPg%40mail.gmail.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fhgfs-user+...@googlegroups.com

> <mailto:fhgfs-user+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/fhgfs-user/CAAKHBKmYAMZeFjiaKvUFkM6YnTGxLVbUshzqwA56F1NRyJon_A%40mail.gmail.com
> <https://groups.google.com/d/msgid/fhgfs-user/CAAKHBKmYAMZeFjiaKvUFkM6YnTGxLVbUshzqwA56F1NRyJon_A%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Lehmann, Greg (IM&T, Pullenvale)]

And yet people use it anyway. It must have something going for it. Price perhaps?

 

Greg Lehmann

Senior High Performance Data Specialist |  CSIRO 

Greg.L...@csiro.au  |  +61 7 3327 4137 | 

To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/CAAKHBKmYAMZeFjiaKvUFkM6YnTGxLVbUshzqwA56F1NRyJon_A%40mail.gmail.com.

[James Burton]

I'm not sure why you are exporting BeeGFS as an NFS share instead of using the BeeGFS client to access the filesystem directly. 

To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/MEAPR01MB45830F63C270E39FC210F17DF01A0%40MEAPR01MB4583.ausprd01.prod.outlook.com.

--

James Burton

OS and Storage Architect

Advanced Computing Infrastructure

Clemson University Computing and Information Technology

340 Computer Court

Anderson, SC 29625

(864) 656-9047

[Schweiss, Chip]

On Wed, Feb 26, 2020 at 10:02 AM James Burton <jbu...@g.clemson.edu> wrote:

I'm not sure why you are exporting BeeGFS as an NFS share instead of using the BeeGFS client to access the filesystem directly. 

Simple.  Access control across networks.   Not every machine that needs to mount the data on a BeeGFS has the same security controls.  NFS allows IP based ACL to specific export directories.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/CAAtNLqnJ0%3DYZ38wx7%2BmF%3D-aqmG7dK7O8W5GnWRd6x4XEQ1mq3Q%40mail.gmail.com.

[James Burton]

That makes sense. BeeGFS might not be the best choice for this use case.

To answer Greg's hypothetical question, what BeeGFS has going for it is Lustre-like performance without a lot of Lustre's headaches. But it doesn't have all the features that other file systems do. It's a good product for what it does, but there's a lot it doesn't do.

To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/CALeZrrQx_2UpBPJ5XB6vHPnN4u%3Df%3D4XtvK_B3GGNWuzPVvzpaA%40mail.gmail.com.