IO errors with fio disk benchmark program and libaio library

848 views
Skip to first unread message

sto...@hotmail.com

unread,
Jun 27, 2014, 8:08:34 AM6/27/14
to fhgfs...@googlegroups.com
Hello

We are setting up a large fhgfs filesystem and during be benchmarks with rio disk benchmark program I ran into this error:

fio: pid=11908, err=22/file:ioengines.c:395, func=io commit, error=Invalid argument
Jobs: 28 (f=28): [XXXWWWWWWWWWWWWWWWWWWWWWWWWXWWWW] [0.4% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 19m:59s]
seq-write: (groupid=0, jobs=1): err=22 (file:ioengines.c:395, func=io commit, error=Invalid argument): pid=11892: Fri Jun 27 13:39:39 2014
  cpu          : usr=0.00%, sys=0.00%, ctx=4, majf=0, minf=44
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

The fio configuration file is 

[global]
bs=2k
iodepth=32
ioengine=libaio
direct=1
size=10g
runtime=1200
fsync_on_close=1
directory=/scratch/fhgfs/cpt01

[seq-write]
nrfiles=1000
filesize=2k
numjobs=32
rw=write
stonewall

The overall configuration is

[root@XXX1 /etc/fhgfs ] grep -v "#" *.conf | grep -v conf:$
fhgfs-admon.conf:sysMgmtdHost                 = XXX1
fhgfs-admon.conf:logLevel                     = 3
fhgfs-admon.conf:logNoDate                    = false
fhgfs-admon.conf:logStdFile                   = /var/log/fhgfs-admon.log
fhgfs-admon.conf:logNumLines                  = 50000
fhgfs-admon.conf:logNumRotatedFiles           = 2
fhgfs-admon.conf:connPortShift                = 0
fhgfs-admon.conf:connMgmtdPortUDP             = 8008
fhgfs-admon.conf:connMgmtdPortTCP             = 8008
fhgfs-admon.conf:connAdmonPortUDP             = 8007
fhgfs-admon.conf:connMaxInternodeNum          = 3
fhgfs-admon.conf:connNetFilterFile            =
fhgfs-admon.conf:connFallbackExpirationSecs   = 900
fhgfs-admon.conf:connAuthFile                 =
fhgfs-admon.conf:tuneNumWorkers               = 4
fhgfs-admon.conf:runDaemonized                = true
fhgfs-admon.conf:httpPort                     = 8000
fhgfs-admon.conf:queryInterval                = 5
fhgfs-admon.conf:databaseFile                 = /var/lib/fhgfs/fhgfs-admon.db
fhgfs-admon.conf:clearDatabase                = false
fhgfs-admon.conf:mailEnabled                  = false
fhgfs-admon.conf:mailSmtpServer               =
fhgfs-admon.conf:mailSender                   =
fhgfs-admon.conf:mailRecipient                =
fhgfs-admon.conf:mailMinDownTimeSec           = 10
fhgfs-admon.conf:mailResendMailTimeMin        = 60
fhgfs-admon.conf:mailCheckIntervalTimeSec     = 30
fhgfs-client-autobuild.conf:buildArgs=-j8 FHGFS_OPENTK_IBVERBS=1
fhgfs-client-autobuild.conf:buildEnabled=true
fhgfs-client.conf:sysMgmtdHost                  = XXX1
fhgfs-client.conf:logLevel                      = 3
fhgfs-client.conf:logType                       = helperd
fhgfs-client.conf:logClientID                   = false
fhgfs-client.conf:logHelperdIP                  =
fhgfs-client.conf:connPortShift                 = 0
fhgfs-client.conf:connMgmtdPortUDP              = 8008
fhgfs-client.conf:connMgmtdPortTCP              = 8008
fhgfs-client.conf:connClientPortUDP             = 8004
fhgfs-client.conf:connHelperdPortTCP            = 8006
fhgfs-client.conf:connUseSDP                    = false
fhgfs-client.conf:connUseRDMA                   = true
fhgfs-client.conf:connRDMABufSize               = 8192
fhgfs-client.conf:connRDMABufNum                = 128
fhgfs-client.conf:connRDMATypeOfService         = 0
fhgfs-client.conf:connMaxInternodeNum           = 12
fhgfs-client.conf:connInterfacesFile            =
fhgfs-client.conf:connNetFilterFile             =
fhgfs-client.conf:connFallbackExpirationSecs    = 900
fhgfs-client.conf:connCommRetrySecs             = 600
fhgfs-client.conf:connAuthFile                  =
fhgfs-client.conf:tuneNumWorkers                = 0
fhgfs-client.conf:tunePreferredMetaFile         =
fhgfs-client.conf:tunePreferredStorageFile      =
fhgfs-client.conf:tuneFileCacheType             = buffered
fhgfs-client.conf:tuneRemoteFSync               = true
fhgfs-client.conf:tuneUseGlobalFileLocks        = false
fhgfs-client.conf:tuneUseGlobalAppendLocks      = false
fhgfs-client.conf:sysCreateHardlinksAsSymlinks  = false
fhgfs-client.conf:sysMountSanityCheckMS         = 11000
fhgfs-client.conf:sysSyncOnClose                = false
fhgfs-client.conf:sysSessionCheckOnClose        = false
fhgfs-client.conf:quotaEnabled                  = false
fhgfs-helperd.conf:logNoDate          = false
fhgfs-helperd.conf:logStdFile         = /var/log/fhgfs-client.log
fhgfs-helperd.conf:logNumLines        = 50000
fhgfs-helperd.conf:logNumRotatedFiles = 5
fhgfs-helperd.conf:connPortShift      = 0
fhgfs-helperd.conf:connHelperdPortTCP = 8006
fhgfs-helperd.conf:connAuthFile       =
fhgfs-helperd.conf:tuneNumWorkers     = 2
fhgfs-helperd.conf:runDaemonized      = true
fhgfs-libopentk.conf:FORCE_IB=0
fhgfs-meta.conf:sysMgmtdHost                 = XXX1
fhgfs-meta.conf:storeMetaDirectory           = /exports/data/meta1
fhgfs-meta.conf:storeAllowFirstRunInit       = true
fhgfs-meta.conf:logLevel                     = 3
fhgfs-meta.conf:logNoDate                    = false
fhgfs-meta.conf:logStdFile                   = /var/log/fhgfs-meta.log
fhgfs-meta.conf:logNumLines                  = 50000
fhgfs-meta.conf:logNumRotatedFiles           = 5
fhgfs-meta.conf:connPortShift                = 0
fhgfs-meta.conf:connMgmtdPortUDP             = 8008
fhgfs-meta.conf:connMgmtdPortTCP             = 8008
fhgfs-meta.conf:connMetaPortUDP              = 8005
fhgfs-meta.conf:connMetaPortTCP              = 8005
fhgfs-meta.conf:connUseSDP                   = false
fhgfs-meta.conf:connUseRDMA                  = true
fhgfs-meta.conf:connRDMATypeOfService        = 0
fhgfs-meta.conf:connBacklogTCP               = 128
fhgfs-meta.conf:connMaxInternodeNum          = 32
fhgfs-meta.conf:connInterfacesFile           =
fhgfs-meta.conf:connNetFilterFile            =
fhgfs-meta.conf:connFallbackExpirationSecs   = 900
fhgfs-meta.conf:connAuthFile                 =
fhgfs-meta.conf:storeUseExtendedAttribs      = false
fhgfs-meta.conf:tuneNumWorkers               = 0
fhgfs-meta.conf:tuneBindToNumaZone           =
fhgfs-meta.conf:tuneTargetChooser            = randomized
fhgfs-meta.conf:tuneRotateMirrorTargets      = false
fhgfs-meta.conf:tuneUsePerUserMsgQueues      = false
fhgfs-meta.conf:runDaemonized                = true
fhgfs-mgmtd.conf:storeMgmtdDirectory            = /exports/data/mgmtd
fhgfs-mgmtd.conf:storeAllowFirstRunInit         = true
fhgfs-mgmtd.conf:sysAllowNewServers             = true
fhgfs-mgmtd.conf:logLevel                       = 3
fhgfs-mgmtd.conf:logNoDate                      = false
fhgfs-mgmtd.conf:logStdFile                     = /var/log/fhgfs-mgmtd.log
fhgfs-mgmtd.conf:logNumLines                    = 50000
fhgfs-mgmtd.conf:logNumRotatedFiles             = 5
fhgfs-mgmtd.conf:connPortShift                  = 0
fhgfs-mgmtd.conf:connMgmtdPortUDP               = 8008
fhgfs-mgmtd.conf:connMgmtdPortTCP               = 8008
fhgfs-mgmtd.conf:connBacklogTCP                 = 128
fhgfs-mgmtd.conf:connInterfacesFile             =
fhgfs-mgmtd.conf:connNetFilterFile              =
fhgfs-mgmtd.conf:connAuthFile                   =
fhgfs-mgmtd.conf:tuneNumWorkers                 = 4
fhgfs-mgmtd.conf:tuneClientAutoRemoveMins       = 30
fhgfs-mgmtd.conf:tuneMetaSpaceLowLimit          = 10G
fhgfs-mgmtd.conf:tuneMetaSpaceEmergencyLimit    = 3G
fhgfs-mgmtd.conf:tuneStorageSpaceLowLimit       = 512G
fhgfs-mgmtd.conf:tuneStorageSpaceEmergencyLimit = 10G
fhgfs-mgmtd.conf:runDaemonized                  = true
fhgfs-mounts.conf:/scratch/fhgfs /etc/fhgfs/fhgfs-client.conf
fhgfs-storage.conf:sysMgmtdHost                 = XXX1
fhgfs-storage.conf:storeStorageDirectory        = /exports/data/storage1,/exports/data/storage2,/exports/data/storage3,/exports/data/storage4,/exports/data/storage5
fhgfs-storage.conf:storeAllowFirstRunInit       = true
fhgfs-storage.conf:logLevel                     = 3
fhgfs-storage.conf:logNoDate                    = false
fhgfs-storage.conf:logStdFile                   = /var/log/fhgfs-storage.log
fhgfs-storage.conf:logNumLines                  = 50000
fhgfs-storage.conf:logNumRotatedFiles           = 5
fhgfs-storage.conf:connPortShift                = 0
fhgfs-storage.conf:connMgmtdPortUDP             = 8008
fhgfs-storage.conf:connMgmtdPortTCP             = 8008
fhgfs-storage.conf:connStoragePortUDP           = 8003
fhgfs-storage.conf:connStoragePortTCP           = 8003
fhgfs-storage.conf:connUseSDP                   = false
fhgfs-storage.conf:connUseRDMA                  = true
fhgfs-storage.conf:connRDMATypeOfService        = 0
fhgfs-storage.conf:connBacklogTCP               = 128
fhgfs-storage.conf:connInterfacesFile           =
fhgfs-storage.conf:connNetFilterFile            =
fhgfs-storage.conf:connAuthFile                 =
fhgfs-storage.conf:tuneNumWorkers               = 12
fhgfs-storage.conf:tuneBindToNumaZone           =
fhgfs-storage.conf:tuneWorkerBufSize            = 4m
fhgfs-storage.conf:tuneFileReadSize             = 32k
fhgfs-storage.conf:tuneFileReadAheadTriggerSize = 4m
fhgfs-storage.conf:tuneFileReadAheadSize        = 0m
fhgfs-storage.conf:tuneFileWriteSize            = 64k
fhgfs-storage.conf:tuneFileWriteSyncSize        = 0m
fhgfs-storage.conf:tuneUsePerUserMsgQueues      = false
fhgfs-storage.conf:runDaemonized                = true

Am I missing something?

Regards

Christian Mohrbacher

unread,
Jun 27, 2014, 10:12:48 AM6/27/14
to fhgfs...@googlegroups.com
Hi,
hmmm, not sure what happens here and why that fails. Could you please provide the FhGFS Log files (/var/log/fhgfs-*.log). Besides that it may be useful to strace fio to see where exactly it fails.

Regards,
Christian
--
You received this message because you are subscribed to the Google Groups "fhgfs-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

sto...@hotmail.com

unread,
Jun 27, 2014, 11:05:54 AM6/27/14
to fhgfs...@googlegroups.com, christian....@itwm.fraunhofer.de
fhgfs_logs.tgz
strace.out

Pete Sero

unread,
Jun 29, 2014, 7:54:56 AM6/29/14
to fhgfs...@googlegroups.com, christian....@itwm.fraunhofer.de
from strace.out:

stat("/tmp//tmp/fio-dump-status", 0x7fff873d23e0) = -1 ENOENT (No such file or directory)

Can you check the tmp dir config for fio?

E.g. is a full path (/tmp/fio-dump-status) specified
where only a file name (fio-dump-status) is expected?
fhgfs_logs.tgz
strace.out

sto...@hotmail.com

unread,
Jul 1, 2014, 8:31:46 AM7/1/14
to fhgfs...@googlegroups.com, christian....@itwm.fraunhofer.de
Hello

Same benchmark on same server but on a local ext4  has the same error message but completes successfully

Best regards

Reply all
Reply to author
Forward
0 new messages