Needs some helps on S3backer's performance

[zhang leo]

hi, 

Thanks for the great effort, I really appreciate for s3backer's thinking and design, it gives me chance to deal with S3 much more efficiently.

However, I have a performance question asking for help.

I set up S3backer with the --test option, backed by local filesystem /data, and run a "dd" test to check S3backer's speed, then I run same "dd" test but on the local /data filesystem, the dd speed decrease more than 10 times, from 241MB/s to 18MB/s, Please give me some guiding on how to  increase the S3backer's speed, thank you in advance.

the setup and test result are as below:

1. create S3backer using below pararmeters

-------------------------------------------------------------------------------------------------------

# cat s3backer_config

--filename=backedfile

--listBlocks

--size=500G

--blockSize=1M

--prefix=poolname/

--debug

--force

--blockCacheThreads=100

--blockCacheSize=32000

--md5CacheSize=32000

# s3backer --configFile=/root/s3backer_config --test /data /root/mnt-s3b/

s3backer: auto-detection disabled; using configured block size 1m and file size 500g

s3backer: listing non-zero blocks...done

s3backer: found 0 non-zero blocks

2. create upper filesystem and mount it 

# mkfs.xfs -f   /root/mnt-s3b/backedfile 

# mount -o loop /root/mnt-s3b/backedfile /s3test

3. run dd test on /data and /s3test to compare

on local /data:

-------------------------------------------------------------------------------------------------------

# time dd if=2Gfile of=/data/testfile1 bs=1M conv=fsync

2285+1 records in

2285+1 records out

2396918784 bytes (2.4 GB) copied, 9.9441 s, 241 MB/s

on /s3test using S3backer:

-------------------------------------------------------------------------------------------------------

# time dd if=2Gfile of=/s3test/testfile1 bs=1M conv=fsync

2285+1 records in

2285+1 records out

2396918784 bytes (2.4 GB) copied, 132.508 s, 18.1 MB/s

4. something about mount-point and filesystems

-------------------------------------------------------------------------------------------------------

# mount | egrep 'data|s3'

/dev/mapper/datavg-datalv on /data type xfs (rw,relatime,attr2,inode64,noquota)

file:///data/poolname/ on /root/mnt-s3b type fuse.s3backer (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

/root/mnt-s3b/backedfile on /s3test type xfs (rw,relatime,attr2,inode64,noquota)

# df -h | egrep 'data|s3'

/dev/mapper/datavg-datalv  1.0T  4.6G 1019G   1% /data

file:///data/poolname/     500G  500G     0 100% /root/mnt-s3b

/dev/loop0                 500G  2.3G  498G   1% /s3test

[Archie Cobbs]

Test mode is for basic testing, not for performance measurement. Besides, the performance of test mode is not relevant to anything.

Moreover, s3backer is not optimized for performance... because what's the point? All performance is going to be dominated by the network I/O. That's why (for example) it's implemented as a userspace daemon (instead of a kernel module) where all traffic goes in and out of the kernel twice on each operation - not exactly a fast way to do things.

-Archie