S3QL on local storage backend provided by NFS - suspected incompatibility?

[Super Strobi]

System: fedora 20
Name: s3ql
Arch: i686
Version: 2.9 (latest available through yum)
Release:: 1.fc20
"Local" storage backend, hosted through NFS share from server:
mount: 192.168.1.195:/nfs/strobiserver on /mnt/local-nfs type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.195,mountvers=3,mountport=59909,mountproto=udp,local_lock=none,addr=192.168.1.195)

Here the steps I performed:

Created filesystem: mkfs.s3ql local:///mnt/local-nfs/myfsdata-ttt/  --> OK
Mounted filesystem: mount.s3ql local:///mnt/local-nfs/myfsdata-ttt /mnt/s3ql-ttt-local/ --allow-other --nfs --> OK (allow-other as s3ql-ttt-local gets exported by nfs/smb)
>> Using 4 upload threads.
>> Autodetected 4052 file descriptors available for cache entries

Filesystem shows up in both linux and through CIFS, copying files works.

Symptoms are coming up like an increasing load:

top - 14:39:08 up  5:00,  3 users,  load average: 5.05, 3.56, 1.76
Tasks: 152 total,   2 running, 150 sleeping,   0 stopped,   0 zombie
%Cpu(s): 80.2 us, 14.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  5.3 si,  0.0 st
KiB Mem:   3101920 total,  3011764 used,    90156 free,    12516 buffers
KiB Swap:  5210108 total,     4416 used,  5205692 free,  2565888 cached

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2977 root      20   0  589884 255564   4648 S **181.3**  8.2  13:32.46 mount.s3ql

I got cpu0 and cpu1 of this kind:

cpu family      : 6
model           : 14
model name      : Intel(R) Core(TM) Duo CPU      T2600  @ 2.16GHz
stepping        : 12
microcode       : 0x54
cpu MHz         : 2167.000
cache size      : 2048 KB

While copy is still running, s3qlstat replies:
Directory entries:    2398
Inodes:               2400
Data blocks:          390
Total data size:      10680.65 MiB
After de-duplication: 3599.33 MiB (33.70% of total)
After compression:    3524.60 MiB (33.00% of total, 97.92% of de-duplicated)
Database size:        0.45 MiB (uncompressed)
(some values do not take into account not-yet-uploaded dirty blocks in cache)

At the time of writing, s3qlstat replies (which does not move anymore):

Directory entries:    303
Inodes:               305
Data blocks:          396
Total data size:      40149.03 MiB
After de-duplication: 3498.80 MiB (8.71% of total)
After compression:    3424.95 MiB (8.53% of total, 97.89% of de-duplicated)
Database size:        0.19 MiB (uncompressed)
(some values do not take into account not-yet-uploaded dirty blocks in cache)

du reports:
41112666        s3ql-ttt-local/ (as from s3ql server)
  3531860         myfsdata-ttt/ (as from NFS server)
so the numbers from s3qlstat seem to fit, but I doubt the de-duplication percentage.

Load is still increasing:
top - 19:10:05 up  3:20,  3 users,  load average: 22.00, 21.83, 20.47
Tasks: 167 total,   1 running, 166 sleeping,   0 stopped,   0 zombie
libnuma: Warning: /sys not mounted or invalid. Assuming one node: No such file or directory
%Cpu(s): 16.4 us,  2.4 sy,  0.0 ni,  4.5 id, 76.0 wa,  0.0 hi,  0.7 si,  0.0 st
KiB Mem:   3101920 total,  3018480 used,    83440 free,    17236 buffers

Logfiles in ~/.s3ql report only the s3ql.mount, no error/warning statement afterwards.

**In summary**: is this setup supported by S3QL: local storage backend by NFS share?

I'm retrying now with sshfs as local storage backend.

Please advice if I'm doing something stupid with s3ql :)
Thank you!

[Super Strobi]

Update on sshfs test setup:

s3qlstat reports:
[root@strobiserver /]# s3qlstat /mnt/s3ql-ttt-local/
Directory entries:    1147
Inodes:               1149
Data blocks:          968
Total data size:      29627.09 MiB
After de-duplication: 5648.56 MiB (19.07% of total)
After compression:    5580.27 MiB (18.84% of total, 98.79% of de-duplicated)
Database size:        0.40 MiB (uncompressed)

(some values do not take into account not-yet-uploaded dirty blocks in cache)

Network speed dramatically dropped:
04 xxx.mp3                                                                                                                                  100%   10MB 563.7KB/s   00:18
05 xxx.mp3                                                                                                                                  100% 9448KB 524.9KB/s   00:18
10 xxx.mp3                                                                                                                       100% 6668KB 351.0KB/s   00:19

It still continues... slowly...

load is still acceptable compared to the NFS setup:
top - 20:32:53 up  1:10,  4 users,  load average: 2.38, 2.46, 2.51
Tasks: 157 total,   1 running, 156 sleeping,   0 stopped,   0 zombie
%Cpu(s): 89.2 us,  8.1 sy,  0.0 ni,  0.7 id,  0.2 wa,  0.0 hi,  1.8 si,  0.0 st
KiB Mem:   3101920 total,  2897324 used,   204596 free,     6796 buffers
KiB Swap:  5210108 total,      132 used,  5209976 free,  2542092 cached

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

 1945 root      20   0  607896 160324   4816 S 175.5  5.2  91:28.72 mount.s3ql

[Nikolaus Rath]

Super Strobi <strob...@gmail.com> writes:
> **In summary**: is this setup supported by S3QL: local storage backend by
> NFS share?

Yes, that should work.

Not sure why you're having performance issues. I'd try to isolate the
problem. For example, does the same thing happen if you use the local
backend? Or use the NFS backend, but don't export the mountpoint via
CIFS or NFS?


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Super Strobi]

Thanks Nikolaus for your feedback.

I tried on a "real" local backend, unfortunately after 8GB I ran out of local diskspace (that's why we use NAS'es for a reason ;-) ) - I retry later today on a new spare disk.

On the otherhand, my sshfs testcase - hosting an s3ql filesystem locally, and exported by CIFS ànd local cp - successfully completed its job:

[root@strobiserver /]# s3qlstat /mnt/s3ql-ttt-local/

Directory entries:    13088
Inodes:               13090
Data blocks:          15196
Total data size:      76280.19 MiB
After de-duplication: 74266.21 MiB (97.36% of total)
After compression:    72883.05 MiB (95.55% of total, 98.14% of de-duplicated)
Database size:        4.02 MiB (uncompressed)

(some values do not take into account not-yet-uploaded dirty blocks in cache)

Network still was "slow", but load of server never exceeded 2.5.

I'll destroy this test case, and rebuild one with local backend and report the results later.

PS. as per documentation uncertainty: if the "local" ~/.s3ql is lost, is the s3ql filesystem still mountable?

[Super Strobi]

Update:

with local storage backend, 8GB got copied successfully (by local cp and CIFS) and parsed by s3ql:

[root@strobiserver mnt]# s3qlstat /mnt/s3ql-ttt-local/
Directory entries:    1479
Inodes:               1481
Data blocks:          1543
Total data size:      7922.60 MiB
After de-duplication: 7851.53 MiB (99.10% of total)
After compression:    7673.06 MiB (96.85% of total, 97.73% of de-duplicated)
Database size:        0.56 MiB (uncompressed)

(some values do not take into account not-yet-uploaded dirty blocks in cache)

The load of the server was remarkably higher than with sshfs, but never reached higher than 5.

I retry the case with NFS.

Thanks for reading.

[Super Strobi]

Update:

with "local" storage backend on NFS server, speed of injection of files into the s3ql is higher (factor 3) than with local and sshfs backend (was 1-1.5MB/s) - speed is not my concern, any component crashing and/or rendering this backup space corrupt is!:

104-xxx-unit.mp3                                                                                                    100% 2974KB   5.9MB/s   00:00
112-xxx-unit.mp3                                                                                                    100% 8843KB   8.6MB/s   00:01
116-xxx-unit.mp3                                                                                                    100% 6614KB   6.5MB/s   00:01

Load is bit higher than with local and sshfs too:
top - 13:01:42 up 17:39,  4 users,  load average: 4.99, 4.06, 2.10

But I'm suspecting something else going on, on the system:

Filesystem                           1K-blocks       Used  Available Use% Mounted on
/dev/mapper/vg_strobiserver-lv_root   51475068   36425216   12412028  75% /
devtmpfs                               1544224          0    1544224   0% /dev
tmpfs                                  1550960          0    1550960   0% /dev/shm
tmpfs                                  1550960        612    1550348   1% /run
tmpfs                                  1550960          0    1550960   0% /sys/fs/cgroup
tmpfs                                  1550960      10304    1540656   1% /tmp
/dev/sda1                               487652     182125     275831  40% /boot
/dev/mapper/vg_strobiserver-lv_home   38766032    5762548   31011192  16% /home
192.168.1.195:/nfs/strobiserver     3841069376 1188181376 2574841920  32% /mnt/local-nfs
local:///mnt/local-nfs/myfsdata-ttt 1073740784     854614 1072886171   1% /mnt/s3ql-ttt-local

The lines in bold are changing: the last two are easy to understand (storage backend & s3ql filesystem), but the first one has my /root/.s3ql cache directory. If it's going to continu like that, that would explain why there was nothing in the log file (disk full?). But that should not have happened because, with sshfs backend I got 70GB of real data uploaded and now with NFS also 60GB+.

Network speed did go down from the aforementioned 5MB/s towards 1MB/s, load has not exceeded 5.

Going to an acceptance phase with real data now, will report back if the symptoms happen again.

Bottom-line:
Unclear what happened with my test setup earlier (it did "lock" more than three times in a row), but having restarted from scratch (and a clean reboot of all components) did proof that both NFS & sshfs storage backends with s3ql do work, whereas the s3ql is exposed towards the network by CIFS. Network bandwidth variance is high.

[Super Strobi]

Update:

Just checked the final state of my test setup with the NFS share hosting the s3ql... and my connections hang, load of server is 10+.

Some info:
[root@strobiserver .s3ql]# s3qlstat /mnt/s3ql-ttt-local/
Directory entries:    6495
Inodes:               6497
Data blocks:          7954
Total data size:      71169.19 MiB
After de-duplication: 63789.25 MiB (89.63% of total)
After compression:    63310.29 MiB (88.96% of total, 99.25% of de-duplicated)
Database size:        2.10 MiB (uncompressed)

(some values do not take into account not-yet-uploaded dirty blocks in cache)

df -f "hangs" at prompt (ctrl-C) does work

[root@strobiserver .s3ql]# cat /etc/mtab
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=1544224k,nr_inodes=212798,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/mapper/vg_strobiserver-lv_root / ext4 rw,relatime,data=ordered 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
tmpfs /tmp tmpfs rw 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
sunrpc /proc/fs/nfsd nfsd rw,relatime 0 0
/dev/sda1 /boot ext4 rw,relatime,data=ordered 0 0
/dev/mapper/vg_strobiserver-lv_home /home ext4 rw,relatime,quota,usrquota,data=ordered 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
192.168.1.195:/nfs/strobiserver /mnt/local-nfs nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.195,mountvers=3,mountport=45764,mountproto=udp,local_lock=none,addr=192.168.1.195 0 0
local:///mnt/local-nfs/myfsdata-ttt /mnt/s3ql-ttt-local fuse.s3ql rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0

No info in ~/.s3ql/mount.log.

ls -alR /mnt/s3ql-ttt-local does show the content, but files cannot be opened (hangs)

ls -alR /mnt/local-nfs hangs too.

NFS server has increased cpu activity:
top - 22:40:08 up  9:53,  1 user,  load average: 5.18, 5.02, 4.71
Tasks:  85 total,   3 running,  82 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us, 53.1 sy,  0.0 ni, 46.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:    232448 total,   164032 used,    68416 free,     5824 buffers
KiB Swap:   500672 total,    97088 used,   403584 free,    45184 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND

 6745 root      20   0     0    0    0 R  42.3  0.0  10:10.57 nfsd
 6743 root      20   0     0    0    0 R  40.6  0.0   8:59.98 nfsd
 6748 root      20   0     0    0    0 S  14.9  0.0   9:44.72 nfsd
 6744 root      20   0     0    0    0 S  10.2  0.0  10:49.00 nfsd

Bottom-line: Google learns me that my nfs mount is crashed, I found articles from 2012, caused by lack of (cpu) hardware - don't think that is the root cause here.

I'll let the server "hang" like this during the night, if someone has an idea for debugging/logging, I'll follow-up tomorrow.

Thank you.

[Nikolaus Rath]

On 10/18/2014 12:00 AM, Super Strobi wrote:
> PS. as per documentation uncertainty: if the "local" ~/.s3ql is lost, is
> the s3ql filesystem still mountable?

If the file system was unmounted properly, yes. Otherwise you may loose
all data that was modified since the last metadata upload.

[Nikolaus Rath]

On 10/18/2014 01:48 PM, Super Strobi wrote:
> [root@strobiserver .s3ql]# cat /etc/mtab

[...]

> local:///mnt/local-nfs/myfsdata-ttt /mnt/s3ql-ttt-local fuse.s3ql
> rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
> 0 0

[...]

>
> ls -alR /mnt/s3ql-ttt-local does show the content, but files cannot be
> opened (hangs)
>
> ls -alR /mnt/local-nfs hangs too.

The former is caused by the latter. If requests to the backend directory
hang, S3QL is bound to hang as well once the cache is full. Fix your NFS
mount, and the S3QL mountpoint will become accessible again as well.

[Super Strobi]

Update:

fixing the NFS mount was only doable by restarting the NFS service, the /mnt/local-nfs became accessible, but mount.s3ql did not reconnect (automatically?). So I killed the mount.s3ql, and ran fsck.s3ql against the NFS storage backend, took a while, committed lots of cached stuff, but stopped on this:

Checking directory reachability...
Checking unix conventions...
Checking referential integrity...
Dropping temporary indices...
Dumping metadata...
..objects..
..blocks..
Uncaught top-level exception:
Traceback (most recent call last):
  File "/bin/fsck.s3ql", line 9, in <module>
    load_entry_point('s3ql==2.9', 'console_scripts', 'fsck.s3ql')()
  File "/usr/lib/python3.3/site-packages/s3ql/fsck.py", line 1217, in main
    dump_metadata(db, fh)
  File "/usr/lib/python3.3/site-packages/s3ql/metadata.py", line 137, in dump_metadata
    dump_table(table, order, columns, db=db, fh=fh)
  File "deltadump.pyx", line 317, in s3ql.deltadump.dump_table (src/s3ql/deltadump.c:4096)
  File "deltadump.pyx", line 364, in s3ql.deltadump.dump_table (src/s3ql/deltadump.c:3746)
ValueError: Can't dump NULL values

Mounting is not possible:

[root@strobiserver ~]# mount.s3ql local:///mnt/local-nfs/myfsdata-ttt /mnt/s3ql-ttt-local --allow-other --nfs

Using 4 upload threads.
Autodetected 4052 file descriptors available for cache entries

Enter file system encryption passphrase:
Using cached metadata.
File system damaged or not unmounted cleanly, run fsck!

Rerunning fsck gives same output as above.

Suggestions to get out of this?

Thank you!

[Nikolaus Rath]

On 10/19/2014 05:50 AM, Super Strobi wrote:
> Dumping metadata...
> ..objects..
> ..blocks..
> Uncaught top-level exception:
> Traceback (most recent call last):
> File "/bin/fsck.s3ql", line 9, in <module>
> load_entry_point('s3ql==2.9', 'console_scripts', 'fsck.s3ql')()
> File "/usr/lib/python3.3/site-packages/s3ql/fsck.py", line 1217, in main
> dump_metadata(db, fh)
> File "/usr/lib/python3.3/site-packages/s3ql/metadata.py", line 137, in
> dump_metadata
> dump_table(table, order, columns, db=db, fh=fh)
> File "deltadump.pyx", line 317, in s3ql.deltadump.dump_table
> (src/s3ql/deltadump.c:4096)
> File "deltadump.pyx", line 364, in s3ql.deltadump.dump_table
> (src/s3ql/deltadump.c:3746)
> ValueError: Can't dump NULL values

This looks like a bug. Could you report it
athttps://bitbucket.org/nikratio/s3ql/issues? Thanks!

> Mounting is not possible:
>
> [root@strobiserver ~]# mount.s3ql local:///mnt/local-nfs/myfsdata-ttt
> /mnt/s3ql-ttt-local --allow-other --nfs
> Using 4 upload threads.
> Autodetected 4052 file descriptors available for cache entries
> Enter file system encryption passphrase:
> Using cached metadata.
> File system damaged or not unmounted cleanly, run fsck!
>
> Rerunning fsck gives same output as above.
>
> Suggestions to get out of this?

Does the file system metadata (i.e., file names, sizes, permissions)
contain confidential information? If not, can you upload it somewhere
and send me the link? Then I'll take a look.

Otherwise I can give you some commands to execute, but it'll take
several emails back and forth...

[Super Strobi]

This looks like a bug. Could you report it
athttps://bitbucket.org/nikratio/s3ql/issues? Thanks!

Sure!
 

Does the file system metadata (i.e., file names, sizes, permissions)
contain confidential information? If not, can you upload it somewhere
and send me the link? Then I'll take a look.

I can share if it, however I run into this:

[root@strobiserver local-nfs]# s3qladm download-metadata local:///mnt/local-nfs/myfsdata-ttt/

Enter file system encryption passphrase:

The following backups are available:
 No  Name                    Date
  0  s3ql_metadata           2014-10-18 12:51:43
Enter no to download: 0
Downloading and decompressing s3ql_metadata...
Reading metadata...

Uncaught top-level exception:
Traceback (most recent call last):

  File "/bin/s3qladm", line 9, in <module>
    load_entry_point('s3ql==2.9', 'console_scripts', 's3qladm')()
  File "/usr/lib/python3.3/site-packages/s3ql/adm.py", line 107, in main
    return download_metadata(backend, options.storage_url)
  File "/usr/lib/python3.3/site-packages/s3ql/adm.py", line 160, in download_metadata
    restore_metadata(tmpfh, cachepath + '.db')
  File "/usr/lib/python3.3/site-packages/s3ql/metadata.py", line 86, in restore_metadata
    db = Connection(tmpfile)
  File "/usr/lib/python3.3/site-packages/s3ql/database.py", line 72, in __init__
    cur.execute(s)
  File "src/cursor.c", line 990, in APSWCursor_execute.sqlite3_prepare
  File "src/statementcache.c", line 386, in sqlite3_prepare
apsw.BusyError: BusyError: database is locked

s3ql filesystem is not mounted:

[root@strobiserver local-nfs]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=1544224k,nr_inodes=212798,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/vg_strobiserver-lv_root on / type ext4 (rw,relatime,data=ordered)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
tmpfs on /tmp type tmpfs (rw)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
sunrpc on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)
/dev/mapper/vg_strobiserver-lv_home on /home type ext4 (rw,relatime,quota,usrquota,data=ordered)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
192.168.1.195:/nfs/strobiserver on /mnt/local-nfs type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.195,mountvers=3,mountport=45764,mountproto=udp,local_lock=none,addr=192.168.1.195)
[root@strobiserver local-nfs]#

However I suspect the fsck.s3ql has thrown a lock?
 

Otherwise I can give you some commands to execute, but it'll take
several emails back and  forth...

If you want, you can start a private conversation, but we'll make sure our conclusions end up here so the general audience can consult it later on.

Thank you!

[Nikolaus Rath]

On 10/19/2014 11:05 AM, Super Strobi wrote:
> Does the file system metadata (i.e., file names, sizes, permissions)
> contain confidential information? If not, can you upload it somewhere
> and send me the link? Then I'll take a look.
>
>
> I can share if it, however I run into this:
>
> [root@strobiserver local-nfs]# s3qladm download-metadata
> local:///mnt/local-nfs/myfsdata-ttt/

[...]

Sorry, I should have been more clear. Just upload the .db file from
~/.s3ql/. No need to download anything.

[Super Strobi]

Update:

I investigated bit more on the backgrounds of compression & number of threads, and came to the following conclusion:

Benchmark with sshfs backend:
Preparing test data...
Measuring throughput to cache...
Cache throughput with   4 KiB blocks: 12446 KiB/sec
Cache throughput with   8 KiB blocks: 15351 KiB/sec
Cache throughput with  16 KiB blocks: 18712 KiB/sec
Cache throughput with  32 KiB blocks: 20681 KiB/sec
Cache throughput with  64 KiB blocks: 20703 KiB/sec
Cache throughput with 128 KiB blocks: 20768 KiB/sec
Measuring raw backend throughput..
Backend throughput: 8244 KiB/sec
Test file size: 1431.98 MiB
compressing with lzma-6...
lzma compression speed: 1172 KiB/sec per thread (in)
lzma compression speed: 1166 KiB/sec per thread (out)
compressing with bzip2-6...
bzip2 compression speed: 2076 KiB/sec per thread (in)
bzip2 compression speed: 2064 KiB/sec per thread (out)
compressing with zlib-6...
zlib compression speed: 7921 KiB/sec per thread (in)
zlib compression speed: 7884 KiB/sec per thread (out)

With 128 KiB blocks, maximum performance for different compression
algorithms and thread counts is:

Threads:                              1           2           4           8
Max FS throughput (lzma):     1172 KiB/s   2344 KiB/s   4688 KiB/s   8287 KiB/s
..limited by:                       CPU         CPU         CPU      uplink
Max FS throughput (bzip2):    2076 KiB/s   4152 KiB/s   8292 KiB/s   8292 KiB/s
..limited by:                       CPU         CPU      uplink      uplink
Max FS throughput (zlib):     7921 KiB/s   8283 KiB/s   8283 KiB/s   8283 KiB/s
..limited by:                       CPU      uplink      uplink      uplink

All numbers assume that the test file is representative and that
there are enough processor cores to run all active threads in parallel.
To compensate for network latency, you should use about twice as
many upload threads as indicated by the above table.

Benchmark with NFS backend:
Preparing test data...
Measuring throughput to cache...
Cache throughput with   4 KiB blocks: 12409 KiB/sec
Cache throughput with   8 KiB blocks: 15658 KiB/sec
Cache throughput with  16 KiB blocks: 19004 KiB/sec
Cache throughput with  32 KiB blocks: 20691 KiB/sec
Cache throughput with  64 KiB blocks: 21500 KiB/sec
Cache throughput with 128 KiB blocks: 21915 KiB/sec
Measuring raw backend throughput..
Backend throughput: 11120 KiB/sec
Test file size: 1431.98 MiB
compressing with lzma-6...
lzma compression speed: 1175 KiB/sec per thread (in)
lzma compression speed: 1169 KiB/sec per thread (out)
compressing with bzip2-6...
bzip2 compression speed: 2305 KiB/sec per thread (in)
bzip2 compression speed: 2292 KiB/sec per thread (out)
compressing with zlib-6...
zlib compression speed: 7939 KiB/sec per thread (in)
zlib compression speed: 7902 KiB/sec per thread (out)

With 128 KiB blocks, maximum performance for different compression
algorithms and thread counts is:

Threads:                              1           2           4           8
Max FS throughput (lzma):     1175 KiB/s   2351 KiB/s   4703 KiB/s   9407 KiB/s
..limited by:                       CPU         CPU         CPU         CPU
Max FS throughput (bzip2):    2305 KiB/s   4611 KiB/s   9222 KiB/s  11185 KiB/s
..limited by:                       CPU         CPU         CPU      uplink
Max FS throughput (zlib):     7939 KiB/s  11172 KiB/s  11172 KiB/s  11172 KiB/s
..limited by:                       CPU      uplink      uplink      uplink

All numbers assume that the test file is representative and that
there are enough processor cores to run all active threads in parallel.
To compensate for network latency, you should use about twice as
many upload threads as indicated by the above table.

As far as I can interprete these numbers, the autodetection thread of 4 could be doubled, whereas compression is not of my main concern, I could switch to zlib to tripple the speeds.

Is this interpretation correct?

Thanks!

[Nikolaus Rath]

Super Strobi <strob...@gmail.com> writes:
> Benchmark with *sshfs backend:*
[...]

> Threads: 1 2 4 8
> Max FS throughput (lzma): 1172 KiB/s 2344 KiB/s 4688 KiB/s 8287
> KiB/s
> ..limited by: CPU CPU CPU uplink
> Max FS throughput (bzip2): 2076 KiB/s 4152 KiB/s 8292 KiB/s 8292
> KiB/s
> ..limited by: CPU CPU uplink uplink
> Max FS throughput (zlib): 7921 KiB/s 8283 KiB/s 8283 KiB/s 8283
> KiB/s
> ..limited by: CPU uplink uplink
> uplink

[...]
>
> Benchmark with *NFS backend:*
[...]

> Threads: 1 2 4 8
> Max FS throughput (lzma): 1175 KiB/s 2351 KiB/s 4703 KiB/s 9407
> KiB/s
> ..limited by: CPU CPU CPU CPU
> Max FS throughput (bzip2): 2305 KiB/s 4611 KiB/s 9222 KiB/s 11185
> KiB/s
> ..limited by: CPU CPU CPU uplink
> Max FS throughput (zlib): 7939 KiB/s 11172 KiB/s 11172 KiB/s 11172
> KiB/s
> ..limited by: CPU uplink uplink uplink

[...]

>
> As far as I can interprete these numbers, the autodetection thread of 4
> could be doubled, whereas compression is not of my main concern, I could
> switch to zlib to tripple the speeds.
>
> Is this interpretation correct?

To maximize speed, you should use NFS with bzip2 compression and 8
threads. The second best option is NFS with zlib compression and 2
threads. Increasing the number of threads above that does not give you
increased performance.

HTH,