S3QL 3.3 performance

[Grunthos]

Obviously using an old version (Ubuntu default)...but I see an odd/interesting performance feature.

I am trying to restore 50GB from google cloud. Initially it ran quite fast (approx 1GB/hour; not great, but not killing my link). After 12 hours it was down to 1/10th of that.

Investigations of GC, my link usage, CPU and IO rates did not really lead me to any obvious conclusion. The download machine was doing 200kb/sec IO, 25% iowait, otherwise idle, internet was about 250kb/sec. It was creating about 5 small files per second.

So...I killed the rsync and replaced it with 4 rsyncs doing different sub-directories. The result was 4 times the throughput: internet link is now at 1Mb/sec, the iowait time is 49% and it is creating small 20 files per second.

Is there an improvement in later versions that would improve the single-threaded behavior, and/or is there a parameter I can use to get better performance on the single-threaded version (ie. only one rsync)?

I have not played with the 'upload-threads' setting since this seems to be a download problem.

Any suggestions would be appreciated!

[Grunthos]

OK..I just found this http://www.rath.org/s3ql-docs/tips.html which suggests exactly the solution I have used.

I wonder if there *might* be a better option for full file-system restore, one of:

• copy the entire collection S3QL data from the remote to local first, then do a restore-from-local
• add a special s3ql command to do a 'tree copy' -- it would know exactly which blocks it needed and download them en-masse while restoring files (and would need a lot of cache, possibly even a temporary cache drive)
• a limited version of the above option to pre-fill the cache with all remote data blocks needed for a particular part of the tree
  

[Nikolaus Rath]

On Apr 13 2021, Grunthos <philip.jo...@gmail.com> wrote:
> OK..I just found this http://www.rath.org/s3ql-docs/tips.html which
> suggests exactly the solution I have used.
>
> I wonder if there *might* be a better option for full file-system restore,
> one of:
>
>

> - copy the entire collection S3QL data from the remote to local first,
> then do a restore-from-local
> - add a special s3ql command to do a 'tree copy' -- it would know

> exactly which blocks it needed and download them en-masse while restoring
> files (and would need a lot of cache, possibly even a temporary cache drive)

> - a limited version of the above option to pre-fill the cache with all

> remote data blocks needed for a particular part of the tree

Yes, all of these would be possible and probably be faster. I think
option (2) would me the best one.

Pull requests are welcome :-).


Best,
-Nikolaus

--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Grunthos]

On Wednesday, April 14, 2021 at 5:36:30 PM UTC+10 Niko...@rath.org wrote:

Yes, all of these would be possible and probably be faster. I think
option (2) would me the best one.

Pull requests are welcome :-).

I had a funny feeling that might be the answer...and in terms of utility and design, ISTM that " add a special s3ql command to do a 'tree copy' -- it would know exactly which blocks it needed and download them en-masse while restoring files (and would need a lot of cache, possibly even a temporary cache drive)" is a good plan.

I am not at all sure I am up for the (probable) deep-dive required, but if I were to look at this could you give some suggested starting points? My very naieve approach (not knowing the internals at all) would be to build a list of all required blocks, do some kind of topo sort, then start multiple download threads. As each block was downloaded, determine if a new file can be copied yet, and if so, copy it, then release and blocks that are no longer needed.

...like I said, naieve, and hightly dependant on internals...and maybe should use some kind of private mount to avoid horror.

[Nikolaus Rath]

I think there's a simpler solution.

1. Add a new special xattr to trigger the functionality (look at
s3qlcp.py and copy_tree() in fs.py)

2. Have fs.py write directly to the destination directory (which should
be outside the S3QL mountpoint)

3. Start a number of async workers (no need for threads) that, in a
loop, download blocks and write them to a given offset in a given fh.

4. Have the main thread recursively traverse the source and issue "copy"
requests to the workers (through a queue)

5. Wait for all workers to finish.

6. Profit.


I wouldn't even bother putting blocks in the cache - just download and
write to the destination on the fly. It may be worth checking if a block
is *already* in the cache and, if so, skip download though.


With this implementation, blocks referenced by multiple files will be
downloaded multiple times. I think this can be improved upon once the
minimum functionality is working.

[Grunthos]

OK...will have a look into this. I'd definitely like a faster restore operation.

In terms of my short term problem, the estimated time for restore using multiple rsyncs is now about 60 hours. The estimated time to just copy the data from the bucket is 14 hours...can I just setup a 'local' rsync using the S3QL encryption key from GC and restore? Or is that too optimistic? 

[Daniel Jagszent]

Grunthos schrieb am 14.04.21 um 16:09:
> [...] can I just setup a 'local' rsync using the S3QL encryption key

> from GC and restore? Or is that too optimistic? 

You need to use https://www.rath.org/s3ql-docs/contrib.html#clone-fs-py
just rclone-ing the GC bucket to local and then mounting the FS as local
S3QL file system does not work since different file systems use
different metadata and different ways to store the metadata with the
objects.

Decrypting the objects yourself with the master key is an involved
process (see https://www.rath.org/s3ql-docs/impl_details.html#encryption
for how S3QL encrypts the data, the nonce for the session key is saved
as metadata to each object AFAIK).

[Grunthos]

That's unfortunate: I assume close-fs.py will suffer the same performance issues.

Are you sure that the raw S3QL data files won't "just work" assuming I provide the mater key etc? The file names and arrangement look similar to a local instance...

[Daniel Jagszent]

Grunthos schrieb am 14.04.21 um 16:47:

That's unfortunate: I assume close-fs.py will suffer the same performance issues.

not necessarily since it does not decrypts/re-encrypts the data

Are you sure that the raw S3QL data files won't "just work" assuming I provide the mater key etc? The file names and arrangement look similar to a local instance...

Yes I am sure. The local backend saves the metadata at the beginning of the file (see https://github.com/s3ql/s3ql/blob/master/src/s3ql/backends/local.py#L245) and all remote backends (including GC) can save the metadata outside the file stream.

[Nikolaus Rath]

On Apr 14 2021, Grunthos <philip.jo...@gmail.com> wrote:

> That's unfortunate: I assume close-fs.py will suffer the same performance
> issues.

No, it should be as fast as rclone.

clone.fs uses multiple threads and downloads each block only
once. You can try to adjust the number of threads, but apart from that I
think you fundamentally cannot do any better than this (neither with
other tools nor with major code changes to S3QL).

[Grunthos]

On Thursday, April 15, 2021 at 5:28:56 AM UTC+10 Niko...@rath.org wrote:

You can try to adjust the number of threads, but apart from that I
think you fundamentally cannot do any better than this (neither with
other tools nor with major code changes to S3QL).

Yep! Seem like you are correct: `clone-fs.py` (16 threads) does a faster download than `gsutil -m rsync...`. Still depressingly slow, though -- which I assume is a limit put in place by my ISP and/or Google. CPU cruising at 30% and link runs at same speed with 8 or 16 threads. FWIW, it does indeed seem that the current fastest option might in fact be to clone the fs then restore locally. The mount/rsync option is much slower. But I will know more one I have actually restored from the local copy.

[Grunthos]

OK...possibly made a silly mistake...I have the bucket mounted (for a copy) elsewhere, while running clone-fs. While no writes are occurring, the metadata does get uploaded periodically, I believe.

Am I therefor likely now to have a corrupt clone? It doesn't look like clone-fs has 'rsync-like' features, so I can't just unmount and re-do to get updates AFAICT. Any thoughts?

[Nikolaus Rath]

On Apr 15 2021, Grunthos <philip.jo...@gmail.com> wrote:
> OK...possibly made a silly mistake...I have the bucket mounted (for a copy)
> elsewhere, while running clone-fs. While no writes are occurring, the
> metadata does get uploaded periodically, I believe.
>
> Am I therefor likely now to have a corrupt clone? It doesn't look like
> clone-fs has 'rsync-like' features, so I can't just unmount and re-do to
> get updates AFAICT. Any thoughts?

If there were no changes, then the metadata has not changed and thus
will not be uploaded.

If there were minor changes (e.g. file atime but no file contents), then
running fsck.s3ql on the clone should clear the dirty flag without
finding any problems. After that, you're good to go.

[Grunthos]

Cool...that all worked...and it has led to another interesting bottleneck. There are a LOT of small files. What I now have is:

Disk A: root

Disk B: /home

Disk C: (temporary, S3QL bucket)

I am copying from C to B....and disk A is being overloaded! I'm *guessing* it's temp files or something used by the s3ql mount process...still investigating...

[Grunthos]

Ahhhhh....it's the cache.

[Grunthos]

It IS frighteningly slow though: 300MB/hour disk to disk! (source is local s3ql dest is btrfs).