S3QL performance and sqlite fragmentation : some comments

[drobertson adhis]

Hi,

I'm using with success S3QL across a variety of backends (s3, local NFS, local RAID6, OVH swift).

Some of these are storing fairly large amounts of data, managed with scripts to cull older backups after X days and months (using a variation of the ideas in the S3QL contrib to expire older copies) to stop the database getting too large.

I've noticed these gradually get slower over time (like even a "ls" or "cd" takes minutes or even longer), I am assuming the culprit is sqlite fragmentation.

umounting the filesystems and doing a fsck.s3ql, which includes a s3qlite vacuum operation substantially speeds things up for a while, though the performance gradually drops (I'm guessing as the fragmentation comes back).

I am storing my s3ql cache files on spinning disk. I guess one obvious thing here might be to try SSD to get a speedup?

I'm just about to script some timing the operation of the "s3qlstat $FILESYSTEM" command to determine when s3qlite fragmentation is requiring an umount/fsck to maintain performance. Although due to the fragmentation these fsck operations themselves can take a lot of time.

Anyway I'm just mentioning this as something I've observed running S3QL for some time which isn't apparent at the start, but can be managed.

[Cliff Stanford]

On 21/02/2020 02:36, drobertson adhis wrote:

> I am storing my s3ql cache files on spinning disk. I guess one obvious
> thing here might be to try SSD to get a speedup?

I had a similar issue. I moved the .s3ql folder onto an ssd and saw a
massive decrease in time taken for things like s3qlstat and df.

I think it is simply the speed of the sqlite database access that made
the difference.

Cliff.

--
Cliff Stanford
London: +44 20 0222 1666 Swansea: +44 1792 469666
Spain: +34 603 777 666 Estonia: +372 5308 9666
UK Mobile: +44 7973 616 666

[drobertson adhis]

On Friday, February 21, 2020 at 9:03:13 PM UTC+11, Cliff Stanford wrote:

I had a similar issue.  I moved the .s3ql folder onto an ssd and saw a
massive decrease in time taken for things like s3qlstat and df.

thanks for the confirmation - an SSD has just been delivered.. once I get some downtime booked I'll install it and see how it helps.