Please test S3QL from master

[Nikolaus Rath]

Hi,

I have just merged a large patch into S3QL. It removes an internal
abstraction layer that was never used. It should make the code more
maintainable, performance better, and reduce database size.

Unfortunately, it also makes backwards incompatible changes to the
filesystem structure.

It would be great if some more people could grab the current version
from Git master and give it a spin before this makes it into the next
stable release.

This will most likely be my last contribution to S3QL for a long time
since other things have taken priority in my life. I'll continue to
apply pull requests and make releases as long and as regularly as I can,
but if someone wants to take a more active role in the project then I
would be delighted to step back even more.


Best,
-Nikolaus

--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Henry Wertz]

Just wanted to report on mine, so far so good!  I have (using local backend on ext4 filesystem) a 4TB drive, 8TB drive, and a 1TB drive (in a portable system), all running s3ql with a variety of workloads (some movie storage -- I know s3ql won't do anything to shrink or dedup that but OK... rsync backups, VirtualBox activities, both .ova files and running VMs out of there, some source code, some games as well.)   Nothing to report!   Unmounted filesystem(s), updated s3ql.  The s3qladm upgrade process was quick and painless, on the 2GB database it took a minute or two, on the others it was nearly instantaneous.  It shrunk the databases about 10-15%.  I decided to fsck.s3ql --force the filesystems too, no problems. The 8TB has a lot of duplication due to rsync backups on it, about 8.8TB data on the 8TB drive using 5.23TB disk space.  This one the DB is now 1.9GB and shrunk about by 200MB.  4TB, it's like 90MB now, it shrunk about 9MB.  It definitely mounts, unmounts, and fscks faster (since there's 1 fewer tables for it to read in, write back, or check.)  I didn't think of doing any benchmarks but "seat of the pants" it does seem a bit faster in use too; not a surprise, when you're pushing the IOPS on there one potential "choke point" are s3qlite synchronous writes, and removing one table means that much less stuff for s3qlite to have to write out.  

[Nikolaus Rath]

Hi Henry,

Glad to hear that, thanks for reporting back!

Best

-Nikolaus

--

GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

On Sun, 24 Apr 2022, at 06:33, Henry Wertz wrote:

Just wanted to report on mine, so far so good!  I have (using local backend on ext4 filesystem) a 4TB drive, 8TB drive, and a 1TB drive (in a portable system), all running s3ql with a variety of workloads (some movie storage -- I know s3ql won't do anything to shrink or dedup that but OK... rsync backups, VirtualBox activities, both .ova files and running VMs out of there, some source code, some games as well.)   Nothing to report!   Unmounted filesystem(s), updated s3ql.  The s3qladm upgrade process was quick and painless, on the 2GB database it took a minute or two, on the others it was nearly instantaneous.  It shrunk the databases about 10-15%.  I decided to fsck.s3ql --force the filesystems too, no problems. The 8TB has a lot of duplication due to rsync backups on it, about 8.8TB data on the 8TB drive using 5.23TB disk space.  This one the DB is now 1.9GB and shrunk about by 200MB.  4TB, it's like 90MB now, it shrunk about 9MB.  It definitely mounts, unmounts, and fscks faster (since there's 1 fewer tables for it to read in, write back, or check.)  I didn't think of doing any benchmarks but "seat of the pants" it does seem a bit faster in use too; not a surprise, when you're pushing the IOPS on there one potential "choke point" are s3qlite synchronous writes, and removing one table means that much less stuff for s3qlite to have to write out.  

--

You received this message because you are subscribed to the Google Groups "s3ql" group.

To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/CAOuPqTEjw%3DhJ-k_MXUyPuruCo-ts8ssY7WiyEXyF0JFk5ESk2w%40mail.gmail.com.

[Daniel Jagszent]

Hello Nikolaus,

[...] It would be great if some more people could grab the current version
from Git master and give it a spin before this makes it into the next
stable release.[...]

I finally got around to test out the master branch. It works as expected here on a Debian buster VM with OVH Swift backend.
Tested fsck.s3ql, mount.s3ql, s3qlstat, s3qlctrl, s3ql_verify --data