S3QL 5.2.0 - s3qlcp taking a long time

[Shannon Dealy]

Hi all,

I have been using S3QL 3.7.3 (and earlier versions) with Amazon S3 for many
years. Due to a variety of circumstances I have just setup version 5.2.0 using
the s3c4 backend with Backblaze. Part of my standard incremental backup script
is to use s3qlcp to make a copy of the latest backup into a timestamped
directory. The s3qlcp command has been running for around 2 hours now with
a subset of the same data set in my old backups and it previously never took
even a quarter of this time to do the copy.

Is anyone aware of problems like this? As I understand it, the s3qlcp copy
is strictly a database operation so the backend shouldn't even be relevant to
performance of this command other than the database upload after the copy is
complete.

Relevant data:
S3QL 5.2.0
s3c4 backend
Database size on 5.2.0: 263M

My old system
S3QL 3.7.3
s3 backend
Database size on 3.7.3: 2.9G (lots of s3qlcp copies in this database)

NOTES:
I didn't try the backblaze backend (b2) as it gave an error when
running the tests (test_delete_nonexisting failed) where the s3c4 backend
passed using the same Backblaze bucket.

I'm trying to avoid debugging anything right now as I just did a massive
system upgrade and am trying to fix everything that it broke :-(

Best regards,

Shannon C. Dealy | DeaTech Research Inc.
de...@deatech.com | Biotechnology Development Services
Telephone USA: +1 541-929-4089 | USA and the Netherlands
Netherlands: +31 85 208 5570 | www.deatech.com

[Nikolaus Rath]

"'Shannon Dealy' via s3ql" <s3...@googlegroups.com> writes:
> I have been using S3QL 3.7.3 (and earlier versions) with Amazon S3 for many
> years. Due to a variety of circumstances I have just setup version 5.2.0 using
> the s3c4 backend with Backblaze. Part of my standard incremental backup script
> is to use s3qlcp to make a copy of the latest backup into a timestamped
> directory. The s3qlcp command has been running for around 2 hours now with a
> subset of the same data set in my old backups and it previously never took even
> a quarter of this time to do the copy.
>
> Is anyone aware of problems like this?

This is the first report that I've seen.

> As I understand it, the s3qlcp copy is
> strictly a database operation so the backend shouldn't even be relevant to
> performance of this command other than the database upload after the copy is
> complete.

That's right.


Best,
-Nikolaus