Upcoming new release needs testers

[Nikolaus Rath]

Hi all,

In the last few months, I've reworked major internals of S3QL. Before I make a new release, I'd appreciate if some more users could give the current Git version a test and report any issues. This is particularly important for users of backends other than S3 and Google Storage (since I only run integration tests for those).

For those interested, some details on the changes:

In terms of user-visible changes, it is now possible to separately specify the number of threads to use for compression/encryption, and the number of connections to use for concurrent uploads (previously these two numbers had to be identical). 

Behind the scenes, this was only possible because the previously awkwardly-entangled mess of multithreaded and async code has been fully cleaned up. Until now, S3QL had an async file-system operations layer that called a partially async, partially threaded block layer, which relied on a synchronous backend layer that awkwardly called an HTTP layer intended for async users. The reasons for this are mostly historic (S3QL was started before Python had proper async support). Now, however, all of S3QL is async all the way to the HTTP layer, and only compression/encryption is cleanly moved to separate worker threads.

Apart from the benefits of a cleaner codebase, this will make it much easier to implement some long-planned features, most importantly metadata upload/download utilizing multiple threads and multiple connections.

So, please give it a try and repart back any issues!

Best,

-Nikolaus

[Daniel Jagszent]

Hi Nikolaus,

[...] Before I make a new release, I'd appreciate if some more users could give the current Git version a test and report any issues. This is particularly important for users of backends other than S3 and Google Storage (since I only run integration tests for those). [...]

I just ran the test suite on main with OpenStack Swift (Keystone v3, OVH) and all tests passed.

I needed to alter the test suite to pick up the backend options from the authinfo file (https://github.com/s3ql/s3ql/pull/410; because Swiftks always needs backend options), but apart from that, no changes were needed.

I did not mount/upgrade any existing filesystems with the new code yet. So no experience with long-running filesystem mounts yet.