Upcoming new release needs testers

[Nikolaus Rath]

Hi all,

In the last few months, I've reworked major internals of S3QL. Before I make a new release, I'd appreciate if some more users could give the current Git version a test and report any issues. This is particularly important for users of backends other than S3 and Google Storage (since I only run integration tests for those).

For those interested, some details on the changes:

In terms of user-visible changes, it is now possible to separately specify the number of threads to use for compression/encryption, and the number of connections to use for concurrent uploads (previously these two numbers had to be identical). 

Behind the scenes, this was only possible because the previously awkwardly-entangled mess of multithreaded and async code has been fully cleaned up. Until now, S3QL had an async file-system operations layer that called a partially async, partially threaded block layer, which relied on a synchronous backend layer that awkwardly called an HTTP layer intended for async users. The reasons for this are mostly historic (S3QL was started before Python had proper async support). Now, however, all of S3QL is async all the way to the HTTP layer, and only compression/encryption is cleanly moved to separate worker threads.

Apart from the benefits of a cleaner codebase, this will make it much easier to implement some long-planned features, most importantly metadata upload/download utilizing multiple threads and multiple connections.

So, please give it a try and repart back any issues!

Best,

-Nikolaus

[Daniel Jagszent]

Hi Nikolaus,

[...] Before I make a new release, I'd appreciate if some more users could give the current Git version a test and report any issues. This is particularly important for users of backends other than S3 and Google Storage (since I only run integration tests for those). [...]

I just ran the test suite on main with OpenStack Swift (Keystone v3, OVH) and all tests passed.

I needed to alter the test suite to pick up the backend options from the authinfo file (https://github.com/s3ql/s3ql/pull/410; because Swiftks always needs backend options), but apart from that, no changes were needed.

I did not mount/upgrade any existing filesystems with the new code yet. So no experience with long-running filesystem mounts yet.

[Paul Harris]

Hi Nikolaus, 

I can offer you a b2 bucket to use for tests. 

Cheers, 

Paul 

[Nikolaus Rath]

Hi Paul,

Thanks for the offer! If you could do the test directly and report success/errors, that would be easiest for me. If that's too much trouble, I'll happily accept the offer though. Could you send me the credentials per personal email?

Best,

-Nikolaus

--

You received this message because you are subscribed to the Google Groups "s3ql" group.

To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+uns...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/s3ql/4f97d514-8e85-4d12-bcad-675811c1dc88n%40googlegroups.com.

[Nikolaus Rath]

[ Moving discussion to the mailing list ]

On Sat, 11 Apr 2026, at 15:42, Paul Harris wrote:

I finally had some time to look at this.

I had this problem when following the developer setup, the docs could use a tweak:

WARNING: Requested pyfuse3<4.0,>=3.2.0 from https://files.pythonhosted.org/packages/98/c8/8d4b00354b0aae1660310c6853cad7228af7efc344b3ff7d8cc4a894343d/pyfuse3-3.4.2.tar.gz (from s3ql==6.0.0), but installing version 0.0.0

Discarding https://files.pythonhosted.org/packages/98/c8/8d4b00354b0aae1660310c6853cad7228af7efc344b3ff7d8cc4a894343d/pyfuse3-3.4.2.tar.gz (from https://pypi.org/simple/pyfuse3/) (requires-python:>=3.10): Requested pyfuse3<4.0,>=3.2.0 from https://files.pythonhosted.org/packages/98/c8/8d4b00354b0aae1660310c6853cad7228af7efc344b3ff7d8cc4a894343d/pyfuse3-3.4.2.tar.gz (from s3ql==6.0.0) has inconsistent version: expected '3.4.2', but metadata has '0.0.0'

And it failed,

Are you perhaps trying to install this on a Python version older than 3.10? S3QL requires Python 3.10 or newer, older Python version are also no longer supported upstream (https://devguide.python.org/versions/).

Best,

-Nikolaus

[Paul Harris]

$ python --version
Python 3.12.3

[Nikolaus Rath]

Hello,

Hm, not sure what's wrong then. The instructions provide the same warnings for me to, but installation succeeds:

> rm -rf .venv

> python3 -m venv .venv

> . .venv/bin/activate.fish

> python3 --version

Python 3.14.3

> pip install --upgrade pip 

Requirement already satisfied: pip in ./.venv/lib64/python3.14/site-packages (25.1.1)

[...]

Successfully installed pip-26.0.1

> pip install --group dev

Collecting pytest

[...]

Installing collected packages: sortedcontainers, urllib3, typing-extensions, types-docutils, types-defusedxml, ty, sphinxcontrib-serializinghtml, sphinxcontrib-qthelp, sphinxcontrib-jsmath, sphinxcontrib-htmlhelp, sphinxcontrib-devhelp, sphinxcontrib-applehelp, snowballstemmer, sniffio, setuptools, roman-numerals, pygments, pluggy, pathspec, packaging, nodeenv, mypy_extensions, MarkupSafe, librt, iniconfig, imagesize, idna, docutils, cython, charset_normalizer, certifi, babel, attrs, alabaster, types-pygments, requests, pytest, pyright, outcome, mypy, Jinja2, trio, sphinx, pytest-trio

Successfully installed Jinja2-3.1.6 MarkupSafe-3.0.3 alabaster-1.0.0 attrs-26.1.0 babel-2.18.0 certifi-2026.2.25 charset_normalizer-3.4.7 cython-3.2.4 docutils-0.22.4 idna-3.11 imagesize-2.0.0 iniconfig-2.3.0 librt-0.9.0 mypy-1.20.1 mypy_extensions-1.1.0 nodeenv-1.10.0 outcome-1.3.0.post0 packaging-26.1 pathspec-1.0.4 pluggy-1.6.0 pygments-2.20.0 pyright-1.1.408 pytest-9.0.3 pytest-trio-0.8.0 requests-2.33.1 roman-numerals-4.1.0 setuptools-82.0.1 sniffio-1.3.1 snowballstemmer-3.0.1 sortedcontainers-2.4.0 sphinx-9.1.0 sphinxcontrib-applehelp-2.0.0 sphinxcontrib-devhelp-2.0.0 sphinxcontrib-htmlhelp-2.1.0 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-2.0.0 sphinxcontrib-serializinghtml-2.0.0 trio-0.33.0 ty-0.0.31 types-defusedxml-0.7.0.20260408 types-docutils-0.22.3.20260408 types-pygments-2.20.0.20260408 typing-extensions-4.15.0 urllib3-2.6.3

> pip install --no-build-isolation --editable .

Obtaining file:///home/nikratio/in-progress/s3ql

[...]

Successfully built s3ql pyfuse3

Installing collected packages: pycparser, pyasn1, oauthlib, defusedxml, apsw, anyio, requests-oauthlib, pyasn1-modules, cffi, pyfuse3, cryptography, google-auth, google-auth-oauthlib, s3ql

Successfully installed anyio-4.13.0 apsw-3.53.0.0 cffi-2.0.0 cryptography-46.0.7 defusedxml-0.7.1 google-auth-2.49.2 google-auth-oauthlib-1.3.1 oauthlib-3.3.1 pyasn1-0.6.3 pyasn1-modules-0.4.2 pycparser-3.0 pyfuse3-3.4.0 requests-oauthlib-2.0.0 s3ql-6.0.0

Same result with just `uv sync`.

Best,

-Nikolaus

--

You received this message because you are subscribed to the Google Groups "s3ql" group.

To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+uns...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/s3ql/CAMSUDdaSCK43Do-wCkHU-LnYOsFt5xDWZsbQEWMZcCe3b%3D3Q6w%40mail.gmail.com.

[Paul Harris]

Did you run the tests?  The error comes from the test.

You received this message because you are subscribed to a topic in the Google Groups "s3ql" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/s3ql/pwMIMlLtM-8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to s3ql+uns...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/s3ql/08e8eefb-1708-46fd-b691-7ec3025ce425%40app.fastmail.com.

[Nikolaus Rath]

Hi Paul,

Ah, apologies, then I misunderstoodd. Since you wrote that the developer setup instructions could use a tweak, I thought there was something wrong with them.

Best,

-Nikolaus

To view this discussion visit https://groups.google.com/d/msgid/s3ql/CAMSUDdYuJ6u0f5CT%3Dnqtx3rDkPgHhEut72_J6DctyTcZBmA8Mw%40mail.gmail.com.