s3qladm clear crash

[Nicolas Gif]

Hello,

I build a S3QL docker image with v3.5.0 : https://gitlab.com/Salokyn/docker-s3ql/-/merge_requests/55

It builds fine.

On the test job I have, the first step is to run 's3qladm clear' on a dedicated swift container (in case an old FS remains here)

And I get this :

+ s3qladm clear swiftks://auth.cloud.ovh.net/GRA:test-s3ql
+ echo yes
Traceback (most recent call last):
File "/usr/local/bin/s3qladm", line 33, in <module>
sys.exit(load_entry_point('s3ql==3.5.0', 'console_scripts', 's3qladm')())
File "/usr/local/bin/s3qladm", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/local/lib/python3.8/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "/usr/local/lib/python3.8/site-packages/s3ql-3.5.0-py3.8-linux-x86_64.egg/s3ql/adm.py", line 18, in <module>
from .metadata import dump_and_upload_metadata, download_metadata
File "<frozen zipimport>", line 259, in load_module
File "/usr/local/lib/python3.8/site-packages/s3ql-3.5.0-py3.8-linux-x86_64.egg/s3ql/metadata.py", line 13, in <module>
from .deltadump import INTEGER, BLOB, dump_table, load_table

ImportError: cannot import name 'INTEGER' from 's3ql.deltadump' (/root/.cache/Python-Eggs/s3ql-3.5.0-py3.8-linux-x86_64.egg-tmp/s3ql/deltadump.cpython-38-x86_64-linux-gnu.so)

ERROR: Job failed: exit code 1

Same test on an image I build in May with S3QL 3.4.1 : It works !

So I rebuilt that 3.4.1 image now : Get the same error than above.

On the working image I have :

/ # python --version
Python 3.8.2
/ # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.11.6
PRETTY_NAME="Alpine Linux v3.11"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"

/ # pip list -l
Package              Version
-------------------- -----------
apsw                 3.9.2.post1
async-generator      1.10
attrs                19.3.0
cachetools           4.1.0
certifi              2020.4.5.1
cffi                 1.14.0
chardet              3.0.4
cryptography         2.9.2
defusedxml           0.6.0
dugong               3.7.5
google-auth          1.14.2
google-auth-oauthlib 0.4.1
idna                 2.9
oauthlib             3.1.0
outcome              1.0.1
pip                  20.1
pyasn1               0.4.8
pyasn1-modules       0.2.8
pycparser            2.20
pyfuse3              1.3.1
requests             2.23.0
requests-oauthlib    1.3.0
rsa                  4.0
s3ql                 3.4.1
setuptools           46.1.3
six                  1.14.0
sniffio              1.1.0
sortedcontainers     2.1.0
trio                 0.14.0
urllib3              1.25.9
wheel                0.34.2

On the one built now :

/ # python --version
Python 3.8.2
/ # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.11.6
PRETTY_NAME="Alpine Linux v3.11"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
/ # pip list
Package              Version
-------------------- -----------
apsw                 3.9.2.post1
async-generator      1.10
attrs                19.3.0
cachetools           4.1.1
certifi              2020.6.20
cffi                 1.14.1
chardet              3.0.4
cryptography         3.0
defusedxml           0.6.0
dugong               3.7.5
google-auth          1.20.0
google-auth-oauthlib 0.4.1
idna                 2.10
oauthlib             3.1.0
outcome              1.0.1
pip                  20.1
pyasn1               0.4.8
pyasn1-modules       0.2.8
pycparser            2.20
pyfuse3              1.3.1
requests             2.24.0
requests-oauthlib    1.3.0
rsa                  4.6
s3ql                 3.4.1
setuptools           49.2.0
six                  1.15.0
sniffio              1.1.0
sortedcontainers     2.2.2
trio                 0.16.0
urllib3              1.25.10
wheel                0.34.2

[Nicolas Gif]

 Well, after trying many versions of almost everything I found the culprit.

It does not wotk with setuptools >= 47.0.

https://gitlab.com/Salokyn/docker-s3ql/-/commit/8c49c663be39b9ae6ebe206bda0036fa57c1da67

My v3.5.0 docker image works.

[r0ps3c]

For reference, I've since built a docker image with s3ql-3.5.1 via a wheel (i.e. packaging into a wheel built with ```pip wheel ...``` then installing with ```pip install ...```) and the dependency on a specific version (or earlier) of setuptools is no longer required, as everything works with the latest versions (as of this writing, 50.3.0). I suspect this is a more subtle issue around eggs and extensions/dependencies. Given eggs are being deprecated in favour of wheels (e.g. see https://packaging.python.org/discussions/wheel-vs-egg/ and https://setuptools.readthedocs.io/en/latest/setuptools.html#bdist-egg-create-a-python-egg-for-the-project) and might make it easier to move towards distributing via pypi, it may be time to change the build instructions/documentation. Does anyone have any thoughts on that?