Ubuntu 18 install for rookies like myself

[Balázs]

Usually takes me a half a day to figure out all the dependencies post a new-release, so in case any other beginners like myself need a bit of step-by-step on getting s3ql going on a fresh Ubuntu 18 box, maybe my notes below will help:

Ubuntu 2018:

apt-get install psmisc

apt-get install sqlite

apt-get install libsqlite3-dev

apt-get install python3

apt-get install python3-pip

pip3 install setuptools

pip3 install --upgrade setuptools (optional)

pip3 install cryptography

apt-get install python3-defusedxml

pip3 install apsw

apt-get install python3-llfuse

apt-get install python3-dugong

apt-get install python3-pytest

pip3 install -U pytest

apt-get install python3-systemd

apt-get install python3-requests

apt-get install python3-google-auth

pip3 install google-auth-oauthlib

python3 -c 'import setuptools; print(setuptools.__version__)'

python3 -c 'import cryptography; print(cryptography.__version__)'

python3 -c 'import defusedxml; print(defusedxml.__version__)'

python3 -c 'import apsw; print(apsw.__version__)'

python3 -c 'import llfuse; print(llfuse.__version__)'

python3 -c 'import dugong; print(dugong.__version__)'

python3 -c 'import pytest; print(pytest.__version__)'

python3 -c 'import systemd; print(systemd.__version__)'

pytest --version (need above v4 for testing to work)

3.4.0 new stuff:

python3 -m pip install --upgrade trio

python3 -c 'import trio; print(systemd.__version__)'

apt-get install pkg-config

apt-get install meson

apt-get install ninja-build

wget https://github.com/libfuse/libfuse/releases/download/fuse-3.9.1/fuse-3.9.1.tar.xz

tar xvf fuse-3.9.1.tar.xz

cd fuse-3.9.1/

mkdir build; cd build

meson ..

ninja

python3 -m pytest test/

ninja install

pkg-config --modversion fuse3 (just to verify)

pip3 install pyfuse3

ldconfig

Download release from https://github.com/s3ql/s3ql/releases

Bunzip2, tar, cd then build:

python3 setup.py build_ext --inplace

python3 -m pytest tests/

nano s3ql-3.4.0/setup.cfg

edit "allow_hosts = None" to "allow_hosts = *"

python3 setup.py install

Good luck!

[Balázs]

Oh, and seems to be safe to ignore the error during build and operation:

"/usr/local/lib/python3.6/dist-packages/trio/_core/_multierror.py:472: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed.".

Ref: https://www.gitmemory.com/issue/python-trio/trio/1065/496824234

[Nicolas Gif]

I buid it to make a Docker image that way: https://gitlab.com/Salokyn/docker-s3ql/-/blob/master/Dockerfile

Nicolas

--
You received this message because you are subscribed to the Google Groups "s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/7bb7d697-51bd-4085-b745-144009d89982%40googlegroups.com.

[drobertson adhis]

Thanks for this, it will save me heaps of time!

one step I had to include which wasn't in the recipe was..

 python3 -m pip install pytest_trio

.. thought when I run the tests it fails at...

tests/t3_verify.py::test_corrupted_body[True] PASSED                                                       [ 84%]
tests/t3_verify.py::test_corrupted_body[False] PASSED                                                      [ 84%]
tests/t4_adm.py::AdmTests::test_clear FAILED                                                               [ 84%]

==================================================== FAILURES ====================================================
______________________________________________ AdmTests.test_clear _______________________________________________
Traceback (most recent call last):
  File "/root/s3ql-3.4.0/tests/pytest_checklogs.py", line 140, in pytest_runtest_call
    check_output(item)
  File "/root/s3ql-3.4.0/tests/pytest_checklogs.py", line 132, in check_output
    check_test_output(capmethod, item)
  File "/root/s3ql-3.4.0/tests/pytest_checklogs.py", line 106, in check_test_output
    raise AssertionError('Suspicious output to stderr (matched "%s")' % hit.group(0))
AssertionError: Suspicious output to stderr (matched "error")
---------------------------------------------- Captured stdout call ----------------------------------------------
I am about to DELETE ALL DATA in local directory /tmp/s3ql-backend-4rhtgqx3.
This includes not just S3QL file systems but *all* stored objects.
Depending on the storage service, it may be neccessary to run this command
several times to delete all data, and it may take a while until the
removal becomes effective.
Please enter "yes" to continue.
>
---------------------------------------------- Captured stderr call ----------------------------------------------
/usr/local/lib/python3.6/dist-packages/trio/_core/_multierror.py:472: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
  category=RuntimeWarning
WARNING: Maximum object sizes less than 1 MiB will degrade performance.
/usr/local/lib/python3.6/dist-packages/trio/_core/_multierror.py:472: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
  category=RuntimeWarning
================================ 1 failed, 286 passed, 5 skipped in 49.08 seconds ===============================

[Balázs]

Thanks Nicolas, looks pretty cool, but I can honestly say I have no idea how to use that Docker image :-) 

On Saturday, March 28, 2020 at 1:56:10 PM UTC+1, Nicolas Gif wrote:

I buid it to make a Docker image that way: https://gitlab.com/Salokyn/docker-s3ql/-/blob/master/Dockerfile

Nicolas

Le sam. 28 mars 2020 à 13:08, Balázs <czv...@gmail.com> a écrit :

Oh, and seems to be safe to ignore the error during build and operation:

"/usr/local/lib/python3.6/dist-packages/trio/_core/_multierror.py:472: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed.".

Ref: https://www.gitmemory.com/issue/python-trio/trio/1065/496824234

--
You received this message because you are subscribed to the Google Groups "s3ql" group.

To unsubscribe from this group and stop receiving emails from it, send an email to s3...@googlegroups.com.

[Balázs]

Great, thanks, I will add it to my notes. Yes, I get the same error when testing, on the same line.

[Daniel Jagszent]

Hi,

I build Debian packages for S3QL for me and for the 3.4 Version I
migrated to use dh_virtualenv ( https://github.com/spotify/dh-virtualenv
). Before that I used the system packages (and needed to package updated
versions of quite a bunch of python libraries).

I use the following requirements.txt to create the virtualenv
apsw >= 3.8.2, < 4.0
defusedxml
cryptography == 2.8
requests
pyfuse3 >= 1.0, < 2.0
dugong >= 3.4, < 4.0
pytest == 4.6.5
google-auth
google-auth-oauthlib
pytest_trio
trio == 0.12.0
attrs == 19.3.0

With these in a virtualenv, all test pass.

The package is not tested that well but the Ubuntu 16.04 (Xenial)
version is running on some machines for a week or so without problems.

I make no promises regarding availability, but you are welcome to use
the packages, too (I compile them for Xenial, Bionic and Buster – I do
not need other distributions):
wget -qO - http://deb.saasweb.net/s3ql/key.asc | sudo apt-key add -
echo -e "deb http://deb.saasweb.net/s3ql $(lsb_release -cs)
main\ndeb-src http://deb.saasweb.net/s3ql $(lsb_release -cs) main\n" |
sudo tee /etc/apt/sources.list.d/s3ql.list
sudo apt-get update
sudo apt-get install s3ql

... or if you want to compile on your own:
wget -qO - http://deb.saasweb.net/s3ql/key.asc | sudo apt-key add -
echo -e "deb http://deb.saasweb.net/s3ql $(lsb_release -cs)
main\ndeb-src http://deb.saasweb.net/s3ql $(lsb_release -cs) main\n" |
sudo tee /etc/apt/sources.list.d/s3ql.list
sudo apt-get update
apt-get build-dep s3ql
apt-get source s3ql
cd s3ql-3.4.0
dpkg-buildpackage -uc -us
cd ..
ls *.deb

For S3QL to still work in Ubuntu 16.04, the packages not only bundle all
needed Python libraries, but also a Python 3.8 installation, too. It
also bundles the Python systemd bindings so that the "--systemd" mount
option works.
Have a look at
http://deb.saasweb.net/s3ql/pool/main/s/s3ql/s3ql_3.4.0-2~xenial.debian.tar.xz
to see all the bells and whistles (If you know debian packaging, that is).


The packages include two very small custom patches:
One to fix a problem with the OVH load balancer (big bulk deletes time
out since the load balancer wants to buffer the whole response)
Another patch introduces a new option --fssize that merely sets the
value, that S3QL is reporting as its file system size. That size does
not get enforced so its quite useless but some software (e.g. Nextcloud)
will indeed heed this constraint. (I sometimes use S3QL as the
filesystem where a Nextcloud instance is storing its data and this
option helps so that Nextcloud is using the space more efficiently)