Error when compiling s3ql 3.5.0

[alex...@silvagoncalves.com]

Hello,

I'm creating a docker container to automate the use of s3ql, and this is the script that performs the installation (attachment). I think I'm respecting all the dependencies:

Nevertheless, I'm getting these errors:

MANIFEST.in exists, running in developer mode
Traceback (most recent call last):
File "setup.py", line 36, in <module>
import s3ql
File "/build_dir/s3ql/src/s3ql/__init__.py", line 39, in <module>
from pyfuse3 import ROOT_INODE
ImportError: libfuse3.so.3: cannot open shared object file: No such file or directory
MANIFEST.in exists, running in developer mode
Traceback (most recent call last):
File "setup.py", line 36, in <module>
import s3ql
File "/build_dir/s3ql/src/s3ql/__init__.py", line 39, in <module>
from pyfuse3 import ROOT_INODE
ImportError: libfuse3.so.3: cannot open shared object file: No such file or directory
python3: can't open file 'runtests.py': [Errno 2] No such file or directory
install s3ql
MANIFEST.in exists, running in developer mode
Traceback (most recent call last):
File "setup.py", line 36, in <module>
import s3ql
File "/build_dir/s3ql/src/s3ql/__init__.py", line 39, in <module>
from pyfuse3 import ROOT_INODE
ImportError: libfuse3.so.3: cannot open shared object file: No such file or directory

Do you have a suggestion to solve this?

Thanks.

[Daniel Jagszent]

Hello,

[...]

ImportError: libfuse3.so.3: cannot open shared object file: No such file or directory

[...]

Looks like your libfuse installation was not successful. Maybe follow https://github.com/libfuse/libfuse#installation more closely (it says "meson .." not "meson")

Also insert "set -e" as second line in your install.sh script (see https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html#The-Set-Builtin for what it does)

[Alexandre Gonçalves]

--
You received this message because you are subscribed to the Google Groups "s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/9d672c2b-aab0-291f-21d3-45bb9b2d9871%40jagszent.de.

Thanks for spotting that mistake... I overlooked it!

The result of the compilation, tests and install of libfuse is this this:

Found ninja-1.8.2 at /usr/bin/ninja
[1/66] Compiling C object 'lib/fuse3@sha/fuse_loop.c.o'.
[2/66] Compiling C object 'lib/fuse3@sha/fuse_signals.c.o'.
[3/66] Compiling C object 'lib/fuse3@sha/fuse_loop_mt.c.o'.
[4/66] Compiling C object 'lib/fuse3@sha/buffer.c.o'.
[5/66] Compiling C object 'lib/fuse3@sha/helper.c.o'.
[6/66] Compiling C object 'lib/fuse3@sha/fuse_opt.c.o'.
[7/66] Compiling C object 'lib/fuse3@sha/cuse_lowlevel.c.o'.
[8/66] Compiling C object 'lib/fuse3@sha/fuse_log.c.o'.
[9/66] Compiling C object 'lib/fuse3@sha/mount_util.c.o'.
[10/66] Compiling C object 'lib/fuse3@sha/mount.c.o'.
[11/66] Compiling C object 'test/test_write_cache@exe/test_write_cache.c.o'.
[12/66] Compiling C object 'lib/fuse3@sha/modules_subdir.c.o'.
[13/66] Generating test_scripts with a custom command.
[14/66] Compiling C object 'lib/fuse3@sha/modules_iconv.c.o'.
[15/66] Compiling C object 'test/wrong_command@exe/wrong_command.c.o'.
[16/66] Linking target test/wrong_command.
[17/66] Compiling C object 'test/test_setattr@exe/test_setattr.c.o'.
[18/66] Compiling C object 'lib/fuse3@sha/fuse_lowlevel.c.o'.
[19/66] Compiling C object 'util/fusermount3@exe/.._lib_mount_util.c.o'.
[20/66] Compiling C object 'util/mount.fuse3@exe/mount.fuse.c.o'.
[21/66] Compiling C object 'example/passthrough@exe/passthrough.c.o'.
[22/66] Compiling C object 'example/hello@exe/hello.c.o'.
[23/66] Compiling C object 'test/test_syscalls@exe/test_syscalls.c.o'.
[24/66] Compiling C object 'util/fusermount3@exe/fusermount.c.o'.
[25/66] Compiling C object 'example/hello_ll@exe/hello_ll.c.o'.
[26/66] Linking target util/fusermount3.
[27/66] Linking target test/test_syscalls.
[28/66] Compiling C object 'example/printcap@exe/printcap.c.o'.
[29/66] Compiling C object 'example/ioctl_client@exe/ioctl_client.c.o'.
[30/66] Compiling C object 'example/poll_client@exe/poll_client.c.o'.
[31/66] Compiling C object 'example/passthrough_fh@exe/passthrough_fh.c.o'.
[32/66] Compiling C object 'example/ioctl@exe/ioctl.c.o'.
[33/66] Compiling C object 'example/cuse_client@exe/cuse_client.c.o'.
[34/66] Compiling C object 'example/cuse@exe/cuse.c.o'.
[35/66] Compiling C object 'example/null@exe/null.c.o'.
[36/66] Compiling C object 'example/notify_inval_inode@exe/notify_inval_inode.c.o'.
[37/66] Compiling C object 'example/invalidate_path@exe/invalidate_path.c.o'.
[38/66] Compiling C object 'example/notify_inval_entry@exe/notify_inval_entry.c.o'.
[39/66] Compiling C object 'lib/fuse3@sha/fuse.c.o'.
[40/66] Compiling C object 'example/notify_store_retrieve@exe/notify_store_retrieve.c.o'.
[41/66] Compiling C object 'example/poll@exe/poll.c.o'.
[42/66] Linking target lib/libfuse3.so.3.9.2.
[43/66] Compiling C object 'example/passthrough_ll@exe/passthrough_ll.c.o'.
[44/66] Generating symbol file 'lib/fuse3@sha/lib/libfuse3.so.3.9.2.symbols'.
[45/66] Linking target test/test_setattr.
[46/66] Linking target test/test_write_cache.
[47/66] Linking target example/hello.
[48/66] Linking target example/hello_ll.
[49/66] Linking target example/passthrough.
[50/66] Linking target util/mount.fuse3.
[51/66] Linking target example/passthrough_fh.
[52/66] Linking target example/printcap.
[53/66] Linking target example/poll_client.
[54/66] Linking target example/ioctl_client.
[55/66] Linking target example/ioctl.
[56/66] Linking target example/cuse.
[57/66] Linking target example/null.
[58/66] Linking target example/cuse_client.
[59/66] Linking target example/invalidate_path.
[60/66] Linking target example/notify_inval_inode.
[61/66] Linking target example/passthrough_ll.
[62/66] Linking target example/notify_store_retrieve.
[63/66] Linking target example/poll.
[64/66] Linking target example/notify_inval_entry.
[65/66] Compiling C++ object 'example/passthrough_hp@exe/passthrough_hp.cc.o'.
[66/66] Linking target example/passthrough_hp.
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /build_dir/libfuse/build/test, inifile: pytest.ini
collecting ... collected 41 items

test/test_ctests.py::test_write_cache[False] SKIPPED                     [  2%]
test/test_ctests.py::test_write_cache[True] SKIPPED                      [  4%]
test/test_ctests.py::test_notify1[True-notify_inval_inode] SKIPPED       [  7%]
test/test_ctests.py::test_notify1[True-invalidate_path] SKIPPED          [  9%]
test/test_ctests.py::test_notify1[False-notify_inval_inode] SKIPPED      [ 12%]
test/test_ctests.py::test_notify1[False-invalidate_path] SKIPPED         [ 14%]
test/test_ctests.py::test_notify_file_size[True] SKIPPED                 [ 17%]
test/test_ctests.py::test_notify_file_size[False] SKIPPED                [ 19%]
test/test_examples.py::test_hello[hello-options0-invoke_directly] SKIPPED [ 21%]
test/test_examples.py::test_hello[hello-options0-invoke_mount_fuse] SKIPPED [ 24%]
test/test_examples.py::test_hello[hello-options0-invoke_mount_fuse_drop_privileges] SKIPPED [ 26%]
test/test_examples.py::test_hello[hello-options1-invoke_directly] SKIPPED [ 29%]
test/test_examples.py::test_hello[hello-options1-invoke_mount_fuse] SKIPPED [ 31%]
test/test_examples.py::test_hello[hello-options1-invoke_mount_fuse_drop_privileges] SKIPPED [ 34%]
test/test_examples.py::test_hello[hello_ll-options0-invoke_directly] SKIPPED [ 36%]
test/test_examples.py::test_hello[hello_ll-options0-invoke_mount_fuse] SKIPPED [ 39%]
test/test_examples.py::test_hello[hello_ll-options0-invoke_mount_fuse_drop_privileges] SKIPPED [ 41%]
test/test_examples.py::test_hello[hello_ll-options1-invoke_directly] SKIPPED [ 43%]
test/test_examples.py::test_hello[hello_ll-options1-invoke_mount_fuse] SKIPPED [ 46%]
test/test_examples.py::test_hello[hello_ll-options1-invoke_mount_fuse_drop_privileges] SKIPPED [ 48%]
test/test_examples.py::test_passthrough[False-passthrough-False] SKIPPED [ 51%]
test/test_examples.py::test_passthrough[False-passthrough-True] SKIPPED  [ 53%]
test/test_examples.py::test_passthrough[False-passthrough_fh-False] SKIPPED [ 56%]
test/test_examples.py::test_passthrough[False-passthrough_fh-True] SKIPPED [ 58%]
test/test_examples.py::test_passthrough[False-passthrough_ll-False] SKIPPED [ 60%]
test/test_examples.py::test_passthrough[False-passthrough_ll-True] SKIPPED [ 63%]
test/test_examples.py::test_passthrough[True-passthrough-False] SKIPPED  [ 65%]
test/test_examples.py::test_passthrough[True-passthrough-True] SKIPPED   [ 68%]
test/test_examples.py::test_passthrough[True-passthrough_fh-False] SKIPPED [ 70%]
test/test_examples.py::test_passthrough[True-passthrough_fh-True] SKIPPED [ 73%]
test/test_examples.py::test_passthrough[True-passthrough_ll-False] SKIPPED [ 75%]
test/test_examples.py::test_passthrough[True-passthrough_ll-True] SKIPPED [ 78%]
test/test_examples.py::test_passthrough_hp[False] SKIPPED                [ 80%]
test/test_examples.py::test_passthrough_hp[True] SKIPPED                 [ 82%]
test/test_examples.py::test_ioctl SKIPPED                                [ 85%]
test/test_examples.py::test_poll SKIPPED                                 [ 87%]
test/test_examples.py::test_null SKIPPED                                 [ 90%]
test/test_examples.py::test_notify_inval_entry[True] SKIPPED             [ 92%]
test/test_examples.py::test_notify_inval_entry[False] SKIPPED            [ 95%]
test/test_examples.py::test_cuse SKIPPED                                 [ 97%]
test/test_examples.py::test_printcap SKIPPED                             [100%]

=========================== short test summary info ============================
SKIPPED [2] test_ctests.py:21: not supported by running kernel
SKIPPED [4] test_ctests.py:42: not supported by running kernel
SKIPPED [2] test_ctests.py:73: not supported by running kernel
SKIPPED [12] test_examples.py:75: FUSE kernel module does not seem to be loaded
SKIPPED [12] test_examples.py:102: FUSE kernel module does not seem to be loaded
SKIPPED [2] test_examples.py:174: FUSE kernel module does not seem to be loaded
SKIPPED [1] test_examples.py:236: not supported by running kernel
SKIPPED [2] test_examples.py: FUSE kernel module does not seem to be loaded
SKIPPED [2] test_examples.py:310: not supported by running kernel
SKIPPED [1] test_examples.py:346: FUSE kernel module does not seem to be loaded
SKIPPED [1] util.py: FUSE kernel module does not seem to be loaded
============================= 41 skipped in 0.50s ==============================
[0/1] Installing files.
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
update-rc.d: warning: stop runlevel arguments (0 6) do not match fuse3 Default-Stop values (none)
Installing lib/libfuse3.so.3.9.2 to /usr/local/lib/x86_64-linux-gnu/libfuse3.so.3.9.2
Installing util/fusermount3 to /usr/local/bin/fusermount3
Installing util/mount.fuse3 to /usr/local/sbin/mount.fuse3
Installing fuse.h to /usr/local/include/fuse3
Installing fuse_common.h to /usr/local/include/fuse3
Installing fuse_lowlevel.h to /usr/local/include/fuse3
Installing fuse_opt.h to /usr/local/include/fuse3
Installing cuse_lowlevel.h to /usr/local/include/fuse3
Installing fuse_log.h to /usr/local/include/fuse3
Installing /build_dir/libfuse/doc/fusermount3.1 to /usr/local/share/man/man1
Installing /build_dir/libfuse/doc/mount.fuse3.8 to /usr/local/share/man/man8
Installing /build_dir/libfuse/build/meson-private/fuse3.pc to /usr/local/lib/x86_64-linux-gnu/pkgconfig
Running custom install script '/build_dir/libfuse/util/install_helper.sh /usr/local/etc /usr/local/bin /lib/udev/rules.d true'

So it looks that libfuse is built and installed, although the tests are all skipped. However, s3ql build is still failing:

MANIFEST.in exists, running in developer mode
Traceback (most recent call last):
  File "setup.py", line 36, in <module>
    import s3ql
  File "/build_dir/s3ql/src/s3ql/__init__.py", line 39, in <module>
    from pyfuse3 import ROOT_INODE

ImportError: libfuse3.so.3: cannot open shared object file: No such file or directory

I've updated the script (attached) with all your suggestions.

Could it be the location where libfuse is installed? Is there a way to put that in an env variable, like the CFLAGS?

Thanks.

[Daniel Jagszent]

Hi,

[...]
Installing lib/libfuse3.so.3.9.2 to /usr/local/lib/x86_64-linux-gnu/libfuse3.so.3.9.2
[...]

is the directory /usr/local/lib/x86_64-linux-gnu in your library search path?
You may need to (a) install libfuse into a standard path of your OS or (b) add the path to /etc/ld/so.conf and run ldconfig
See https://linux.die.net/man/8/ldconfig

[Alexandre Gonçalves]

That was it! After adding the path of libfuse lib using ldconfig, everything went smoothly!

Thanks.

 

[Alexandre Gonçalves]

After successfully building s3ql, and passed the tests, I'm seeing these errors:

Traceback (most recent call last):

  File "/usr/local/bin/mount.s3ql", line 33, in <module>
    sys.exit(load_entry_point('s3ql==3.5.0', 'console_scripts', 'mount.s3ql')())
  File "/usr/local/bin/mount.s3ql", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/lib/python3.6/dist-packages/importlib_metadata/__init__.py", line 105, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/usr/local/lib/python3.6/dist-packages/s3ql-3.5.0-py3.6-linux-x86_64.egg/s3ql/mount.py", line 18, in <module>
    from .metadata import (download_metadata, upload_metadata, dump_and_upload_metadata,
  File "/usr/local/lib/python3.6/dist-packages/s3ql-3.5.0-py3.6-linux-x86_64.egg/s3ql/metadata.py", line 13, in <module>
    from .deltadump import INTEGER, BLOB, dump_table, load_table
ImportError: cannot import name 'INTEGER'

Any suggestions?

Thanks.

[Marcin Ciesielski]

I have the same problem, any ideas?

[Daniel Jagszent]

You are missing the deltadump.c file in src/s3ql. You have two options:

(a) do not directly use the git versions of S3QL but install via the releases: https://github.com/s3ql/s3ql/releases/tag/release-3.5.0  these files include the deltadump.c file. Remove the line
CFLAGS=$MY_CFLAGS python3 setup.py build_cython
from your install script.

(b) Install Cython ( https://cython.org/ ) and then the two lines of your compile script
CFLAGS=$MY_CFLAGS python3 setup.py build_cython
CFLAGS=$MY_CFLAGS python3 setup.py build_ext  --inplace
actually can do what they are suppose to do.


Again, do add a a "set -e" as second line of your script (see my first message on what that does). 

This
#!/bin/bash
export DEBIAN_FRONTEND=noninteractive

will become
#!/bin/bash
set -e
export DEBIAN_FRONTEND=noninteractive

[Daniel Jagszent]

I compile from the release archive

I can see the deltadump.c file
however, I am still getting this[...]

do you execute "python3 setup.py build_ext  --inplace" (or something similar)?

[Marcin Ciesielski]

I am sorry I think I must have answered via email and it git lost
Yes I do precisely this command
The I also execute testing and it passes without issues

Buster

Python 3.6

[Marcin Ciesielski]

this is precisely the error I am getting
building docker image from python:3.6-slim - hence working as root

Traceback (most recent call last):

  File "/root/.local/bin/mkfs.s3ql", line 33, in <module>
    sys.exit(load_entry_point('s3ql==3.5.0', 'console_scripts', 'mkfs.s3ql')())
  File "/root/.local/bin/mkfs.s3ql", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/root/.local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 105, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in

_load_backward_compatible
  File "/root/.local/lib/python3.6/site-packages/s3ql-3.5.0-py3.6-linux-x86_64.egg/s3ql/mkfs.py", line 15, in <module>
    from .metadata import dump_and_upload_metadata, create_tables
  File "/root/.local/lib/python3.6/site-packages/s3ql-3.5.0-py3.6-linux-x86_64.egg/s3ql/metadata.py", line 13, in <module>

    from .deltadump import INTEGER, BLOB, dump_table, load_table
ImportError: cannot import name 'INTEGER'

On Tuesday, 21 July 2020 17:38:42 UTC+1, Daniel Jagszent wrote:

[Marcin Ciesielski]

plus I get this error regardless if working from git repo or from release archive

On Tuesday, 21 July 2020 17:38:42 UTC+1, Daniel Jagszent wrote:

[r0ps3c]

I am seeing the same issue, and have been seeing it since at least 3.4.1. Looking at setup.py's history, looks like the cython build code hasn't been touched, and likely hasn't been updated, in a while.

For reference, I've been able to reproduce this on (docker-ised) ubuntu 20.04.1 and alpine 3.12.

[Marcin Ciesielski]

I finally was able to avoid this issue using fedora image

[Chris Robinson]

I also get the error: ImportError: cannot import name 'INTEGER' when running 'sudo s3qladm' or any other s3qlxxx command globally.

Ubuntu 18.04, S3QL 3.5.1 installed with python3 setup.py build_ext --inplace and then sudo python3 setup.py install.

All tests ran successfully after the build.

I can run  s3ql commands successfully by changing back into my build directory and running 'bin/s3qladm' or any other command, so I imagine that some component is not getting installed globally by setup.py install.

The problem is that I need to have the commands available globally for ease of scripting, etc.

Thanks for a really great FS and such superb documentaion.

[Nikolaus Rath]

On Thu, 1 Oct 2020, at 11:39, Chris Robinson wrote:

I also get the error: ImportError: cannot import name 'INTEGER' when running 'sudo s3qladm' or any other s3qlxxx command globally.

Ubuntu 18.04, S3QL 3.5.1 installed with python3 setup.py build_ext --inplace and then sudo python3 setup.py install.

This means that the path that the deltadump.so (filename approximate) was installed into is in your user's $PYTHONPATH, but not in root's PYTHONPATH.

Not sure why, but hopefully this will help you to find out and fix it :-).

-Nikolaus

--

GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«