Re: image not portable(?) with Qt5 libs not found when running on some hosts

[Dave Dykstra]

[moving from singu...@lbl.gov to new dis...@apptainer.org list]

Hi Thomas,

Can you show us how to get the nc.d container so we can potentially
reproduce the problem?

Dave


On Mon, Feb 07, 2022 at 04:00:11PM +0100, Thomas Hartmann wrote:
> Hi all,
>
> we see an odd issue, where a container seems not to be portable between
> hosts.
>
> The container is based on FC34 (tried CentOS7 as well) and only the
> nextcloud-client is installed from the default repos. The builds seem
> unssupicious on various hosts.
> On my FC34 laptop, I can run the application without problems when
> shelling into the resulting container image [1].
> However, when trying the same on other hosts (CentOS7 etc.), the
> application throws an error about missing its QT5 shared libraries [2]?!
> Which is (nominally) nonsense as the library is present in the
> container's namespace [2.b] and envvars are set in the instances paths.
>
> Comparing the linked libraries [3] actually shows, that in the instance
> on a CentOS7 host the library is actually not found - where on the same
> container image on FC34 it is found/linked.
>
> Maybe somebody has an idea what could be causing this strange behaviour?
>
> Cheers and thanks,
> Thomas
>
> [1]
> > singularity --version
> singularity version 3.8.5-2.fc34
>
> > singularity shell --containall nc.d/
> Singularity> nextcloudcmd --help
> nextcloudcmd - command line Nextcloud client tool
>
> Usage: nextcloudcmd [OPTION] <source_dir> <server_url>
> ...
>
>
>
>
> [2]
> > singularity --version
> singularity version 3.8.5-2.el7
>
> > singularity shell --containall nc.d/
> Singularity> nextcloudcmd --help
> nextcloudcmd: error while loading shared libraries: libQt5Core.so.5:
> cannot open shared object file: No such file or directory
>
> [2.b]
> Singularity> ls /usr/lib*/libQt5Core.so.5
> /usr/lib64/libQt5Core.so.5
>
>
> [3]
> > sdiff /tmp/fc34.ldd.sort /tmp/co7.ldd.sort
> /lib64/ld-linux-x86-64.so.2 (0x00007fae5db73000) |
> /lib64/ld-linux-x86-64.so.2 (0x00007f6a1c4a9000)
> libbrotlicommon.so.1 => /usr/lib64/libbrotlicommon.so |
> libbrotlicommon.so.1 => /usr/lib64/libbrotlicommon.so
> libbrotlidec.so.1 => /usr/lib64/libbrotlidec.so.1 (0x |
> libbrotlidec.so.1 => /usr/lib64/libbrotlidec.so.1 (0x
> libbz2.so.1 => /usr/lib64/libbz2.so.1 (0x00007fae58b9 | libbz2.so.1 =>
> /usr/lib64/libbz2.so.1 (0x00007f6a19e4
> libcap.so.2 => /usr/lib64/libcap.so.2 (0x00007fae5ad1 | libcap.so.2 =>
> /usr/lib64/libcap.so.2 (0x00007f6a19c4
> libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x0000 |
> libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x0000
> libcrypto.so.1.1 => /usr/lib64/libcrypto.so.1.1 (0x00 |
> libcrypto.so.1.1 => /usr/lib64/libcrypto.so.1.1 (0x00
> libc.so.6 => /usr/lib64/libc.so.6 (0x00007fae5cdf5000 | libc.so.6 =>
> /usr/lib64/libc.so.6 (0x00007f6a1bcae000
> libdbus-1.so.3 => /usr/lib64/libdbus-1.so.3 (0x00007f | libdbus-1.so.3
> => /usr/lib64/libdbus-1.so.3 (0x00007f
> libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007fae5bb870 | libdl.so.2 =>
> /usr/lib64/libdl.so.2 (0x00007f6a1aa3e0
> libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00 |
> libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00
> libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007fae | libgcc_s.so.1
> => /usr/lib64/libgcc_s.so.1 (0x00007f6a
> libgcrypt.so.20 => /usr/lib64/libgcrypt.so.20 (0x0000 |
> libgcrypt.so.20 => /usr/lib64/libgcrypt.so.20 (0x0000
> libGLdispatch.so.0 => /usr/lib64/libGLdispatch.so.0 ( |
> libGLdispatch.so.0 => /usr/lib64/libGLdispatch.so.0 (
> libglib-2.0.so.0 => /usr/lib64/libglib-2.0.so.0 (0x00 |
> libglib-2.0.so.0 => /usr/lib64/libglib-2.0.so.0 (0x00
> libGL.so.1 => /usr/lib64/libGL.so.1 (0x00007fae5b0ad0 | libGL.so.1 =>
> /usr/lib64/libGL.so.1 (0x00007f6a1a5aa0
> libGLX.so.0 => /usr/lib64/libGLX.so.0 (0x00007fae58f9 | libGLX.so.0 =>
> /usr/lib64/libGLX.so.0 (0x00007f6a1a27
> libgpg-error.so.0 => /usr/lib64/libgpg-error.so.0 (0x |
> libgpg-error.so.0 => /usr/lib64/libgpg-error.so.0 (0x
> libgraphite2.so.3 => /usr/lib64/libgraphite2.so.3 (0x |
> libgraphite2.so.3 => /usr/lib64/libgraphite2.so.3 (0x
> libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 |
> libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2
> libharfbuzz.so.0 => /usr/lib64/libharfbuzz.so.0 (0x00 |
> libharfbuzz.so.0 => /usr/lib64/libharfbuzz.so.0 (0x00
> libicudata.so.67 => /usr/lib64/libicudata.so.67 (0x00 |
> libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00
> libicui18n.so.67 => /usr/lib64/libicui18n.so.67 (0x00 |
> libkeyutils.so.1 => /usr/lib64/libkeyutils.so.1 (0x00
> libicuuc.so.67 => /usr/lib64/libicuuc.so.67 (0x00007f | libkrb5.so.3
> => /usr/lib64/libkrb5.so.3 (0x00007f6a1a
> libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00 |
> libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0
> libkeyutils.so.1 => /usr/lib64/libkeyutils.so.1 (0x00 | liblz4.so.1 =>
> /usr/lib64/liblz4.so.1 (0x00007f6a19c4
> libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007fae5a | liblzma.so.5
> => /usr/lib64/liblzma.so.5 (0x00007f6a19
> libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 | libm.so.6 =>
> /usr/lib64/libm.so.6 (0x00007f6a1a806000
> liblz4.so.1 => /usr/lib64/liblz4.so.1 (0x00007fae5ad2 |
> libnextcloud_csync.so.0 => /usr/lib64/libnextcloud_cs
> liblzma.so.5 => /usr/lib64/liblzma.so.5 (0x00007fae5a |
> libnextcloudsync.so.0 => /usr/lib64/libnextcloudsync.
> libm.so.6 => /usr/lib64/libm.so.6 (0x00007fae5b1cb000 |
> libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 (0x0000
> libnextcloud_csync.so.0 => /usr/lib64/libnextcloud_cs | libpcre.so.1
> => /usr/lib64/libpcre.so.1 (0x00007f6a1a
> libnextcloudsync.so.0 => /usr/lib64/libnextcloudsync. | libpng16.so.16
> => /usr/lib64/libpng16.so.16 (0x00007f
> libpcre2-16.so.0 => /usr/lib64/libpcre2-16.so.0 (0x00 |
> libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x0000
> libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 (0x0000 |
> libQt5Core.so.5 => not found
> libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fae59 |
> libQt5Core.so.5 => not found
> libpng16.so.16 => /usr/lib64/libpng16.so.16 (0x00007f |
> libQt5Core.so.5 => not found
> libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x0000 |
> libQt5Core.so.5 => not found
> libQt5Core.so.5 => /usr/lib64/libQt5Core.so.5 (0x0000 |
> libQt5Core.so.5 => not found
> libQt5DBus.so.5 => /usr/lib64/libQt5DBus.so.5 (0x0000 |
> libQt5Core.so.5 => not found
> libQt5Gui.so.5 => /usr/lib64/libQt5Gui.so.5 (0x00007f |
> libQt5Core.so.5 => not found
> libqt5keychain.so.1 => /usr/lib64/libqt5keychain.so.1 |
> libQt5Core.so.5 => not found
> libQt5Network.so.5 => /usr/lib64/libQt5Network.so.5 ( |
> libQt5Core.so.5 => not found
> libQt5Svg.so.5 => /usr/lib64/libQt5Svg.so.5 (0x00007f |
> libQt5Core.so.5 => not found
> libQt5WebSockets.so.5 => /usr/lib64/libQt5WebSockets. |
> libQt5DBus.so.5 => /usr/lib64/libQt5DBus.so.5 (0x0000
> libQt5Widgets.so.5 => /usr/lib64/libQt5Widgets.so.5 ( | libQt5Gui.so.5
> => /usr/lib64/libQt5Gui.so.5 (0x00007f
> libresolv.so.2 => /usr/lib64/libresolv.so.2 (0x00007f |
> libqt5keychain.so.1 => /usr/lib64/libqt5keychain.so.1
> librt.so.1 => /usr/lib64/librt.so.1 (0x00007fae5ae660 |
> libQt5Network.so.5 => /usr/lib64/libQt5Network.so.5 (
> libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x0000 | libQt5Svg.so.5
> => /usr/lib64/libQt5Svg.so.5 (0x00007f
> libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x0000 |
> libQt5WebSockets.so.5 => /usr/lib64/libQt5WebSockets.
> libssl.so.1.1 => /usr/lib64/libssl.so.1.1 (0x00007fae |
> libQt5Widgets.so.5 => /usr/lib64/libQt5Widgets.so.5 (
> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f | libresolv.so.2
> => /usr/lib64/libresolv.so.2 (0x00007f
> libsystemd.so.0 => /usr/lib64/libsystemd.so.0 (0x0000 | librt.so.1 =>
> /usr/lib64/librt.so.1 (0x00007f6a19d960
> libX11.so.6 => /usr/lib64/libX11.so.6 (0x00007fae58e4 |
> libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x0000
> libXau.so.6 => /usr/lib64/libXau.so.6 (0x00007fae58b8 |
> libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x0000
> libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007fae58bb | libssl.so.1.1
> => /usr/lib64/libssl.so.1.1 (0x00007f6a
> libXext.so.6 => /usr/lib64/libXext.so.6 (0x00007fae58 | libstdc++.so.6
> => /usr/lib64/libstdc++.so.6 (0x00007f
> libz.so.1 => /usr/lib64/libz.so.1 (0x00007fae5ca37000 |
> libsystemd.so.0 => /usr/lib64/libsystemd.so.0 (0x0000
> libzstd.so.1 => /usr/lib64/libzstd.so.1 (0x00007fae5a | libX11.so.6 =>
> /usr/lib64/libX11.so.6 (0x00007f6a1a12
> linux-vdso.so.1 (0x00007ffd0f5a8000) | libXau.so.6 =>
> /usr/lib64/libXau.so.6 (0x00007f6a19b0
> > libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007f6a19e5
> > libXext.so.6 => /usr/lib64/libXext.so.6 (0x00007f6a1a
> > libz.so.1 => /usr/lib64/libz.so.1 (0x00007f6a1b8f0000
> > libzstd.so.1 => /usr/lib64/libzstd.so.1 (0x00007f6a19
> > linux-vdso.so.1 (0x00007ffc020ff000)
>
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/5d117f69-0a47-ddf5-6861-86e31cb85139%40desy.de.

[Oliver Freyermuth]

Hi Thomas et al,

we've seen this kind of issue, too, it's caused by Qt5 using a system call which is not present in the kernels shipped with CentOS 7.
Sadly, the error message is less than intuitive.

You can get some more background information here:
https://github.com/dnschneid/crouton/wiki/Fix-error-while-loading-shared-libraries:-libQt5Core.so.5
(renameat2 system call, used starting from Qt 5.10). The page also contains an ugly workaround (patching the library).

Hence, in case somebody else wants to reproduce, the Qt versions included in Debian 10, Debian 11, Ubuntu 20.04
(not yet in Ubuntu 18.04) all trigger this behaviour (i.e. any program loading QtCore in such a container running on a CentOS 7 host with distro kernel will show this).

Since the system call API is usually always kept backwards compatible, the only real safeguard against such incompatibilities seems to be to always use a newer kernel on the host OS
than the newest container OS you want to run. Luckily, most software does not rely on too recent syscalls for this to bite us more often.

Cheers and hope this helps,
Oliver

Am 07.02.22 um 16:00 schrieb Thomas Hartmann:

Oliver Freyermuth
Universität Bonn
Physikalisches Institut, Raum 1.047
Nußallee 12
53115 Bonn
--
Tel.: +49 228 73 2367
Fax: +49 228 73 7869
--

[Thomas Hartmann]

Hi Oliver,

many thanks for the information!

A kernel/system call dependency is something I am not so enthusiastic
about... 😑

Cheers,
Thomas