error while loading shared libraries: libc.so.6:

[Nick Moore]

Getting the following output when trying to run the container:

I've tried this on multiple releases.

docker run --name some-rundeck -p 4440:4440 -v data:/home/rundeck/server/data rundeck/rundeck:3.4.8
/tini: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory

[rac...@rundeck.com]

Hi Nick,

I tried to reproduce your issue unsuccessfully, but researching a bit I found that is a problem related to your OS kernel and the container image (or a docker image integrity problem, so, a good fix is to try with a fresh new clean image).

Greetings!

[Nick Moore]

(base) root@itnet-bastion0:~# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

(base) root@itnet-bastion0:~# docker system prune -a
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all images without at least one container associated to them
  - all build cache

Are you sure you want to continue? [y/N] y
Deleted Containers:
0d9f4d243436d8e61568c0b42330145a3fa6e40d10464db5445bbf9e4c772518

Deleted Networks:
welcome-project-community_default

Deleted Images:
untagged: rundeck/rundeck:3.4.9

(base) root@itnet-bastion0:~# docker pull rundeck/rundeck:3.4.9
3.4.9: Pulling from rundeck/rundeck
92dc2a97ff99: Pull complete
be13a9d27eb8: Pull complete
c8299583700a: Pull complete
9e6bddf44d59: Pull complete
e4213de99ee1: Pull complete
70f2ef2fdcfc: Pull complete
9df1c8c1f3a7: Pull complete
1c330a0c4af6: Pull complete
2e2c4859c37e: Pull complete
377b47559c01: Pull complete
0306ab509900: Pull complete
7cf24ac718e2: Pull complete
95d036bf1f2a: Pull complete
Digest: sha256:1ee5c671686d5fcfc21dd8ace421286ee09eb086296f1f5ad6880a7ef7a4f74b
Status: Downloaded newer image for rundeck/rundeck:3.4.9
docker.io/rundeck/rundeck:3.4.9

(base) root@itnet-bastion0:~# docker image ls --all
REPOSITORY        TAG       IMAGE ID       CREATED       SIZE
rundeck/rundeck   3.4.9     17429bfbe955   3 hours ago   738MB

(base) root@itnet-bastion0:~# docker inspect 17429bfbe955 | jq '.[].RepoDigests'
[
  "rundeck/rundeck@sha256:1ee5c671686d5fcfc21dd8ace421286ee09eb086296f1f5ad6880a7ef7a4f74b"

(base) root@itnet-bastion0:~# docker run --name some-rundeck -p 4440:4440 -v data:/home/rundeck/server/data rundeck/rundeck:3.4.9

/tini: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory

[Nick Moore]

(base) root@itnet-bastion0:~# hostnamectl | grep Kernel
            Kernel: Linux 4.15.0-1021-aws

[Nick Moore]

I pruned all existing images pulled down a new one and validated the hash, still seeing the same error:

Kernel: Linux 4.15.0-1021-aws

Ubuntu 18.04.1 LTS

Docker version 20.10.11, build dea9396

[rac...@rundeck.com]

Hi,

Could you try in another environment to discard? Pretty sure that is related to OS/Docker installation.

Also, does your OS has the libc.so.6 library? (it seem required by the container).

Regards.

[Nick Moore]

It runs fine on my local mac os, I found the library under /lib/x86_64-linux-gnu and created a symlink to point it to /lib, still no dice

(base) root@itnet-bastion0:/lib/x86_64-linux-gnu# ls -l  grep libc*
ls: cannot access 'grep': No such file or directory
-rwxr-xr-x 1 root root 2030928 Dec  7  2020 libc-2.27.so
lrwxrwxrwx 1 root root      12 Dec  7  2020 libc.so.6 -> libc-2.27.so
lrwxrwxrwx 1 root root      18 Sep 24  2017 libcap-ng.so.0 -> libcap-ng.so.0.0.0
-rw-r--r-- 1 root root   18712 Sep 24  2017 libcap-ng.so.0.0.0
lrwxrwxrwx 1 root root      14 Nov 16  2017 libcap.so.2 -> libcap.so.2.25
-rw-r--r-- 1 root root   22768 Nov 16  2017 libcap.so.2.25
-rw-r--r-- 1 root root  190784 Dec  7  2020 libcidn-2.27.so
lrwxrwxrwx 1 root root      15 Dec  7  2020 libcidn.so.1 -> libcidn-2.27.so
lrwxrwxrwx 1 root root      17 Jan 22  2020 libcom_err.so.2 -> libcom_err.so.2.1
-rw-r--r-- 1 root root   14248 Jan 22  2020 libcom_err.so.2.1
-rw-r--r-- 1 root root   39208 Dec  7  2020 libcrypt-2.27.so
lrwxrwxrwx 1 root root      16 Dec  7  2020 libcrypt.so.1 -> libcrypt-2.27.so
lrwxrwxrwx 1 root root      23 Aug  3  2020 libcryptsetup.so.12 -> libcryptsetup.so.12.2.0
-rw-r--r-- 1 root root  314136 Aug  3  2020 libcryptsetup.so.12.2.0
(base) root@itnet-bastion0:/lib/x86_64-linux-gnu# pwd
/lib/x86_64-linux-gnu
(base) root@itnet-bastion0:/lib/x86_64-linux-gnu# sudo ln -s /lib/x86_64-linux-gnu/libc.so.6 /lib/libc.so.6

(base) root@itnet-bastion0:~# ls -l /lib/libc*
lrwxrwxrwx 1 root root 31 Dec 22 18:10 /lib/libc.so.6 -> /lib/x86_64-linux-gnu/libc.so.6

(base) root@itnet-bastion0:~# docker run --name some-rundeck -p 4440:4440 -v data:/home/rundeck/server/data rundeck/rundeck:3.4.9
/tini: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory

[Nick Moore]

What is the expected path for ibc.so.6?

[rac...@rundeck.com]

Hi,

Following this, it seems an OS integrity problem. So, you can obtain more clues about the container libs behavior with docker run -p 4440:4440 -e LD_DEBUG=all rundeck/rundeck:3.4.9.

Greetings.

​

[Nick Moore]

Thank you it was looking for the library in /usr/lib, I created another symlink but it still can't find the library:

         1:          trying file=/usr/lib/libc.so.6
         1:

/tini: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory

(base) root@itnet-bastion0:~# ls -l /usr/lib/libc*
lrwxrwxrwx 1 root root 31 Dec 23 16:05 /usr/lib/libc.so.6 -> /lib/x86_64-linux-gnu/libc.so.6

[Nick Moore]

Also when I run the debug on my mac I can actually see the container is looking for the libc in the original path /lib/x86_64-linux-gnu/:

         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/libjava.so [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/libverify.so [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/lib/x86_64-linux-gnu/libgcc_s.so.1 [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
         7:        /usr/lib/jvm/java-11-openjdk-amd64/lib/libjava.so: error: symbol lookup error: undefined symbol: Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand (fatal)
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/libnio.so [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/libjava.so [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/libnet.so [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/libverify.so [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
         7:        symbol=Java_com_sun_management_internal_DiagnosticCommandImpl_executeDiagnosticCommand;  lookup in file=/lib/x86_64-linux-gnu/libgcc_s.so.1 [0]

Is there possibly.a permissions reason that it can't open the file or is it that it can't actually locate the file?

(base) root@itnet-bastion0:~# ls -lah /lib/x86_64-linux-gnu/libc.so.6
lrwxrwxrwx 1 root root 12 Dec  7  2020 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.27.so

[Nick Moore]

(base) root@itnet-bastion0:~# locate libc.so.6
/lib/libc.so.6
/lib/x86_64-linux-gnu/libc.so.6
/snap/core/11993/lib/i386-linux-gnu/libc.so.6
/snap/core/11993/lib/x86_64-linux-gnu/libc.so.6
/snap/core/5328/lib/i386-linux-gnu/libc.so.6
/snap/core/5328/lib/x86_64-linux-gnu/libc.so.6
/snap/core18/2246/lib/i386-linux-gnu/libc.so.6
/snap/core18/2246/lib/x86_64-linux-gnu/libc.so.6
/snap/core18/2253/lib/i386-linux-gnu/libc.so.6
/snap/core18/2253/lib/x86_64-linux-gnu/libc.so.6
/var/lib/docker/overlay2/e719cf1a2673c285d229e8a8b24c5a586540ea36cc68d36a381181ed41bf8f0e/diff/lib/x86_64-linux-gnu/libc.so.6

[Nick Moore]

Any ideas on why it's not finding the symlinked library?