Could not resolve host: mirrors.centos.org during the build of the docker container image

jean-christophe manciot

unread,

Jun 2, 2023, 10:17:09 AM6/2/23

to AWX Project

I'm trying to build our own docker container image with:

su - awx
git clone -b 22.3.0 https://github.com/ansible/awx.git
cd awx
git switch -c release_4.4

. /opt/awx/venv/bin/activate
make docker-compose-build

which leads to:

ansible-playbook tools/ansible/dockerfile.yml \
-e dockerfile_name=Dockerfile.dev \
-e build_dev=True \
-e receptor_image=quay.io/ansible/receptor:devel
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [Render AWX Dockerfile and sources] ************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [localhost]

TASK [dockerfile : Create _build directory] *********************************************************************************************************************************
ok: [localhost]

TASK [dockerfile : Render supervisor configs] *******************************************************************************************************************************
ok: [localhost] => (item=supervisor_web.conf)
ok: [localhost] => (item=supervisor_task.conf)
ok: [localhost] => (item=supervisor_rsyslog.conf)

TASK [dockerfile : Render Dockerfile] ***************************************************************************************************************************************
changed: [localhost]

PLAY RECAP ******************************************************************************************************************************************************************
localhost : ok=4 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

DOCKER_BUILDKIT=1 docker build \
-f Dockerfile.dev \
-t ghcr.io/ansible/awx_devel:release_4.4 \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from=ghcr.io/ansible/awx_devel:release_4.4 .
[+] Building 75.7s (11/47)
=> [internal] load build definition from Dockerfile.dev 0.0s
=> => transferring dockerfile: 7.57kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 56B 0.0s
=> [internal] load metadata for quay.io/ansible/receptor:devel   0.4s
=> [internal] load metadata for quay.io/centos/centos:stream9   0.4s
=> importing cache manifest from ghcr.io/ansible/awx_devel:release_4.4   0.8s
=> [builder 1/10] FROM quay.io/centos/centos:stream9@sha256:d075b8cb028de107de53d601512f6f2fc70c8d3c1313959d5925a5386dd1d665   0.0s
=> [internal] load build context 0.0s
=> => transferring context: 47.03kB 0.0s
=> FROM quay.io/ansible/receptor:devel@sha256:b14b7bc6a77fa04e61fd550bdec0e4134aedba23a4144bcd13ce738523c6acf9   0.0s
=> CACHED [builder 2/10] RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial 0.0s
=> ERROR [builder 3/10] RUN dnf -y update && dnf install -y 'dnf-command(config-manager)' && dnf config-manager --set-enabled crb && dnf -y install gcc 74.4s
=> ERROR [stage-1 3/32] RUN dnf -y update && dnf install -y 'dnf-command(config-manager)' && dnf config-manager --set-enabled crb && dnf -y install acl 74.3s
------
> [builder 3/10] RUN dnf -y update && dnf install -y 'dnf-command(config-manager)' && dnf config-manager --set-enabled crb && dnf -y install gcc gcc-c++ git-core gettext glibc-langpack-en libffi-devel libtool-ltdl-devel make nodejs nss openldap-devel patch postgresql postgresql-devel python3-devel python3-pip python3-psycopg2 python3-setuptools swig unzip xmlsec1-devel xmlsec1-openssl-devel:
#0 0.413 /bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
#0 0.471 Failed to set locale, defaulting to C.UTF-8
#0 74.30 CentOS Stream 9 - BaseOS 0.0 B/s | 0 B 01:13
#0 74.30 Errors during downloading metadata for repository 'baseos':
#0 74.30 - Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Could not resolve host: mirrors.centos.org]
#0 74.31 Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Could not resolve host: mirrors.centos.org]
------
------
> [stage-1 3/32] RUN dnf -y update && dnf install -y 'dnf-command(config-manager)' && dnf config-manager --set-enabled crb && dnf -y install acl git-core git-lfs glibc-langpack-en krb5-workstation nginx "openldap >= 2.6.2-3" postgresql python3-devel python3-libselinux python3-pip python3-psycopg2 python3-setuptools rsync rsyslog-8.2102.0-106.el9 subversion sudo vim-minimal which unzip xmlsec1-openssl && dnf -y clean all:
#0 0.412 /bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
#0 0.471 Failed to set locale, defaulting to C.UTF-8
#0 74.30 CentOS Stream 9 - BaseOS 0.0 B/s | 0 B 01:13
#0 74.30 Errors during downloading metadata for repository 'baseos':
#0 74.30 - Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http&countme=1 [Could not resolve host: mirrors.centos.org]
#0 74.30 - Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Could not resolve host: mirrors.centos.org]
#0 74.31 Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Could not resolve host: mirrors.centos.org]
------
Dockerfile.dev:82
--------------------
81 | # Install runtime requirements
82 | >>> RUN dnf -y update && dnf install -y 'dnf-command(config-manager)' && \
83 | >>> dnf config-manager --set-enabled crb && \
84 | >>> dnf -y install acl \
85 | >>> git-core \
86 | >>> git-lfs \
87 | >>> glibc-langpack-en \
88 | >>> krb5-workstation \
89 | >>> nginx \
90 | >>> "openldap >= 2.6.2-3" \
91 | >>> postgresql \
92 | >>> python3-devel \
93 | >>> python3-libselinux \
94 | >>> python3-pip \
95 | >>> python3-psycopg2 \
96 | >>> python3-setuptools \
97 | >>> rsync \
98 | >>> rsyslog-8.2102.0-106.el9 \
99 | >>> subversion \
100 | >>> sudo \
101 | >>> vim-minimal \
102 | >>> which \
103 | >>> unzip \
104 | >>> xmlsec1-openssl && \
105 | >>> dnf -y clean all
106 |
--------------------
ERROR: failed to solve: process "/bin/sh -c dnf -y update && dnf install -y 'dnf-command(config-manager)' && dnf config-manager --set-enabled crb && dnf -y install acl git-core git-lfs glibc-langpack-en krb5-workstation nginx \"openldap >= 2.6.2-3\" postgresql python3-devel python3-libselinux python3-pip python3-psycopg2 python3-setuptools rsync rsyslog-8.2102.0-106.el9 subversion sudo vim-minimal which unzip xmlsec1-openssl && dnf -y clean all" did not complete successfully: exit code: 1
make: *** [Makefile:572: docker-compose-build] Error 1

There is no such DNS resolution issue on the host (which is a KVM VM):

$ ping mirrors.centos.org
PING wildcard.fedoraproject.org (18.159.254.57) 56(84) bytes of data.
64 bytes from ec2-18-159-254-57.eu-central-1.compute.amazonaws.com (18.159.254.57): icmp_seq=1 ttl=48 time=12.7 ms

The iptables rules are automatically defined for the docker0 linux bridge:

# grep -i docker /etc/iptables/rules.v4
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN

I have also disabled the Source Address Verification for docker0 in /etc/sysctl.d/10-network-security.conf:

echo "net.ipv4.conf.docker0.rp_filter=0" >> /etc/sysctl.d/10-network-security.conf
systemctl restart systemd-sysctl.service

According to the official docker documentation:

By default, containers inherit the DNS settings of the host, as defined in the /etc/resolv.conf configuration file

in the host /etc/resolv.conf:

nameserver 8.8.4.4

But since the issue is there, I also tried to define the DNS server in /etc/docker/daemon.json:

{
"debug": true,
"dns": ["8.8.4.4"],
"ipv6": false
}

I'm running out of options and I don't understand why this is happening.

Any suggestion/explanation?

jean-christophe manciot

unread,

Jun 2, 2023, 10:41:33 AM6/2/23

to AWX Project

Same issue with:

- awx_devel:release_4.2

- awx_devel:release_4.3

jean-christophe manciot

unread,

Jun 2, 2023, 12:00:35 PM6/2/23

to AWX Project

I have no such issue using a CentOS stream 9 host instead of Ubuntu.

Any idea why?

AWX Project

unread,

Jun 2, 2023, 2:23:01 PM6/2/23

to AWX Project

We see this error and seems relevant to your issue

#0 74.30 - Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Could not resolve host: mirrors.centos.org]

are you able to successfully curl this url from the terminal that you are running the make docker-compose-build https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http

AWX Team

jean-christophe manciot

unread,

Jun 5, 2023, 8:19:08 AM6/5/23

to AWX Project

Yes:

(venv) awx@sdx1-kvm:~/git-awx$ curl https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http
[1] 137878
[2] 137879
(venv) awx@sdx1-kvm:~/git-awx$ <?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Mon, 05 Jun 2023 11:44:37 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager">

</metalink>

[1]- Done curl https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream
[2]+ Done arch=x86_64
(venv) awx@sdx1-kvm:~/git-awx$

The issue is still there when 'make docker-compose-build' is run as awx (non-root) user.

Here is some issue(s) which appear in the docker log:

# echo 'q' | sudo systemctl --no-pager --full status docker

● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; preset: enabled)
Active: active (running) since Mon 2023-06-05 11:42:24 UTC; 14min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 59654 (dockerd)
Tasks: 12
Memory: 216.0M
CPU: 3.006s
CGroup: /system.slice/docker.service
└─59654 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Jun 05 11:50:58 sdx1-kvm dockerd[59654]: time="2023-06-05T11:50:58.721767833Z" level=debug msg="2023/06/05 11:50:58 WARNING: [core] [Channel #11 SubChannel #12] grpc: addrConn.createTransport failed to connect to {" library=grpc
Jun 05 11:50:58 sdx1-kvm dockerd[59654]: time="2023-06-05T11:50:58.721808791Z" level=debug msg=" \"Addr\": \"localhost\"," library=grpc
Jun 05 11:50:58 sdx1-kvm dockerd[59654]: time="2023-06-05T11:50:58.721813951Z" level=debug msg=" \"ServerName\": \"localhost\"," library=grpc
Jun 05 11:50:58 sdx1-kvm dockerd[59654]: time="2023-06-05T11:50:58.721817898Z" level=debug msg=" \"Attributes\": null," library=grpc
Jun 05 11:50:58 sdx1-kvm dockerd[59654]: time="2023-06-05T11:50:58.721821385Z" level=debug msg=" \"BalancerAttributes\": null," library=grpc
Jun 05 11:50:58 sdx1-kvm dockerd[59654]: time="2023-06-05T11:50:58.721824711Z" level=debug msg=" \"Type\": 0," library=grpc
Jun 05 11:50:58 sdx1-kvm dockerd[59654]: time="2023-06-05T11:50:58.721828698Z" level=debug msg=" \"Metadata\": null" library=grpc
Jun 05 11:50:58 sdx1-kvm dockerd[59654]: time="2023-06-05T11:50:58.721831995Z" level=debug msg="}. Err: connection error: desc = \"transport: Error while dialing only one connection allowed\"" library=grpc
Jun 05 11:52:33 sdx1-kvm dockerd[59654]: time="2023-06-05T11:52:33.724284994Z" level=warning msg="leaked execError detected and released"

jean-christophe manciot

unread,

Jun 7, 2023, 5:23:54 AM6/7/23

to AWX Project

Any suggestion regarding this "Error while dialing only one connection allowed" with grpc?

AWX Project

unread,

Jun 9, 2023, 1:20:55 PM6/9/23

to AWX Project

can you provide information about where you are running the make command? is this on a linux VM? if so which OS and version, and which docker version are you using?

AWX Team

jean-christophe manciot

unread,

Jun 9, 2023, 2:15:11 PM6/9/23

to AWX Project

- Ubuntu 23.04 lunar on a KVM VM running on a similar host

- docker-ce 5:24.0.2-1~ubuntu.23.04~lunar from the official docker PPA (deb [arch=amd64] https://download.docker.com/linux/ubuntu lunar stable edge)

- python3.9 3.9.5-3ubuntu0~21.04.1 (backported from Ubuntu hirsute 21.04)

- pip/wheel/ansible/docker/docker-compose/redis/setuptools-scm installed from PyPi using pip3 inside a virtual env

jean-christophe manciot

unread,

Jun 14, 2023, 11:57:14 AM6/14/23

to AWX Project

Exact same issue with:

- Ubuntu 22.04 jammy LTS on a KVM VM
- docker-ce 5:24.0.2-1~ubuntu.22.04~jammy from the official docker PPA (deb [arch=amd64] https://download.docker.com/linux/ubuntu jammy stable edge)

- python3.9 3.9.17-1+jammy1 (from https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy/main amd64 Packages)

- pip/wheel/ansible/docker/docker-compose/redis/setuptools-scm installed from PyPi using pip3 inside a virtual env

Same strange grpc error:

# echo q | systemctl --no-pager --full status docker

● docker.service - Docker Application Container Engine

Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-06-14 15:43:04 UTC; 11min ago

TriggeredBy: ● docker.socket
Docs: https://docs.docker.com

Main PID: 211829 (dockerd)
Tasks: 12
Memory: 214.3M
CPU: 4.695s
CGroup: /system.slice/docker.service
└─211829 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Jun 14 15:45:59 sdx1-kvm dockerd[211829]: time="2023-06-14T15:45:59.935797363Z" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.3)"
Jun 14 15:45:59 sdx1-kvm dockerd[211829]: time="2023-06-14T15:45:59.935818162Z" level=debug msg="Released address Address:172.17.0.3 Sequence:Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:5"
Jun 14 15:46:03 sdx1-kvm dockerd[211829]: time="2023-06-14T15:46:03.897765133Z" level=debug msg="2023/06/14 15:46:03 WARNING: [core] [Channel #11 SubChannel #12] grpc: addrConn.createTransport failed to connect to {" library=grpc
Jun 14 15:46:03 sdx1-kvm dockerd[211829]: time="2023-06-14T15:46:03.903195475Z" level=debug msg=" \"Addr\": \"localhost\"," library=grpc
Jun 14 15:46:03 sdx1-kvm dockerd[211829]: time="2023-06-14T15:46:03.903237455Z" level=debug msg=" \"ServerName\": \"localhost\"," library=grpc
Jun 14 15:46:03 sdx1-kvm dockerd[211829]: time="2023-06-14T15:46:03.903246712Z" level=debug msg=" \"Attributes\": null," library=grpc
Jun 14 15:46:03 sdx1-kvm dockerd[211829]: time="2023-06-14T15:46:03.903254036Z" level=debug msg=" \"BalancerAttributes\": null," library=grpc
Jun 14 15:46:03 sdx1-kvm dockerd[211829]: time="2023-06-14T15:46:03.903261229Z" level=debug msg=" \"Type\": 0," library=grpc
Jun 14 15:46:03 sdx1-kvm dockerd[211829]: time="2023-06-14T15:46:03.903273694Z" level=debug msg=" \"Metadata\": null" library=grpc
Jun 14 15:46:03 sdx1-kvm dockerd[211829]: time="2023-06-14T15:46:03.903280757Z" level=debug msg="}. Err: connection error: desc = \"transport: Error while dialing only one connection allowed\"" library=grpc

Reply all

Reply to author

Forward