Error pulling docker images

2,302 views
Skip to first unread message

Vasudevan Balakrishnan

unread,
Nov 5, 2015, 1:02:07 PM11/5/15
to CoreOS User
I'm a newbie to CoreOS. I've successfully installed CoreOS cluster with Kubernetes on VMWare. 

When I try to create a docker container, it fails with the 401 error. I do have the .dockercfg file (as per the registry authentication). Can anyone point me in the right direction?

My service file

# /etc/systemd/system/test.service

[Unit]

Description=My Service

After=docker.service


[Service]

User=core

ExecStart=/usr/bin/docker run busybox /bin/sh -c "while true; do echo Hello World; sleep 1; done"


[Install]

WantedBy=multi-user.target


# service log 

Nov 05 12:40:32 queenbee-2 systemd[1]: Started My Service.

Nov 05 12:40:32 queenbee-2 systemd[1]: Starting My Service...

Nov 05 12:40:32 queenbee-2 docker[17851]: Unable to find image 'busybox:latest' locally

Nov 05 12:40:33 queenbee-2 docker[17851]: latest: Pulling from busybox

Nov 05 12:40:33 queenbee-2 docker[17851]: d1592a710ac3: Pulling fs layer

Nov 05 12:40:33 queenbee-2 docker[17851]: 17583c7dd0da: Pulling fs layer

Nov 05 12:40:34 queenbee-2 docker[17851]: Pulling repository busybox

Nov 05 12:40:36 queenbee-2 docker[17851]: 17583c7dd0da: Pulling image (latest) from busybox

Nov 05 12:40:36 queenbee-2 docker[17851]: 17583c7dd0da: Pulling image (latest) from busybox, endpoint: https://registry-1.docker.io/v1/

Nov 05 12:40:36 queenbee-2 docker[17851]: 17583c7dd0da: Pulling dependent layers

Nov 05 12:40:36 queenbee-2 docker[17851]: d1592a710ac3: Pulling metadata

Nov 05 12:40:36 queenbee-2 docker[17851]: d1592a710ac3: Pulling fs layer

Nov 05 12:40:37 queenbee-2 docker[17851]: d1592a710ac3: Error pulling dependent layers

Nov 05 12:40:37 queenbee-2 docker[17851]: 17583c7dd0da: Error pulling image (latest) from busybox, endpoint: https://registry-1.docker.io/v1/, Server error: Status 401 while fetching image layer (d1592a710ac323612bd786fa8ac20727c58d8a67847e5a65177c594f43919498)

Nov 05 12:40:37 queenbee-2 docker[17851]: 17583c7dd0da: Error pulling image (latest) from busybox, Server error: Status 401 while fetching image layer (d1592a710ac323612bd786fa8ac20727c58d8a67847e5a65177c594f43919498)

Nov 05 12:40:37 queenbee-2 systemd[1]: test.service: Main process exited, code=exited, status=1/FAILURE

Nov 05 12:40:37 queenbee-2 systemd[1]: test.service: Unit entered failed state.

Nov 05 12:40:37 queenbee-2 systemd[1]: test.service: Failed with result 'exit-code'.


# docker logs


DEBU[0011] Calling POST /containers/create              

INFO[0011] POST /v1.19/containers/create                

ERRO[0011] Handler for POST /containers/create returned error: No such image: busybox (tag: latest) 

ERRO[0011] HTTP Error                                    err=No such image: busybox (tag: latest) statusCode=404

DEBU[0011] Calling POST /images/create                  

INFO[0011] POST /v1.19/images/create?fromImage=busybox&tag=latest 

DEBU[0011] pulling image from host "docker.io" with remote name "library/busybox" 

DEBU[0011] pinging registry endpoint https://index.docker.io/v1/ 

DEBU[0011] attempting v1 ping for registry endpoint https://index.docker.io/v1/ 

INFO[0012] Trust graph fetch failed: error unmarshalling content: invalid character '<' looking for beginning of value 

DEBU[0012] Fetched 0 base graphs at 2015-11-05 13:00:41.583794525 -0500 EST 

DEBU[0012] pulling v2 repository with local name "busybox" 

DEBU[0012] pinging registry endpoint https://registry-1.docker.io/v2/ 

DEBU[0012] attempting v2 ping for registry endpoint https://registry-1.docker.io/v2/ 

DEBU[0012] hostDir: /etc/docker/certs.d/registry-1.docker.io 

DEBU[0012] Getting authorization for library/busybox [pull] 

DEBU[0012] Pulling tag from V2 registry: "latest"       

DEBU[0012] Getting bearer token with map[realm:https://auth.docker.io/token service:registry.docker.io] for vasubalakrishnan 

DEBU[0012] hostDir: /etc/docker/certs.d/auth.docker.io  

DEBU[0012] hostDir: /etc/docker/certs.d/registry-1.docker.io 

DEBU[0013] provided manifest reference "latest" is not a digest: invalid checksum digest format 

DEBU[0013] Key check result: no graph                   

DEBU[0013] pulling blob "sha256:55dc925c23d1ed82551fd018c27ac3ee731377b6bad3963a2a4e76e753d70e57" to V1 img d1592a710ac323612bd786fa8ac20727c58d8a67847e5a65177c594f43919498 

DEBU[0013] Using cached token for vasubalakrishnan      

DEBU[0013] hostDir: /etc/docker/certs.d/registry-1.docker.io 

DEBU[0013] pulling blob "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to V1 img 17583c7dd0dae6244203b8029733bdb7d17fccbb2b5d93e2b24cf48b8bfd06e2 

DEBU[0013] Using cached token for vasubalakrishnan      

DEBU[0013] hostDir: /etc/docker/certs.d/registry-1.docker.io 

DEBU[0013] hostDir: /etc/docker/certs.d/dseasb33srnrn.cloudfront.net 

DEBU[0013] hostDir: /etc/docker/certs.d/dseasb33srnrn.cloudfront.net 

ERRO[0013] Error from V2 registry: Authentication is required. 

DEBU[0013] image does not exist on v2 registry, falling back to v1 

DEBU[0013] pulling v1 repository with local name "busybox" 

DEBU[0013] hostDir: /etc/docker/certs.d/index.docker.io 

DEBU[0014] Retrieving the tag list                      

DEBU[0014] hostDir: /etc/docker/certs.d/registry-1.docker.io 

DEBU[0015] Registering tags                             

DEBU[0015] hostDir: /etc/docker/certs.d/registry-1.docker.io 

DEBU[0015] Ancestry: [17583c7dd0dae6244203b8029733bdb7d17fccbb2b5d93e2b24cf48b8bfd06e2 d1592a710ac323612bd786fa8ac20727c58d8a67847e5a65177c594f43919498] 

DEBU[0015] hostDir: /etc/docker/certs.d/registry-1.docker.io 

DEBU[0016] hostDir: /etc/docker/certs.d/registry-1.docker.io 

DEBU[0016] hostDir: /etc/docker/certs.d/dseasb33srnrn.cloudfront.net

 

 

# /etc/os-release

NAME=CoreOS

ID=coreos

VERSION=766.4.0

VERSION_ID=766.4.0

BUILD_ID=

PRETTY_NAME="CoreOS 766.4.0"

ANSI_COLOR="1;32"


# docker version

Client version: 1.7.1

Client API version: 1.19

Go version (client): go1.4.2

Git commit (client): df2f73d-dirty

OS/Arch (client): linux/amd64

Server version: 1.7.1

Server API version: 1.19

Go version (server): go1.4.2

Git commit (server): df2f73d-dirty

OS/Arch (server): linux/amd64 

kyle....@coreos.com

unread,
Nov 5, 2015, 5:42:54 PM11/5/15
to CoreOS User
Vasudevan,

Nov 05 12:40:37 queenbee-2 docker[17851]: 17583c7dd0da: Error pulling image (latest) from busybox, Server error: Status 401 while fetching image layer (d1592a710ac323612bd786fa8ac20727c58d8a67847e5a65177c594f43919498)

This seem to indicate that your dockerhub account credentials are invalid or inactive. Can you login again with `docker login`? and make sure that the resulting .dockercfg is in the /home folder of the core user. 


--kyle

Ihor Dvoretskyi

unread,
Nov 8, 2015, 2:27:38 PM11/8/15
to CoreOS User
Vasudevan,

Could you please explain, what registry do you use? Private or public (dockerhub)? As Kyle mentioned below, the error that you've provided here describes that you have an authentication issue.

I may recommend you to re-check the registry credentials is you're using the private registry.

Vasudevan Balakrishnan

unread,
Nov 9, 2015, 4:38:57 PM11/9/15
to CoreOS User
I'm running the cluster behind our corporate firewall. Turned out the proxy was blocking the docker requests. We had to configure proxy (Bluecoat) to let the docker traffic through (using static ip(s)). I would've preferred to have used DHCP instead of assigning static ip. Is there any recommended / best practices available for this setup?


Vasu

anton....@coreos.com

unread,
Nov 10, 2015, 3:19:06 AM11/10/15
to CoreOS User
Dear Vasudevan,

You can use "$private_ipv4" substitute variables with VMware hypervisors. These variables are available through VMware guestinfo (https://github.com/coreos/coreos-cloudinit/blob/master/Documentation/vmware-guestinfo.md) and work only in CoreOS release 801.0.0 and greater so I can suggest you to use latest beta. Please follow this example to know how you can use this feature: https://github.com/endocode/coreos-docs/blob/kayrus/vmware/os/booting-on-vmware.md#vmware-backdoor. 

Let me know if you have any further questions.

Regards,
Anton

Vasudevan Balakrishnan

unread,
Nov 10, 2015, 10:58:40 AM11/10/15
to CoreOS User
Pardon my ignorance. What is $private_ipv4? How is going to let the docker traffic past corporate proxy? 

To give you some background, I tried setting HTTP_PROXY variable with credentials for docker. Apparently our proxy server (Bluecoat) ignores these credentials for CONNECT traffic. So the only alternative, I'm told by my network team is to assign static IP(s) to each of node. It works fine then, I was trying to see if I can avoid that and/or if there is any alternative.

Vasu

anton....@coreos.com

unread,
Nov 10, 2015, 12:41:07 PM11/10/15
to CoreOS User
Dear Vasudevan,

My last answer related to configuring CoreOS node to use DHCP and how you can configure your CoreOS apps to use IP which was provided by DHCP server: https://coreos.com/os/docs/latest/cloud-config.html#etcd2

Unfortunately I don't know how exactly your internal network works but probably my info will help you.

Regards,
Anton

Jerry Hwang

unread,
Nov 10, 2015, 8:01:39 PM11/10/15
to CoreOS User
Vasudevan,

It seems either you would need to use a static ip or DHCP reservation (for the consistent ip assigned).
Or I would talk to the bluecoat admin on how the proxy exactly does about your docker pull traffic.
Does it run any sort of contents type filtering or any kind of deep packet inspection for https traffic? (for the DPI case, typically to import & trust the certificate of the bluecoat (proxy) into your coreos may help).

Regards,
Jerry
Reply all
Reply to author
Forward
0 new messages