KubernetesPodOperator: Failed to apply default image tag docker pull us.gcr.io...couldn't parse image reference...invalid reference format: InvalidImageName

[Gaurav Gupta]

Hi,

I am trying to run a docker image from private GCR using KubernetesPodOperator in Cloud Composer, but getting the following error:

ERROR: Pod launching failed : Pod took too long to start 

At first I tried increasing the "startup_timeout_seconds" but it didn't help.

Looking at the Composer created GKE cluster logs gave me the following error:

Failed to apply default image tag "docker pull us.gcr.io/<my-proj-name>/<myimage-name>:latest": couldn't parse image reference "docker pull us.gcr.io/s<my-proj-name>/<myimage-name>:latest": invalid reference format: InvalidImageName

I tried pulling the docker image on my local machine and it worked fine, not sure where is the issue.

For reference: my CircleCI config.yml contains following

- run: echo ${GOOGLE_AUTH} > ${HOME}/gcp-key.json
- run: docker build --rm=false -t us.gcr.io/${GCP_PROJECT}/${IMAGE_NAME}:latest .
- run: gcloud auth activate-service-account --key-file ${HOME}/gcp-key.json
- run: gcloud --quiet config set project ${GCP_PROJECT}
- run: gcloud docker -- push us.gcr.io/${GCP_PROJECT}/${IMAGE_NAME}:latest

Could anyone please guide me?

[Alberto Vicente]

Did you validate the following troubleshooting steps? 

https://cloud.google.com/kubernetes-engine/docs/troubleshooting

https://kukulinski.com/10-most-common-reasons-kubernetes-deployments-fail-part-1/

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/aa0e993f-3c68-4197-9bb9-0e5627f311ee%40googlegroups.com.

--

Alberto Vicente |  Tech Director @ Big Data Studio GLOBANT AR: +54 11 4109 1700 ext 16054 US: +1 877 215 5230 ext. 16054

The information contained in this e-mail may be confidential. It has been sent for the sole use of the intended recipient(s). If the reader of this message is not an intended recipient, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution or copying of this communication, or any of its contents, is strictly prohibited. If you have received it by mistake please let us know by e-mail immediately and delete it from your system. Many thanks.

 

La información contenida en este mensaje puede ser confidencial. Ha sido enviada para el uso exclusivo del destinatario(s) previsto. Si el lector de este mensaje no fuera el destinatario previsto, por el presente queda Ud. notificado que cualquier lectura, uso, publicación, diseminación, distribución o copiado de esta comunicación o su contenido está estrictamente prohibido. En caso de que Ud. hubiera recibido este mensaje por error le agradeceremos notificarnos por e-mail inmediatamente y eliminarlo de su sistema. Muchas gracias.

[Gaurav Gupta]

Hi,

This link https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod tells me that

"All pods in a cluster will have read access to images in this registry.

The kubelet will authenticate to GCR using the instance’s Google service account. The service account on the instance will have a https://www.googleapis.com/auth/devstorage.read_only, so it can pull from the project’s GCR, but not push"

which means the pod should be able to pull image from GCR. FYI, I am using a service account to provision my composer env and it has sufficient permission to read from GCS bucket. 

Also, I did the following steps to add secret :

Step 1: gcloud container clusters get-credentials <cluster_name>

Step 2: kubectl create secret generic gc-storage-rw-key --from-file=key.json=<path_to_serv_accnt_key>

Step 3: Add secret to my DAG in following way:

secret_file = secret.Secret(

    deploy_type='volume',

    deploy_target='/tmp/secrets/google',

    secret='gc-storage-rw-key',

    key='<path of serv acct key file>.json')

Step 4: Refer it as secrets=[secret_file]  inside KubernetesPodOperator operator in DAG

I have added image_pull_policy='Always' in my DAG as well but not working...

Can anyone help me on this?

[Marcel S. Gongora]

I don't think your issue is permissions, judging by the message, you are not using the right docker image. But if, in any case your issue is related to permissions, you should take a look what's the service account used for composer and then assign read permission to gcs corresponding to gcr