GCR/Google Storage timeout

[Chandan Pasunoori]

In the middle of kubernetes cluster upgrade, pods failed to start because image pull causing because of GCR timeout, by log I can see that it's because of google storage timeout

Failed to pull image "gcr.io/<project>/<project>-<app>:<tag>": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/<project>/<project>-<app>/manifests/<tag>: Get https://gcr.io/v2/token?account=_token&scope=repository%3A<project>%2F<project>-<app>%3Apull&service=gcr.io: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Failed to pull image "gcr.io/<project>/<project>-<app>:<tag>": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Failed Apr 26, 2020, 12:17:05 AM Apr 26, 2020, 12:17:05 AM 1

Error: ErrImagePull

Failed create pod sandbox: rpc error: code = Unknown desc = error pulling image configuration: Get https://storage.googleapis.com/asia.artifacts.google-containers.appspot.com/containers/images/sha256:<hash>?access_token=<token>: dial tcp 172.217.160.176:443: i/o timeout

[Adebisi Ibirogba]

Thanks for reporting this issue.

I see that you noticing the a failed to pull image. The " ErrImagePull" troubleshooting steps is documented here. 

However, in order to assist you properly, I would suggest you file it in the issue tracker