App Engine periodic failure when restarting, failure to download docker image

[Neill Shazly]

Dear App Engine Users,

We have several running apps on app engine. When the instance restarts it will sometimes encounter a failure downloading the docker image and end up in a continuous restart loop. When this happens only a redeploy of a newly built image will resolve the situation. This usually get s the app back up and running.

When inspecting the logs of the instance that is restarting I find the following error during the instance startup script:

     error pulling image configuration: unknown blob

When attempting to pull the exact image using the gcloud docker tool I get the same error:

gcloud docker -- pull us.gcr.io/my-project/my-image@sha256:abcdef123acdef1234abcdef123acdef1234abcdef123acdef1234abcdef123acde
sha256:abcdef123acdef1234abcdef123acdef1234abcdef123acdef1234abcdef123acde: Pulling from my-project/my-image
a21f4015913a: Already exists
2ace7c4f1f17: Pulling fs layer
f5d3977f5836: Pulling fs layer
02ef7894239f: Pulling fs layer
7a092c3d225b: Waiting
06a9dc0e1652: Waiting
26dc6a9a4f3b: Waiting
06d8ced85f62: Waiting
40d758b3622f: Waiting
b1bab8396bf8: Waiting
e417ff3a233c: Waiting
142d66e5e367: Waiting
c5060d8f926c: Waiting
492db658d3f3: Waiting
error pulling image configuration: unknown blob

The image does exist in the google container registry. Running the command above recreates the same error. It has become corrupted since the original deploy.

1. What is causing these images to get corrupted, we have not manipulated the stored image in any way ?

2. How can we prevent this from happening ?

Much thanks,

Neill Shazly

[Katayoon (Cloud Platform Support)]

For specific technical questions, I recommend that you ask your redacted question on Stack Overflow using one of the tags monitored by our community technical support team, since there would be more visibility on it.