backends emergency

[radzish]

Hi,

I am experiencing problem with backends right now.

One of  backend is not updating.

Here what I see in the logs:

...

99% Will check again in 60 seconds.

99% Will check again in 60 seconds.

99% Will check again in 60 seconds.

99% Will check again in 60 seconds.

I noticed that it appeared in the list of backends but when I press "Start" button first I was forwarded to generic error message screen, now red message "The backend could not be started because there are no versions ready." appears. I noticed that though I am requesting backend of class B8, it is listed as B2 in the list I tried several times. I disabled and reenabled application, but it does not help, other backends starting up normally. The same backend is updated fine on other application. Quota is OK and I am not exceeding neither backends number limit nor total instances number limit. 

Also I noticed that several static instances of other backend are not started also and are not starting after updating backend 

Additional information:

I have 3 backends:

1) 9 static instances, B1 (infinite loop)

2) 1 static instance, B8

3) 1 static instance, B8

One of B8 instance is not starting at all, also 3 instances of 1) are not running.

I have just noticed that on other application instances of 1) are stuck and I do not see what is going on with them.

Could someone please assist me asap as we need this to be up and running by the start of 4:00am UTC. If someone from GAE team could contact me directly I would provide app id and all other relevant information.

thanks in advance,

Alex 

[radzish]

A little update ..

Seems like B8 instance that was not updating updated after ~30 minutes of waiting. But some of small B1 instances of 1) backend are not running. This is not acceptable for me as each instance is linked to different task.

I feel like this somehow is caused by GAE system to be overloaded or something. Status page says everything is "Normal". Can someone provide more details on what is going on.