AppEngie OUTAGE

[Paolo Conte]

Hello,

suddently most requests return error 500, log shows "The request failed because the instance could not start successfully"
Project is in europe-west

Anyone else?

[mark negus]

Yes, we are having problems for the last hour in europe-west. Mainly 500s. Looks like the auto-scaling is having issues.

[mark negus]

https://status.cloud.google.com/?_ga=2.187839338.-379351283.1560761695

[Paolo Conte]

Thanks Mark. I'm trying to use manual scaling to see if it improves

[mark negus]

Hi Paolo,

Let me know how you get on. We were manual scaling for ages as we had terrible issues with auto-scaling a while back. Once it randomly launched 40 instances and burnt through our daily budget in a few hours. Its been ok though this last year.

[Paolo Conte]

Manual scaling seems to also have trouble starting instances. For now I kept autoscaling with a high minimum instances number and the error rate is almost zero at the moment 

[mark negus]

This app version won't even start at all. What else did you do?

java8, F4

automatic_scaling:

  min_idle_instances: 3

  max_idle_instances: automatic

  min_pending_latency: automatic

  max_pending_latency: automatic

  max_instances: 4

[Paolo Conte]

My settings are below. By looking at the logs of /_ah/warmup to me it seems that some instances start and some don't, so you need some luck to get a good amount of working instances

<instance-class>F1</instance-class>

<automatic-scaling>
    <min-instances>35</min-instances>
    <max-instances>35</max-instances>
    <min-idle-instances>3</min-idle-instances>
    <max-idle-instances>5</max-idle-instances>
    <max-pending-latency>500ms</max-pending-latency>
    <max-concurrent-requests>5</max-concurrent-requests>               
    <target-throughput-utilization>0.9</target-throughput-utilization>
    <target-cpu-utilization>0.8</target-cpu-utilization>
</automatic-scaling>

[mark negus]

Thanks, thats useful.

Yes I see a few of these  A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. (Error code 121)  
during warmup calls but just can't get enough started.

Just had 15mins without an error and now they are back again. Sigh.

[Paolo Conte]

Looks normal now

[Doug Stoddart]

yes major problems on EU servers for 49 mins already..

[Doug Stoddart]

Still no acknowledgement of this incident from Google

https://status.cloud.google.com/?_ga=2.187839338.-379351283.1560761695

disgraceful behaviour.

[Doug Stoddart]

yes, same here for nearly 1hr

On Tuesday, October 8, 2019 at 9:48:59 AM UTC+2, Paolo Conte wrote:

[Rob Curtis]

Hi, In the past I've found it's better to report downtime issues on their issue tracker. https://issuetracker.google.com/issues/

If you're not already, also subscribe to the downtime notify group https://groups.google.com/forum/#!forum/google-appengine-downtime-notify 

Good luck

[Carl Emmoth]

This affected our customers the whole day. Is it possible to get compensation?

[Carl Emmoth]

I've filed a bug 

Den tisdag 8 oktober 2019 kl. 09:48:59 UTC+2 skrev Paolo Conte:

[Doug Stoddart]

thanks, Rob,

good resources.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/30dedf4b-384c-4332-a903-27f6b6572a8a%40googlegroups.com.

--

Doug Stoddart CEO E: do...@bikerentalmanager.com  W: bikerentalmanager.com

[Doug Stoddart]

NICE

it took them 8 hours to approve my posts about the incident, which lasted 3 hours.

thanks Google.  Really helpful.

On Tuesday, October 8, 2019 at 4:47:42 PM UTC+2, Doug Stoddart wrote: