Frequent restart of instances - App Engine Standard PHP 7.3

[Magnus Kåre Skjeggedal Andersen]

Hi

We are having some issues with frequent restarts of App Engine instances, typically every 10-30 minutes. Sometimes the instances can run for some hours too. There are no server errors recorded that I can see in the console, and no signs of memory leaks on the graphs in the instances-dashboard.

Is this normal behavior or is something wrong? Any suggestions on how I can debug this? We have set minimum instances to 3 to avoid users getting loading requests, as a work-around.

Thanks for any ideas.

-Magnus

[Sebastien Dupere]

This is not the norm, my GAE Flexible projects have instances that can run for days easily.

Quick Questions to try and help:

* What is your app engine setup (Language/server/etc)?

* What are your app engine machines type?

** Or configuration if using AppEngine Flexible.

* Do you have a liveness_check and/or readiness_check setup?

** Could that fail?

* AppEngine or AppEngine Flexible?

* Which logs do you check to say there's no errors?

** Do you have Cloud Logging enabled in your application to have the app log available to you?

*** This usually can pinpoint if you at least raised an exception fairly quickly.

Maybe giving us your app.yaml might give us a better look into your project. Obviously, if you do, remove the information you find private.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/26be3ba2-00e4-4def-822f-7ebc9763422e%40googlegroups.com.

--

Thank you,

Sebastien Dupere

[Magnus Kåre Skjeggedal Andersen]

Hi and thanks for your reply

We run App Engine Standard Second Gen PHP 7.3. Machine type is F1. We dont have any liveness_check or readiness_check, it looks like that only applies to flexible.

I have checked the logs when you go to App engine -> Services and then select debug or logs under "Tools". 

Looking into Stackdriver and Cloud Debugger now.

Here is the app.yaml:

	runtime: php73
	entrypoint: serve public_html/index.php
	instance_class: F1
	automatic_scaling:
	min_instances: 3
	inbound_services:
	- warmup

- Magnus

onsdag 29. april 2020 20.58.26 UTC+2 skrev Sebastien Dupere følgende:

This is not the norm, my GAE Flexible projects have instances that can run for days easily.

Quick Questions to try and help:

* What is your app engine setup (Language/server/etc)?

* What are your app engine machines type?

** Or configuration if using AppEngine Flexible.

* Do you have a liveness_check and/or readiness_check setup?

** Could that fail?

* AppEngine or AppEngine Flexible?

* Which logs do you check to say there's no errors?

** Do you have Cloud Logging enabled in your application to have the app log available to you?

*** This usually can pinpoint if you at least raised an exception fairly quickly.

Maybe giving us your app.yaml might give us a better look into your project. Obviously, if you do, remove the information you find private.

On Wed, Apr 29, 2020 at 11:41 AM Magnus Kåre Skjeggedal Andersen <m.k.s...@gmail.com> wrote:

Hi

We are having some issues with frequent restarts of App Engine instances, typically every 10-30 minutes. Sometimes the instances can run for some hours too. There are no server errors recorded that I can see in the console, and no signs of memory leaks on the graphs in the instances-dashboard.

Is this normal behavior or is something wrong? Any suggestions on how I can debug this? We have set minimum instances to 3 to avoid users getting loading requests, as a work-around.

Thanks for any ideas.

-Magnus

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/26be3ba2-00e4-4def-822f-7ebc9763422e%40googlegroups.com.

[Sebastien Dupere]

Hi Magnus,

I do not have much experience with this setup as most of my projects are in Python, but I will give you a last suggestion.

When you look at the logs in tools, could you look at the following logs (if they exist, also they are not shown by default):

* app

* nginx.X (X can be warning, error, etc...There's a bunch of them)

* vm.X (Same as above)

* activity_log

* uptime_check

Note:

* You have to go in log BASIC mode to see the list, otherwise you have to write your own filter and know what log to look for (unlikely that you have that list handy hehe).

* I took that list based on my project, so you might have different logs, but it can't hurt to see what is available.

** I think by default you only see:

*** stderr

*** nginx.request

Sorry I can't provide more help :(

Hopefully others will come with better suggestions.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/0fde4753-58d5-462a-b236-6bc3f379cc56%40googlegroups.com.

[Katayoon (Cloud Platform Support)]

Hi Magnus,

I see you are using instance class F1 that may not be fit to your work load. You may look into your App Engine dashboard console (App Engine>Dashboard) and check the "CPU Utilization" and "Memory Usage" for the serving version. If you see some spikes correlating with your instance restarts, you may upgrade your instance class to resolve this issue. If the issue still persists, I recommend to create a private report in the Issue tracker and provide us with your project id, the service name and the version number, so that we would be able to look into it and assist you to resolve the issue efficiently. 

[Magnus Kåre Skjeggedal Andersen]

Hi

Thanks for your reply. I have tried with F2-instances too, with the same outcome, so I reduced it to F1 and 3 instances all the time.

Will report it in the issue tracker now.

-Magnus

[DFB]

Hi Magnus,

Did it get resolved? I'm facing the same issue. New instances are restarted and the previous one shutdown for no obvious reason. 

[Katayoon (Cloud Platform Support)]

Hi,

As previously recommended, you may create a private report in the Issue tracker and provide us with your project id, the service name and the version number, so that we would be able to look into it and assist you to resolve the issue efficiently.