Persistent instance and out of memory with automatic scaling and no traffic

[vvv vvv]

I am on the AppEngine Standard Python 3.7 environment. I have set up a cron job to execute every two hours. This cron job creates a bunch of threads, they execute some task and finish after some seconds. My web service receives no traffic other than this currently.

I am attaching the number of instances graph, and the memory usage. In the instances graph the blue line shows that an instance runs for 15 minutes every two hours, which is what one would expect with automatic scaling and scaling to 0 instances. The green line shows that there are two instances running persistently, with the only exception at 23:00 when one instance gets killed because it goes out of memory, as one can see from the memory graph attached.

My question is, why is there one instance running all the time? Who has created it? I'm trying to sort out my memory issue. Thanks a lot.
P.S attached utilization graph to rule out the possibility that my instance is still alive after 15 minutes.

[Rahul Ravindran]

The instance might stay alive after it's been idle for 15 minutes, but you won't be billed for it. Billing is based on 15 min blocks as long as there is at least one active requet in the 15 min block.

We kill clones lazily to prevent excessive cold starts.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/44af4256-0b24-46fd-b971-515b1944c3de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Amit (Google Cloud Support)]

Hello

According to this document, a service with manual scaling server uses resident instances that continuously run the specified number of instances irrespective of the load level. A requests can run for up to 24 hours. A manually-scaled instance can choose to handle /_ah/start and execute a program or script for many hours without returning an HTTP response code.

For auto scaling method, I would recommend to check the min_idle_instances setting in app.yaml file which is set to ensure the number of instances to be kept running and ready to serve traffic.

You can check your instance list by going to the versions page of your console. To do that, login to your console, then  click on hamburger sign on your upper left corner and select ‘Versions’ from ‘App Engine’ section. You will find ‘Instances’ column which indicates how much instances are currently active for that application version. By clicking on this link will show you the instances are running now. You can check this for your running instances for the app version.   

I also found this group thread discussed about this instance graph specifically. If required you can also check this for more clarification.

[vvv vvv]

Thanks for the answers. In my app.yaml I have "automatic_scaling: min_idle_instances: automatic "

I realized an older version of my service was created and running, so I deleted it. What is not clear to me is, after 15 minutes of inactivity, when my instance becomes idle, why isn't the graph that shows the created instances goes down to 0? Isn't inactivity after 15 mins for auto scaling the same as "scale to 0 instances" ? The fact that it doesn't get shut down is the reason it exceeds its memory limit, since the instance keeps using more and more memory the more it runs.

[Amit (Google Cloud Support)]

It seems like an expected behavior. As this document mentioned, for min_idle_instances , the number of instances to be kept running and ready to serve traffic. As you set it to automatic, at least one instance is active to ensure immediate response for sudden traffic spikes. So your graph shows minimum number of active instance which is one.