Configure ah-builtin-python-module Idle Instance Count

[Ivan Bulanov]

Our app uses ah-builtin-python-module for backup. Backups are configured to execute quickly. We noticed that there are several (6) idle instances allocated to ah-builtin-python-module. We believe that the high instance count is caused by automatic scaling. The algorithm expects the next spike of activity caused by regular backup jobs and leaves some instances idle to reduce initial request latency. In the case of backups we don't really care about request latency.

The question is: can we configure ah-builtin-python-module to spin down instances quickly like we do with the default service?

Backup queue config:

- name: backup
  mode: push
  rate: 100/s
  bucket_size: 20
  target: ah-builtin-python-bundle

Default service config:
automatic_scaling:
  max_concurrent_requests: 32
  max_idle_instances: 1
  min_idle_instances: 0
  max_pending_latency: "100ms"
  min_pending_latency: "30ms"


Ivan

[George (Cloud Platform Support)]

Hello Ivan, 

More detail is needed to properly start an investigation in this issue, especially in what concerns the ah-builtin-python-module. Do you mean ah-builtin-python-bundle? 

How did you determine that the spinning up of new instances is caused by the backup jobs? Why would the algorithm leave some instances running for such an extended span of time, between successive backups? 

It would help a lot if we may become able to reproduce the issue locally, with a significant version of your app and your exact procedure for backup. To maintain privacy and data security, you can edit out sensitive details and send information by private email, using the drop-down menu of the reply button, at the top right of the window. 

[George (Cloud Platform Support)]

Hello Ivan, 

In your situation, you may try to play around with various settings that may influence app performance and scaling, as described on the "Managing App Resources" page. A special attention may be dedicated to the pending latency paragraph. By trying these settings, you will find out in your particular case, which ones result in a level of instance scaling that is acceptable. 

[Ivan Bulanov]

George,

Thanks for the prompt. I have read the documentation. However, I'm under impression that settings in app.yaml affect only the corresponding service and not the ah-builtin-python-bundle. The bundle seems to be a separate service. Thus it is not clear how to configure it. Do you have any suggestions?

[George (Cloud Platform Support)]

It is still the queue.xml and appengine-web.xml configuration files that can influence your backup jobs. Of particular interest is the "scaling elements" paragraph in the latter, with the <idle-timeout> basic scaling element. 

[Ivan Bulanov]

Scaling parameters in app.yaml don't seem to affect ah-builtin-python-bundle at all. I configured default module of the app the following way:

basic_scaling:
  max_instances: 10
  idle_timeout: 10s

In theory it should have brought the number of instances of the default service to 0 after 10 to 40 seconds of receiving the last request. In practice Google Cloud Console still shows some idle instances assigned to version ah-builtin-python-bundle. You can also see on the screenshot that the version is configured with autoscaled mode. See Instances (autoscaled). Moreover the console also incorrectly shows the graph of instances. It shows 0 total instances whereas there should be 6.

I'm bringing this issue up because it is affecting the monitoring for the app. While there could be 2-3 instances that are actually serving app requests there are 6-7 idle instances associated to ah-builtin-python-bundle that just sit there. It is not clear if they are counted in the billing. Our app has an alert that gets triggered if there is a big number of instances for more that an hour but the alert is currently triggered quite frequently due to the big and unpredictable number of idle instances. Ideally I'd like to configure ah-builtin-python-bundle to bring down instances ASAP. What can we do about it?

[George (Cloud Platform Support)]

Hi Ivan, 

On your screen capture there are no other instances for that time period except for the auto-scaled ones. The way to control these can be read from the "Change auto scaling performance settings" paragraph on the "Managing App Resources" documentation page. Relevant parameters are Max Idle Instances and Min Pending Latency.