resident instances are not handling requests

[Miroslav Genov]

Hello, 

I'm encountering strange issue with resident instances. It seems that instance is up and running, but the GAE load balancer does not pass requests to it. Is this a normal behaviour or some kind of bug? 

Here is a screenshot from my dashboard:

Thanks in advance. 

[Patrice (Cloud Platform Support)]

Hi Miroslav,

Normally, Resident instances are for modules, so unless your requests go straight to the module, the behavior you are experiencing is normal.

Do you have a module in your application?

Cheers!

[Miroslav Genov]

This is an old app (maybe 3-4 years), so we have not migrated our stuff to use modules. In next 2 days, we will give it try to see where the things gonna change. 

Thanks for the fast feedback. 

[Patrice (Cloud Platform Support)]

Hi again Miroslav,

Happy that you like the speed, we've been working on being better on that :).

In any case, it turns out even backends, seems like they need to be explicitly called to serve, so that may be what's happening here.

Cheers!

[Miroslav Genov]

Hello again, 

We tried some scenario, and here are the results. 

<automatic-scaling>
    <min-idle-instances>3</min-idle-instances>
    <!-- ‘automatic’ is the default value. -->
    <max-idle-instances>10</max-idle-instances>
    <!-- ‘automatic’ is the default value. -->
    <min-pending-latency>30ms</min-pending-latency>
    <max-pending-latency>automatic</max-pending-latency>
    <max-concurrent-requests>100</max-concurrent-requests>
  </automatic-scaling>

<automatic-scaling>
    <min-idle-instances>1</min-idle-instances>
    <!-- ‘automatic’ is the default value. -->
    <max-idle-instances>10</max-idle-instances>
    <!-- ‘automatic’ is the default value. -->
    <min-pending-latency>30ms</min-pending-latency>
    <max-pending-latency>automatic</max-pending-latency>
    <max-concurrent-requests>50</max-concurrent-requests>
  </automatic-scaling>

<automatic-scaling>
    <min-idle-instances>1</min-idle-instances>
    <!-- ‘automatic’ is the default value. -->
    <max-idle-instances>10</max-idle-instances>
    <!-- ‘automatic’ is the default value. -->
    <min-pending-latency>30ms</min-pending-latency>
    <max-pending-latency>automatic</max-pending-latency>
    <max-concurrent-requests>10</max-concurrent-requests>
  </automatic-scaling>

As you can see, there is no difference where module app is used or single app. Can you give us some light on this ? 

[Patrice (Cloud Platform Support)]

Hi Miroslav. 

I'm unclear on what you mean by this. 

Normally modules don't just serve requests, unless they get called explicitely. Scaling will change (depending on where you put that scaling) how the instances for the regular traffic serves. 

Since you need to specifically call modules, traffic to them will not just go there, you need to call it.

[Kaan Soral]

In general, my instance charts look similar

It's also important to note that, in general, my general instances are >> my billed instances, so likely there are instances there to serve my requests, yet I don't pay for them, that's nice :)

Resident instances are there as he specified min_instances, so it shouldn't be related to modules and backends and stuff like that, he is just using everything as default as far as I understand

Miroslav, your charts look regular to me, it might be a good idea to set min and max instances to 1, that way you will only pay for 1 resident instance and the extras will not be billed (they will only be billed if you are actually using them, so this will likely not affect your performance significantly but reduce your costs instead)

I think the main issue is mostly confusion and the black box nature of the instance scheduler

[troberti]

The interaction between resident instances and the instance settings are pretty complex. The scheduler tries to keep the resident idle, so what you are seeing is correct. If non-resident instances are busy, the request will be routed to the resident one. This also affects the meaning of the pending latency settings (which are confusing by themselves). The docs mention this in a note:

Note: If you set a minimum number of idle instances, the pending latency setting will have less effect on your application's performance. Because App Engine keeps idle instances in reserve, it is unlikely that requests will enter the pending queue except in exceptionally high load spikes. You will need to test your application and expected traffic volume to determine the ideal number of instances to keep in reserve.

As for max-idle-instances, I would put that to 1, as Kaan suggested. For us this cut our instance bill in half (!), with no noticeable effect on performance. I have no idea why this is not the default.

Then you also have max_concurrent_requests, which also affect performance, at least on Python. I would recommend that you try a low number, like 2. This might seem counterintuitive, but running more requests concurrently can actually increases response time of all of them. This is something you really have to try and see what fits your app best.

All things considered, I think the application settings are very complex and it is very easy to shoot yourself in the foot with them and make your app perform badly. This is bad for the user, and as a consequence, for App Engine itself. I rather see one default setting that processes requests as fast as possible, which is the ultimate goal right?

[Patrice (Cloud Platform Support)]

Kaan, thanks for the clarification, I blanked on that part of resident. Since he mentioned he was moving to modules, I focused only on that. In any case, to explain why your shown instances are greater than your billed, it's because we don't charge for idle, so it's to be expected.

But yeah, your explanation and troberti are EXACTLY right. The scheduler is a bit "black-boxy", so you need to test it and find what is right for your app.

Everything said is definitely on par with the behavior of the scheduler. I would drop max_concurrent_requests a bit, make sure you don't have too many max_idle_instances.

All in all, this is very dependent on the nature of your app, your users, and your traffic, so very hard to help you with this, except by suggesting tips like the above ones, and suggesting you to test and explore the different parameters.

Cheers