I am not able to quantify what are App Engine Frontend Instance Hours

[Naresh Pokuri]

I have started my GAE app with Auto-scaling having min-idle-instances 3, each with 1GB RAM and 2.4GHz processor(i.e F4_1G). And I have a cron job which runs on every 5 minutes. With this setup keeping application idle for one day should equal to 72 instance hours. But I see that it already reached 428 instance hours. So, I am clueless here how GAE calculates instance hours, with this alone I can keep my budget in control. Can someone help me in this instance hours

[Anastasios Hatzis]

There might be more than one reason for this:

From the documentation:

Important: When you are billed for instance hours, you will not see any instance classes in your billing line items. Instead, you will see the appropriate multiple of instance hours. For example, if you use an F4 instance for one hour, you do not see "F4" listed, but you will see billing for four instance hours at the F1 rate.

So in your case with F4_1G, which costs 0.30$ per hour, rather than 0.05$ per hour of a B1 instance, this is factor 6. The billings always show instance hours as B1, so 72 hours F4_1G translates to 432 hours B1. The same applies to the billing dashboard.

However, there can indeed be more instances with your setup, than the 3 you want, hence the "min-idle-instances":

From the docs:

Auto scaling modules use dynamic instances - but if you specify a number, N, of minimum idle instances, the first N instances will be resident, and additional dynamic instances will be created as necessary.

For example, if your cron job handler has an error and exponentially adds tasks into the queue. Or if tasks run much longer than expected, so additional instances are needed to handle the overlapping tasks. With your setup it is possible that you end up with more than three F4_1G instances during peeks, but never will fall below three. If I understand idle instances correctly, that means that at least three instances will be running that do nothing. If your cron-job starts, a fourth instance would be started, so there will always be three idle instances, plus the instances that are processing the cron-jobs.

You can limit this, either in auto-scaling (e.g. max_idle_instances), or kind of ultimately, you can set a daily budget. I recommend to check again the dashboard from your screenshot, just to be certain what the reason is for your instance hours. There is the "Summary" button on top. Toggle it to "Instances" diagram and switch the period to 1 day, for example, just to see if there were any peeks with more than 3 instances.

As far as I understand, you have no user-facing features in your app (or this module), but only cron-jobs? Do you expect to have always same load for your app or module? Maybe it doesn't matter if your cron-jobs have to wait a few additional seconds until a new instance is spawned (just in case). If the answers are all yes, you could consider basic-scaling with max-instances set to 3.  If you set the idle_timeout to something longer than your 5 minute cron-job schedule (e.g. 10m), the setup could work fine for your use-case.

[Naresh Pokuri]

Thanks Anastasios Hatzis for the detailed explanation. That threw some light on my understanding of Instance Hours. You said,

If I understand idle instances correctly, that means that at least three instances will be running that do nothing. If your cron-job starts, a fourth instance would be started, so there will always be three idle instances, plus the instances that are processing the cron-jobs.

Does that mean that min-idle-instances are always idle ? And won't take up any requests to process ? Why will a fourth instance be started when we already have three idle instances ? What does min-idle-instances signify ?

Now, we have enabled max-idle-instances to 3 as well. So, the instance hours with this setup {min-idle: 3, max-idle: 3}, should come up to 24 * 3 * 6 = 432  F1 Instance Hours, And I should not be charged for any dynamic instances that will spin up beyond 3. Correct me if I am wrong.

Thank you very much.

[Ryan (Cloud Platform Support)]

Anastasios hit the nail square on the head. Idle instances are instance that are doing nothing. When a request comes in it will start a 4th. With your setup it will be more aggressive in shutting down the spare instances that's all. They are meant to have instances always ready to take requests. Depending on the language you are using it is more or less important to have these. Python is the fastest to boot, so you don't need as many, Java is the longest to boot so you need more idle. If Anastasios assumptions are correct and there are no end user interactions you can leave it at 0 and just take a slight hit when you send your first request.

It looks like you want to only ever have 3 instances. You should use Manual scaling with instances set to 3.

If you have no need for instant response you can use basic with max_instances set to 3, this will scale down the number when they are not needed but cap at 3.

@Anastasios be careful using daily budget to limit the number of instances. When you reach the cap it will stop all traffic request and not do any work until it resets.

[Alex Martelli]

On Fri, Oct 23, 2015 at 6:43 AM, Ryan (Cloud Platform Support) <rbru...@google.com> wrote:

Anastasios hit the nail square on the head. Idle instances are instance that are doing nothing. When a request comes in it will start a 4th. With your setup it will be more aggressive in shutting down the spare instances that's all. They are meant to have instances always ready to take requests. Depending on the language you are using it is more or less important to have these. Python is the fastest to boot, so you don't need as many, Java is the longest to boot so you need more idle.

When I last measured this for myself (a while ago) I noticed that Go was fastest (for a "hello world" app responding to a request starting from a state with zero instances running). Considering that Go is the only GAE runtime where applications are compiled to native-code binaries (no JVM or Python or PHP interpreter needs to spin up), this did not surprise me. However, you're better off doing your own measurements, since things may change.

 

If Anastasios assumptions are correct and there are no end user interactions you can leave it at 0 and just take a slight hit when you send your first request.

It looks like you want to only ever have 3 instances. You should use Manual scaling with instances set to 3.

If you have no need for instant response you can use basic with max_instances set to 3, this will scale down the number when they are not needed but cap at 3.

That's what I've been doing, personally, for modules that don't respond to users' requests, but rather only do "background processing" (cron and/or queued tasks) -- I forego autoscaling for those modules, in favor of basic or manual scaling, since for those modules I can usually predict the load reasonably well in advance and don't need to provision for sudden surges as I have to for modules that respond to users' requests (and besides having more predictability and no real fear of sudden surges, as you notice, Ryan, such "background processing" modules usually don't suffer much if some extra latency is added, as "user-facing" modules would).

To choose between basic and manual scaling, consider that you're charged for (wall-clock elapsed time) 15 minutes *after* an instance is done serving its last request. So, it's not very useful if an instance goes idle (i.e is done serving its last request), shuts down say 5 minutes after going idle, then another 5 minutes later another request comes and a new instance has to spin up for the new request: you're still being charged for the instance that's shut down (and will be charged for a further 5 minutes more) PLUS the new one being spun up. It would have been a bit cheaper, as well as a bit faster, in this case, to just not let the first instance shut down -- i.e, to use manual scaling (where no automatic spin-up nor shut-down happens) rather than basic scaling.

I'm mentioning this because the OP wrote about a cron job running every 5 minutes (although in that case it's not entirely clear how many instances will be needed to serve all those jobs).

Alex

 

@Anastasios be careful using daily budget to limit the number of instances. When you reach the cap it will stop all traffic request and not do any work until it resets.

On Friday, October 23, 2015 at 1:41:00 AM UTC-4, Naresh Pokuri wrote:

I have started my GAE app with Auto-scaling having min-idle-instances 3, each with 1GB RAM and 2.4GHz processor(i.e F4_1G). And I have a cron job which runs on every 5 minutes. With this setup keeping application idle for one day should equal to 72 instance hours. But I see that it already reached 428 instance hours. So, I am clueless here how GAE calculates instance hours, with this alone I can keep my budget in control. Can someone help me in this instance hours

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/27eb091a-b210-4d1c-86e2-1d38a143ac65%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Naresh Pokuri]

Idle instances are instance that are doing nothing. When a request comes in it will start a 4th. With your setup it will be more aggressive in shutting down the spare instances that's all. They are meant to have instances always ready to take requests.

 

So, why GAE spins up 4th instance when I have three Idle instances to address the load. For one QA I have deployed app and load will be very less. In that case I feel that those idle instance can take up the load I feel.

[Anastasios Hatzis]

Naresh, your assumption is right, that one of the idle instances is taking up the load. That's the purpose of an idle instance: to quickly take up load, with-out the time-delay that is added by starting up a new instance.

But, while that instance is processing the request, it is not idle anymore. You have defined that you want at least three idle instances, so a fourth instance will be started, as soon as one of the instance isn't idle anymore. Let's see an example:

1. 09:00:00 instance #1: idle, instance #2: idle, instance #3: idle (total idle: 3, total instances: 3)
   
2. now cron-job request comes in, so it is assigned to one of your idle instances (let's assume your cron-job doesn't spawn many tasks in parallel, it's just one single request... result:
3. 09:00:01 instance #1: utilized with your cron-job, instance #2: idle, instance #3: idle (total idle: 2, total instances: 3)
   GAE will start another instance, because there are only two idle instances left, but you defined min-idle-instances: 3
   
4. 09:00:10 instance #1: utilized with your cron-job, instance #2: idle, instance #3: idle, instance #4: idle (total idle: 3, total instances: 4)
   
5. 09:00:41 now let's assume your cron-job finished (i.e. request took 40 seconds)
6. 09:00:42 instance #1: idle, instance #2: idle, instance #3: idle, instance #4: idle (total idle: 4, total instances: 4)
   because you have defined that you want at most 3 idle instances, GAE will immediately shutdown one of the idle instances
7. 09:00:43 instance #1: idle, instance #2: idle, instance #3: idle (total idle: 3, total instances: 3)
   

Because you have defined max-idle-instances, GAE is more aggressive in shutting down idle instances (see Ryan's post). The default behavior would be to shut down after 15 minutes of inactivity.

At 09:05:00 the same behavior as above will repeat. You will again have a forth instance started, and at the end one of the idle instances will be killed too early, so at 09:10 the same, and so forth.

As Alex mentioned, each instance will always be billed for at least 15 minutes, even so an instance was only idle/active for a few seconds or minutes. So, your max-idle-instances will cost you even more in the example earlier. If there were no max, GAE would let an idle instance live for the default of 15 minutes. But your next request comes already every 5 minutes, so it wouldn't shutdown an instance only to restart another one soon after, billing you both.

Based on your descriptions, I think that you don't need any idle instances, because your app only does the one cron-job, and typically a cron-job can wait for any start-up time (which is unlikely in a 5 minute schedule anyway). But I can only assume, which configuration would make sense for your app or this module, there are many questions left (e.g. if your app serves end-users, if start-up time for cron-jobs is tolerable, how many requests are caused per cron-job, how long do these requests run).

[Naresh Pokuri]

You are awesome. Its very clear now. Now I will do my analysis and choose my type scaling and no.of instances. Thanks once again