Did the standard environment auto scaling scheduler change recently?

[Ryan B]

Hi all! The steady state instance count for a couple of my apps (on Python 3 Standard) increased recently, over the last week or two, without any obvious code or architecture or traffic or load or change on my end. Did the App Engine scheduler itself change recently?

Here's the active instance count for one app. I haven't deployed a new version since April, and load (etc) haven't meaningfully changed either.

Here's another. I deploy this one more often, multiple times a week, but nothing related to utilization recently that I can think of, and there haven't been any other architectural, app config, or load for months.

Here are the app.yaml files; neither has changed since March, when I upgraded both apps to Python 3.9.

https://github.com/snarfed/bridgy-fed/blob/main/app.yaml

https://github.com/snarfed/bridgy/blob/main/app.yaml

If the scheduler did change, any chance anyone knows how to configure it to get my steady state instance counts back to where they were before?

Thanks in advance!

[Ryan B]

I changed my autoscaling settings to be much more aggressive, below, and steady state instance count is back down to more or less where it was before. I'm stil very curious what changed on App Engine's end, but I'm at least out of the woods for now.

automatic_scaling:

 max_idle_instances: 1

 target_cpu_utilization: .9

 min_pending_latency: 500ms

 max_concurrent_requests: 30

[yananc]

Hi, 

As per the Release Notes, there is no update recently on App Engine Standard auto-scaling scheduler. In the app.yaml files you provided, I could see a huge difference for ‘min_pending_latency’. Are you aware of it? If it is expected, I would suggest sampling typical periods of time between the different patterns and closely inspecting and cross checking on any possible traffic difference. Hope it helps!

[Ryan B]

Thanks for looking! I made that app.yaml change (including min_pending_latency) after I noticed the scheduling change. It successfully brought my instance count back down, but it didn't explain what caused the initial increase. I'm still curious about that.

[wokmou]

In order to received additional information on the issue you experience, I would suggest that you open a case with the support service [1] so that a root a root cause analysis can be performed.

[1] https://cloud.google.com/support

[solsTiCe d'Hiver]

I am seeing a similar change;

I mean I don't know the cause yet but previously, I was billed around 5€ for a month of running my project on app engine, but now, I am billed around 1€ by day, so this means that by the end of the month it will be 30€ !!

I can't find any metrics or statistics (I am lost there) to identify the problem.

It could be that more users use my app, and then  more instance are running and cause more uptime.

But that is a huge difference, where my users are still the same amount around 200 visitors per day.

Something changed since the last month.

I am using the python 3 standard env.

[barrado]

There hasn't been any recent change in the App Engine Standard environment that could have lead to an increase in the billing amount. According to the documentation an App Engine Standard service is billed for the number of instances it has during the time and for the network traffic. 

I strongly suggest you to use Cloud Monitoring  to check if you usage patterns have change in the last weeks.

[solsTiCe d'Hiver]

I am not alone.

some people seems to think there was definitvly a change on App Egine runtime scheduling: https://serverfault.com/a/1073568/297044

And for me, I migrated to another host, where my costs are capped, and will not have such surprised each month.